Neuronal Synchrony in Complex-Valued Deep Networks
Authors: David Reichert, Thomas Serre
ICLR 2014

Abstract: Deep learning has recently lead to great successes in tasks such as image recognition (e.g Krizhevsky et al., 2012). However, deep networks are still outmatched by the power and versatility of the brain, perhaps in part due to the richer neuronal computations available to real cortical circuits. The challenge is to identify which neural mechanisms are relevant, and to find suitable abstractions to model them. Here, we show how aspects of spike timing, long hypothesized to play a crucial role in cortical information processing, could be incorporated into deep networks to build richer, versatile deep representations. 
We introduce a neural network formulation based on complex-valued neuronal units that is not only biologically meaningful but also amenable to a variety of deep learning frameworks. Here, units are attributed both a firing rate and a phase, the latter indicating properties of spike timing. We show how this formulation qualitatively captures several aspects thought to be related to neuronal synchrony, including gating of information processing and dynamic binding of distributed object representations. Focusing on the latter aspect, we demonstrate the potential of the approach in several simple experiments. Thus, synchrony could implement a flexible mechanism that fulfills multiple functional roles in deep networks.


Peer review:
[Review]:
SUMMARY                                                                                       
  
The paper 'Neural Synchrony in Complex-Valued Deep Networks' tackles the important question of how it is that neural representations encode information about multiple objects independently and simultaneously.
The idea that the relative phase of periodic neural responses has been in circulation for some time (and this paper provides a good overview of relevant literature), but to date the principle has not gained traction in the pattern recognition side of neural modeling. This paper aims to change that by showing that a complex-valued Deep Boltzmann Machine can naturally segment images (in some simple synthetic cases) according to the various visible objects, through the phase of latent responses. 
 
The basis for the technical contribution of this paper is a novel response function.  The [complex-valued] response function described by equations 1 and 2 is a function of a complex-valued weight vector applied to a complex-valued feature vector. Output z_j = r_j e^{i 	heta_j} of each model neuron is determined by arg(z_j) = arg(w . x)
  
    f( alpha  |sum_j w_j . x_j | + eta sum_j (w_j  |x_j|) )
 
Comment: The text leading up to equation 1 is confusing regarding z. Is it an output or an input?  It doesn't seem like we're dealing with a dynamical system, and the input was called x in the paragraph above.
  
Comment: the use of |x| in equation 2 is confusing because presumably |w . x| is a vector norm whereas in w . |x| it denotes elementwise magnitude of the complex elements of x. Right?
 
Comment: the authos mention that the two terms in f() can be weighted, but don't include those weights in Eq. 2, as I have done above (alpha, eta).
  
Why use this function? The authors make an intuitive argument in the text that these two terms capture salient aspects of a more detailed spiking network based on Hodgkin-Huxley neurons, and Figure 1 illutrates the effect quantitatively for a specific, simple 2-to-1 feedforward network of rhythmic neurons. The value of this transfer function as a surrogate for detailed compartmental models is interesting, but is not the focus of the remainder of the paper.
  
 
The paper's section 3 'Experiments: the case of binding by synchrony' was somewhat difficult for me to understand. A [conventional, real-valued] DBM was trained on small pictures with horizontal and vertical bars, and then 'converted' to a complex-valued network (and was the activation function changed to the one from Eq. 2? What does that mean in terms of inference in the DBM?) It was found that when clamping the visible-unit magnitudes to a particular picture, and 'sampling' (is this actually sampling from a probability distribution?) their phase and the hidden units' magnitude and phase, that there were groups of hidden units with phases that lined up with particular bars. This is good because it suggests a means of teasing apart the DBM's latent representation into groups that are 'working together' to represent something independently from other groups. (I wanted to see some sort of control trial, showing that a plain old real-valued DBM could not achieve the same thing, but I can't really think of the right thing to try.)
 
The demonstration in Figure 3 shows that already with this set of bars, that there is an issue of phase resolution: it appears that four different bars are all coded the same shade of green. Is this a problem? A readout mechanism might be confused and judge all these bars to be one object, even though bars occur independently in the training data. Figure 3 illustrates what happens after 100 iterations of sampling, what happens after more iterations? Do the co-incidentally green bars change color independently of one another?

Overall, this research is highly relevant to the aims of the ICLR conference. It is at an early stage of development, in that no learning algorithm has been adapted to work with these complex-valued neurons (although the authors might consider adapting the ssRBM), the images used in the experiments are simple and synthetic, and the authors themselves lament that 'conversion' of a DBM is unreliable. Still, the idea of phase-based coding has a lot of potential, and it is worth exploring.  This paper would be an important step in that process. I would strongly suggest that the authors upload their Pylearn2 code so that others can reproduce the effects presented in this paper, especially if training and conversion of DBMs is as unreliable as they suggest.
 
 
NOVELTY AND QUALITY
 
- the use of complex-valued phase to perform segmentation in a DBM is novel

- quality of presentation is very good
 
 
PRO & CON
 
pro: phase-based segmentation is an intriguing idea from theoretical neuroscience, it's great to see it put to the test in engineering terms
 
con: the stimuli are quite simple compared with other deep learning and vision applications

con: the model is at an early proof-of-concept stage

con: no natural learning algorithm yet for the model