Efficient Visual Coding: From Retina To V2
Authors: Honghao Shan, Garrison Cottrell
ICLR 2014

Abstract: The human visual system has a hierarchical structure consisting of layers of processing, such as the retina, V1, V2, etc. Understanding the functional roles of these visual processing layers would help to integrate the psychophysiological and neurophysiological models into a consistent theory of human vision, and would also provide insights to computer vision research. One classical theory of the early visual pathway hypothesizes that it serves to capture the statistical structure of the visual inputs by efficiently coding the visual information in its outputs. Until recently, most computational models following this theory have focused upon explaining the receptive field properties of one or two visual layers. Recent work in deep networks has eliminated this concern, however, there is till the retinal layer to consider. Here we improve on a previously-described hierarchical model Recursive ICA (RICA) [1] which starts with PCA, followed by a layer of sparse coding or ICA, followed by a component-wise nonlinearity derived from considerations of the variable distributions expected by ICA. This process is then repeated. In this work, we improve on this model by using a new version of sparse PCA (sPCA), which results in biologically-plausible receptive fields for both the sPCA and ICA/sparse coding. When applied to natural image patches, our model learns visual features exhibiting the receptive field properties of retinal ganglion cells/lateral geniculate nucleus (LGN) cells, V1 simple cells, V1 complex cells, and V2 cells. Our work provides predictions for experimental neuroscience studies. For example, our result suggests that a previous neurophysiological study improperly discarded some of their recorded neurons; we predict that their discarded neurons capture the shape contour of objects.


Peer review:
[Review]:
The paper applies the recursive ICA algorithm to visual data and describes the connections between the layers in the model to neural representations.  The RICA algorithm is modified from its original form to utilize the author's previous innovation of 'sparse PCA.'  The results are conveyed through a few depictions of the receptive fields and some qualitative links to neurophysiology.

Contributions:
> The authors have updated the RICA algorithm to utilize SPCA instead of PCA.
> The authors claim that sPCA on the outputs of the first-layer ICA, produces V1-complex cells.
> The authors suggest that some unoriented neurons in a V2 experiment correspond to a subset of their second ICA layer.

Overall, this is an interesting direction and there are signs of promising results.  However, the poor exposition, preliminary nature of the result, and lack of quantitative matching to neurophysiology (or other strong metrics for differentiation/novelty), indicate that this work is not yet ready for publication/acceptance.  The update of RICA to use SPCA is a minor innovation, and the claims about V1 complex cells and V2 are not well supported by experiments (matching to neural findings).  In the following I provide some feedback that I hope will assist the authors in this work and a future publication of this work.

In the background, more attention should be paid to similar techniques to RICA.  For example:
Chen and Gopinath 2001 Gaussianization. NIPS.
Lyu and Simoncelli 2009, Nonlinear extraction of 'Independent Components' of natural images using radial Gaussianization. Neural Comp.

and there are a number of works that produce findings similar to your second ICA layer:
Y. Karklin and M. S. Lewicki, A hierarchical Bayesian model for learning non-linear statistical regularities in non-stationary natural signals, Neural Computation, 2005.
A. Hyvärinen, M. Gutmann and P.O. Hoyer. Statistical model of natural stimuli predicts edge-like pooling of spatial frequency channels in V2. BMC Neuroscience, 6:12, 2005. 
Honglak Lee, Chaitu Ekanadham, and Andrew Y. Ng. Sparse deep belief net model for visual area V2. NIPS 20, 2008.
CF Cadieu, BA Olshausen, Learning intermediate-level representations of form and motion from natural movies. Neural computation 24 (4), 827-866
These should be discussed.

Some points on exposition:
> be careful when referring between human visual system and then relying on non-human primate data.  You seem to use these different species interchangeably
> your statement that 'a previous neurophysiological study improperly discarded some of their recorded neurons' could be rephrased.  I think you can be more fair in discussing the actions of our colleagues.
> You repeated state 1st layer ICA algorithms (including yours) learn edge/bar shaped receptive fields.  This statement is false.  The generative fields (and receptive fields) resemble Gabor functions.  Gabor functions are not edge or bar shaped.  They are localized in space, orientation and position.  While edges/bars are broadband.  Don't fall into this sloppy language.  Note also that results in neurophys. indicate that V1 are not edge/bar selective, but selective for stimuli localized in space, orientation and position (just like Gabors, and unlike edges/bars).
> you use both 'sPCA' and 'SPCA'.
> you are missing citations that you refer to in the text.
> the term 'autoencoder' refers to a specific class of models and I do not believe your optimization falls into this class.

My major suggestion is to spend more time on the link between the properties of the intermediate layers of your model and the neurophysiology literature.  There is a tremendous amount of data on V1-complex cells and you can produce non-trivial (meaning you need a control) links between your model and these quantitative findings.  While there are fewer results on V2, there are quantitative experiments to run on your model.  Your current results are too qualitative and require the reader to make leaps from the receptive field depictions you use and an experimental result using certain stimuli, plotted in a completely different methodology (Figure 5 and 6).  The experiment in Figure 4 is a step in the right direction, but showing one cell isn't sufficient, there is no control, and their are quantitative distribution metrics that are relevant in the Chen, Han, Poo and Dan PNAS 2007 paper (e.g. Figure 5 in that paper).