Ecient learning of sparse image representations using homeostatic regulation

One core advantage of sparse representations is the ecient coding of complex signals using compact codes. For instance, it allows for the representation of any sample as a combination of few elements drawn from a large dictionary of basis functions. In the context of the ecient processing of natural images, we propose here that sparse coding can be optimised by designing a proper homeostatic rule regulating the competition between the elements of the dictionary. Indeed, a common design for unsupervised learning rules relies on a gradient descent over a cost measuring representation quality with respect to sparseness. e sparseness constraint introduces a competition which can be optimised by ensuring that each item in the dictionary is selected as o en as others. We implemented this rule by introducing a gain normalisation similar to what is observed in biological neural networks. We validated this theoretical insight by challenging the matching pursuit sparse coding algorithm with the same learning rule but with or without homeostasis. Simulations show that for a given homeostasis rule, gradient descent performed similarly the learning of a dataset of image patches. While the coding accuracy did not vary much, including homeostasis changed qualitatively the learned features. In particular, homeostasis results in a more homogeneous set of orientation selective lters, which is closer to what is found in the visual cortex of mammals. To further validate these results, we applied this algorithm to the optimisation of a visual system to be embedded in an aerial robot. In summary, this biologicallyinspired learning rule demonstrates that principles observed in neural computations can help improve real-life machine learning algorithms. It is observed that simple cell neurones in mammalian primary visual cortex are selective to orientation, spatial localisation, and frequencies [1]. It is demonstrated that developing a coding strategy that maximises sparseness is sucient to form receptive elds that account for all three of the above properties [2]. Visual items composing natural images are oen sparse, such that the brain may use this sparseness to reconstruct images with only a few set of these items [3]. is is supporting the idea that an unsupervised learning algorithm based on sparse coding could be use to describe eciently image processing in the primary visual cortex. Most of existing models of unsupervised learning aim at optimising a cost dened on prior assumptions on representation’s sparseness. For instance, learning is accomplished in SparseNet [4] on patches taken from natural images as a sequence of coding and learning steps. First, knowing a dictionary of receptive elds Φi , the sparse coding is achieved using a gradient descent over a convex cost derived from a sparse prior probability distribution function of the coecients ai . en, knowing this sparse solution, learning is dened as slowly changing the dictionary using Hebbian learning. In general, the parameterisation of the prior has major impacts on results of the sparse coding and thus on the emergence of edge-like receptive elds and requires proper tuning. In fact, the denition of the prior corresponds to an objective sparseness and does not always t to the observed probability distribution function of the coecients. In particular, this could be a problem during learning if we use the cost to measure representation eciency for this learning step. An alternative is to use a more generic `0 norm sparseness, by simply counting the number of non-zero coecients: