Title: On Representation of Natural Image Patches
Venue: NeurIPS 2023
Accepted: no
Keywords: neural coding

sparse binary code

natural image

edge detection

generative model
Abstract: To optimize survival, organisms need to accurately and efficiently relay new information throughout their systems for processing and responses. 
Furthermore, they benefit from predicting environmental occurrences, or in mathematical terms, understanding the probability distribution of their environment, 
based on both personal experiences and inherited evolutionary memory.
These twin objectives of information transmission and learning environmental probabilistic distributions form the core of an organism's information processing system. 
While the early vision neuroscience field has primarily focused on the former, employing information theory as a guiding framework, 
the latter is largely explored by the machine learning community via probabilistic generative models. 
However, the relationship between these two objectives has not been thoroughly investigated.
In this paper, we study a biologically inspired information processing model and prove that these two objectives can be achieved independently.
By evenly partitioning the input space to model input probability, our model bypasses the often intractable normalization factor computation. 
When applied to image patches, this model produces a sparse, nonlinear binary population code similar to early visual systems, 
with features like edge-detection and orientation-selective units. 
Our results not only offer potential new insights into the functioning of neurons in early vision systems, 
but also present a novel approach to represent natural image patches.

Abstract

To optimize survival, organisms need to accurately and efficiently relay new information throughout their systems for processing and responses. Furthermore, they benefit from predicting environmental occurrences, or in mathematical terms, understanding the probability distribution of their environment, based on both personal experiences and inherited evolutionary memory. These twin objectives of information transmission and learning environmental probabilistic distributions form the core of an organism's information processing system. While the early vision neuroscience field has primarily focused on the former, employing information theory as a guiding framework [3, 32, 19, 1, 9, 28], the latter is largely explored by the machine learning community via probabilistic generative models. However, the relationship between these two objectives has not been thoroughly investigated. In this paper, we study a biologically inspired information processing model and prove that these two objectives can be achieved independently. By evenly partitioning the input space to model input probability, our model bypasses the often intractable normalization factor computation. When applied to image patches, this model produces a sparse, nonlinear binary population code similar to early visual systems, with features like edge-detection and orientation-selective units. Our results not only offer potential new insights into the functioning of neurons in early vision systems, but also present a novel approach to represent natural image patches.

1 Introduction

Nature, through billions of years of evolution, has likely developed optimal methods for processing 22 visual information within the constraints of biological feasibility. However, attempting to precisely 23 emulate every detail of these biological systems [30, 21] in order to construct an optimal visual 24 information processing model presents significant complexities, especially without a comprehensive 25 understanding of the underlying principles. 26

In parallel, deep learning models, particularly Convolutional Neural Networks (CNNs), have demon27

strated exceptional performance in various computer vision tasks, such as image classification, object 28

detection, and semantic segmentation. Despite their success, these models still fall short of biological 29

vision systems in several key areas such as the ability to generalize from limited data, robustness to 30

variations, 3D understanding, and processing speed and efficiency. 31

Moreover, deep learning models pose several unresolved challenges. Firstly, their decision-making 32

processes are often opaque, leading to the "black box" label. Secondly, deep learning architectures 33

typically involve numerous layers without explicit functions associated with each layer [12, 8], unlike 34

the biological brain where each stage of visual processing has a distinct role and purpose [7]. Lastly,he 35

sequence of information processing in deep learning models is largely dictated by their architectural 36 structure. It is still a question how to ascertain when one processing stage is finished and when it's 37 appropriate to pass information to the next layer. 38

Drawing inspiration from biological systems and prior studies [19, 1, 9, 28], and with a view to 39 find an alternative approach to the current deep learning framework, this paper aims to explore the 40 fundamentals of the first stage of an optimal visual information processing system. This exploration is 41 undertaken incrementally, starting from a single pixel, and progressively advancing to image patches. 42

2 One Pixel 43

We begin with the simplest conceivable case, where the input to the model consists of a single pixel 44 with one color channel. Although seemingly trivial, this model can represent various biological units. 45 For example, it could model the eyespot of single-celled organisms like Euglena, the large monopolar 46 cells found in an insect's compound eye or the bipolar cells in the retina. The single pixel case has 47 been studied [14, 1]. We aim to review it to introduce the central concepts and the main theory, which 48 will also be applicable to more complex scenarios. 49

Let us denote the light intensity of the pixel as x , and let p ( x ) represent its probability distribution. 50 We define an information processing unit (IPU) as a model that receives inputs and passes processed 51 information to subsequent stages. As the first stage in an organism's information processing system, 52 the single pixel IPU carries the same dual objectives as later stages: transmitting information and 53 learning environmental probabilistic distributions. Therefore, the two objectives for the single pixel 54 IPU are to transmit information about x efficiently through its output and to learn p ( x ) . 55

56

57

58

59

60

61

Information is quantified by Shannon entropy. To compute the Shannon entropy of the input, we assume x is a discrete variable with M states, after all light intensity is quantized according to quantum mechanics, although M could be a very large number, making x practically indistinguishable from a continuous variable. The information obtained from the pixel when we know the intensity is x can be represented as I ( x ) = -log p ( x ) . The average information a state of the pixel contains, the Shannon entropy, is given by:

The IPU transforms the input x into the output y = f ( x ) , where the output space comprises N 62 distinct states. In contrast to previous studies that assumed one-to-one mapping between input and 63 output, here we posit that N ≪ M . This assumption is more congruent with biological constraints; 64 for instance, the luminance resolution levels at synapse terminals in a zebrafish's retina are only about 65 10 [25]. Moreover, this assures that after processing the information is significantly reduced, thereby 66 simplifying the tasks for subsequent stages. 67

The function f ( x ) assigns x into N groups, each corresponding to a fixed y . We denote all x values 68 in group j as G j , and the size of this group as n j . The entropy of the output is given by 69

where Q ( y ) represents the probability distribution of the output states. 70

Previous research [14, 1] has primarily emphasized the first objective of an IPU, which is maximizing 71 the rate of transmission [19]. This goal is particularly relevant for early-stage IPUs, where the 72 distinction between signal and noise is not yet clear. Maximizing the rate of transmission is equivalent 73 to maximizing H Q (see proof in Appendix A), leading to a constant Q ( y ) . Biological neurons have 74 been observed to follow this coding scheme [14]. 75

Simultaneously, an IPU should also strive to fullfil the second objective and model p ( x ) as accurately 76 as possible. Mathematically, this involves minimizing the Kullback-Leibler divergence between 77

p ( x ) and the distribution learned by the IPU. This raises an interesting question: Are these two 78 optimization objectives contradictory, or do they essentially represent the same task? 79

80

3 Even Code Principle

To determine how an IPU models p ( x ) we need to translate the output probability distribution Q ( y ) 81 into the input space as q ( x ) . q ( x ) is a step function: 82

and we have the following relations: 83

Minimizing the difference between p ( x ) and q ( x ) can be achieved by minimizing their Kullback-Leibler divergence:

where H pq is the cross entropy. It can be proved that the cross entropy H pq is equal to the entropy of the learned distribution in the input space defined as (see proof in Appendix B):

Since H p is fixed, minimizing the KL divergence requires minimizing H q . The previous question now 89 transforms into understanding the relationship between maximizing the entropy of the distribution in 90 the output space ( H Q ) and minimizing the entropy of the learned distribution in the input space ( H q ). 91

Suppose we have two adjacent zones in the transformed space where the corresponding Q ( y 1 ) and 92 Q ( y 2 ) are not equal, let's assume Q ( y 1 ) > Q ( y 2 ) . One can reduce the inequality by shifting the 93 boundary between these two zones and moving one x value from G 1 to G 2 . This shift corresponds 94 to a small change of probability, δ , for both zones. Note that δ is comparable to q 1 and q 2 , as we 95 assume the distribution is smooth. We know that reducing the inequality of Q ( y 1 ) and Q ( y 2 ) always 96 increases H Q . If the two optimization problems are the same, then H q should increase; if they are 97 contradictory, H q should decrease. The change of H q can be calculated as: 98

The change can either be positive or negative depending on q 1 and q 2 . Since minimizing H q and 99 maximizing H Q are not contradictory, these objectives can be tackled independently. Given a fixed 100 number of output levels N , we first maximize H Q to retain as much input information as possible. If 101 further refinement of p ( x ) modeling is required, we can increase the output resolution N . 102

The aforementioned reasoning extends naturally to multivariate scenarios, as no assumptions about 103 one-dimensionality of the input were made. We articulate the goal of a general information processing 104 unit as follows: An information processing unit (IPU) transforms input space with M states into 105 output space with N states, where N ≪ M . Given a smooth input probability distribution as 106 M →∞ and a piecewise smooth transformation function, the sole goal of an IPU with a fixed output 107 resolution N is to yield an even output probability distribution, hence retaining maximum information 108 from the input. To attain better modeling precision, the output resolution N of the IPU should be 109 increased. This will be referred to as the principle of even code. In the next sections, we will apply 110

84

85

86

87

88

and we get

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

4 Two Pixels

For two pixels ( x 1 , x 2 ) , we can either use one IPU directly to model p ( x 1 , x 2 ) or use two IPUs to model p ( x 1 ) and p ( x 2 ) separately, followed by another IPU to model the outputs p ( y 1 , y 2 ) . We will use the second approach, as processing as much information locally reduces the cost of information transfer. In fact, when images are stored on computers, gamma encoding is utilized to create an approximately even distribution of pixel values. When these images are displayed, pixel values undergo gamma correction to recover the original statistics for human eyes to process. In the following sections, we will assume that all pixel values x have already been processed by dedicated IPUs, resulting in a roughly even probability distribution.

The probability distribution p ( x 1 , x 2 ) of natural images is relatively simple. The majority of the probability is concentrated around the diagonal line x 1 -x 2 = 0 , with p ( x 1 , x 2 ) rapidly decaying as | x 1 -x 2 | increases (see Fig. 1 (a) for example). Intuitively, we can use lines parallel or/and perpendicular to x 1 -x 2 = 0 to divide the probability distribution into even partitions.

4.1 One Basis

To investigate how IPUs learn p ( x 1 , x 2 ) , we conduct numerical experiments using a multilayer perceptron (MLP) as the IPU to approximate y = f ( x ) and model p ( x ) [23]. Other function approximation methods may also be applicable. To partition the input probability distribution with one set of parallel lines, only one IPU with N output nodes is needed. According to the even code principle, for each input, only one of the N output nodes should be activated, and the probability of activating any one of the N output nodes should be equal. We use the softmax function as the last layer of the MLP to ensure each output value is within [0, 1], and that if a node is activated (output value equals 1), it is the only node being activated. We use stochastic gradient descent and the following loss function to train the MLP:

y si represents the value of the i-th output node for the s-th input sample, while ⟨⟩ s denotes the average over all samples in a training batch. The first term in the loss function ensures each output node has an equal chance to be activated on average. The second term promotes activation of only one node per input while suppressing the remaining nodes, mimicking lateral inhibition when combined with the softmax function. The factor k balances the two terms to achieve the desired result. Fig. 1 (a) show the results learned by MLPs with 16 output nodes.

4.2 Multiple Bases

To partition the input space with two sets of orthogonal lines we need two MLPs. The orthogonality is achieved by enforcing

where N 1 and N 2 represent the number of output nodes of the two MLPs (refer Appendix C for proof). If more than two orthogonal bases are required for partitioning the space, we can enforce Eq. (12) for each combination of two bases to ensure orthogonality between them. The loss function for multiple orthogonal bases with independent states is

where b is the base index, and B is the number of bases. ∑ <b,b ′ > denotes the sum over all ( B 2 ) combinations of two distinct bases. y bsi y b ′ sj is the probability Q ( y b , y b ′ ) for the sample s when y b and y b ′ take their i-th and j-th value respectively.

Fig. 1 (b) shows an example of two-pixel input space partitioning using two orthogonal bases. For a 151 more detailed discussion on the experiments and additional results, refer to Appendix D. 152

(a)

(b)

1.0

1.0

0.8

0.8

0.6

0.6

2

2

0.4

0.4

0.2

0.2

Base1partitions

One basepartitions

Base2partitions

0.0+

0.0+

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pixel 1

Pixel 1

Additionally, it's worth noting that orthogonal bases with independent states might model grid cells 153 [11] in the entorhinal cortex, though this topic is beyond the scope of the current paper. 154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

5 Image Patches

Next we move on to study gray and color image patches. We use x to represent the vector of input pixel values of the image patch. The multivariate input probability distribution p ( x ) is considerably more complex compared to the previous examples. If we use only one basis to discretize the input probability space (e.g. Fig. 1 (a) in Appendix D), the required number of independent states for a good approximation would be very large, making the evaluation of the softmax function computationally expensive. On the other hand, using multiple orthogonal bases would also significantly increase the computational cost to ensure orthogonality if the number of bases is more than just a few. Additionally, determining the optimal number of bases and the number of independent states for each basis are challenging.

Aside from the computational cost, another issue arises when working with image patches: we want the representation to capture the similarity between inputs. However, using orthogonal bases with independent states makes it difficult to gauge input similarity through methods such as calculating the difference between representations, even if we can establish an order for the states of each basis. Therefore, we need a more suitable coding scheme for complex inputs like image patches.

Real-valued vectors are a natural choice, given their extensive use in representing a variety of entities 170 such as images, texts, and categorical variables [13, 22, 10]. The norm of the difference between 171 two vectors can function as a measure of similarity. Nevertheless, if we want the representation y to 172 mirror input similarity, each value of y should encapsulate all samples perceived as identical within 173 the same group G . Under this constraint, Q ( y ) cannot remain constant, thereby conflicting with the 174 even code principle. 175

The resolution to this conflict involves permitting the representation to mirror input similarity at the 176 most granular level, while enforcing the even code principle at a larger scale in the transformed space. 177 We will detail this method in subsequent sections. 178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

5.1 Loss Function

To promote even distribution, we incorporate a loss function that compels input samples to repel each other in the transformed space. This repulsive force diminishes with increasing distance, as described by the following equation:

Here, -ln | y s -y s ′ | represents the potential energy due to the repulsive force, which is proportional to the inverse of the Minkowski distance between the representations of samples s and s ′ in the transformed space. Alternative forms of potential energy and distance measures could also be applicable. ⟨⟩ <s,s ′ > denotes the average over all sample pairs.

Should numerous samples converge at one point in the transformed space, they will exert a strong repulsive force in the surrounding area, thereby discouraging other samples from occupying nearby positions. To prevent samples from pushing each other infinitely far apart, we restrict the representation values to be within the range [0 , 1] . With this constraint, the repulsive force pushes samples towards the vertices of the unit hypercube, effectively reducing the representations from real vectors to binary vectors. As a result, an even distribution is achieved on a larger scale in the transformed space, which consists solely of the vertices.

In the context of binary vectors, the collection of output nodes can be viewed as a vocabulary, and activated nodes by an input image patch act as its representative tokens. Unlike fixed-length representations with real-valued vectors, binary representations can employ fewer tokens for more common image patches (e.g., homogeneous patches), and more tokens for less common, structurallyrich patches. This can be accomplished by introducing a second term, ⟨| y s |⟩ s , to the loss function, which echoes the sparsity regularization term found in various studies [9, 16, 26, 27, 29, 4]. The updated loss function becames:

where α is a free parameter to adjust sparsity.

In practice, we add a small value ϵ = 10 -38 to the distance, allowing slightly different samples to share the same representation and enhancing numerical stability. Another approach to improve numerical stability involves using a theoretically equivalent form of the loss function, which instead of allowing samples to repel each other in the output space, we enable nodes to repel one another, encouraging output nodes to be as independent as possible [24].

5.2 Experiments

In the following experiments, we use either a single MLP with N outputs, or N MLPs each with one output, as the IPU to approximate the transformation function y = f ( x ) and model p ( x ) . The last layer of the MLP is a sigmoid layer, ensuring the output value ranges between 0 and 1. Our training data comprises random image patches extracted from the COCO 2017 image dataset [18] or the ImageNet dataset [6]. No image preprocessing is used. Additional training details are provided in Appendix E.

5.2.1 Output Statistics

First, we examine the statistics of the learned representation. Across all experiments, we observe qualitatively similar output statistics, irrespective of the IPU architectures and training specifics, provided the training has properly converged. For illustration, we present an example using a model trained on 5 × 5 color image patches. It uses 96 MLPs, each with one output node and a middle layer of 48 nodes, as the IPU. Following training, the model is used to generate representations for 1 million random image patches for this analysis.

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

(a)

(b)

10

0.175

Mean: 0.141

Probability

0.150

Probability Density

101

0.125

0.100

Activation

100

0.075

0.050

10-1

0.025

0.000

0.0

0.2

0.4

0.6

0.8

1.0

0

20

40

60

80

Outputvalue

Output NodeIndex

Fig. 2 (a) presents the histogram of output values on a logarithmic scale. The vast majority of output values are either at 0 or 1. As such during inference, we can round the outputs to yield a binary representation. Fig. 2 (b) illustrates the probability of an output node being activated by a random image patch. All nodes demonstrate similar activation probabilities, indicating an even distribution at this coarse scale. Further statistical analysis of the output representation is available in Appendix F.

5.2.2 Image Patch Similarity

Next, to examine how the learned representation reflects the similarity between image patches, we display 16 random image patches, each followed by 9 image patches similar to them in the binary representation space, as shown in Fig. 3 (a). The same 5 × 5 color image patch model is used. The learned representation clearly captures perceptual similarity. The results shown in Fig. 2, Fig. 3 (a), and additional results in Appendix E confirm that with the loss function Eq. (15), we can indeed learn a sparse binary representation which reflects the image similarity while adhering to the even code principle.

For comparison with a traditional convolutional neural network, we present the results generated with the first 10 layers of a VGG16 model [31] pre-trained on ImageNet in Fig. 3 (b). The image patch representation from the first 10 layers of the VGG16 model is a float vector of size 128. The even code model, with only 96 binary outputs, achieves results similar to the VGG16 model. These 96 binary outputs occupy the same storage space as a float vector of length 3 - just 1/42 of the VGG16 representation's size, which underscores the exceptional efficiency of the even code method in image patch representation.

5.2.3 Local Edge Detectors and Orientation-Selective Units

Biological visual systems' initial stages are known to possess local edge detectors and orientationselective units [17, 2]. While CNNs have been successfully trained to detect boundaries via supervised learning [20], their initial layers have not shown proficiency in edge detection [15]. Notably, prevalent local edge detection algorithms, such as the Canny edge detector [5], still primarily rely on non-deep learning methods.

Does the even code model, proposed as the initial stage of an optimal image processing system, 247 resemble biological systems more closely? To answer this, we trained an even code model on 4 × 4 248 grayscale image patches and applied it to images with a stride of 1 pixel, generating feature maps for 249 each output node. The model comprises a MLP with 64 outputs and an intermediary layer with 100 250 nodes. Fig. 4 illustrates the feature maps of 4 output nodes of the even code model for 4 different 251 input images. It also shows edges generated by the Canny edge detector as comparison. Interestingly, 252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

(a) Even code model with 96 binary outputs

(b) Early layers of VGG16 with 128 real outputs

with this simple network architecture, the even code model demonstrated a remarkable capability in edge detection, rivaling the multi-stage Canny edge detector.

Furthermore, Fig. 5 shows the feature maps of 5 output nodes for a sample bike image. Spokes of different orientations activate different nodes, indicating that these output nodes of the even code model have varying orientation preferences, similar to orientation-selective units found in bilogical systems.

6 Conclusion

In summary, this paper demonstrates that maximizing the information-carrying capacity of output channels and modeling the input probability distribution are not mutually exclusive objectives and can be pursued independently. Given a specific output resolution, the sole goal of an information processing unit is to preserve as much information from the input as possible by ensuring an even distribution of samples in the output space. We applied the even code principle to study the probability distributions of two-pixel systems and image patches. For the two-pixel system, we learned orthogonal bases with independent states to model its probability distribution. For image patches, the even code approach naturally led to a nonlinear sparse binary representation. The even code model also shares additional similarities with early visual systems, such as the presence of local edge-detecting and orientation-selective units. These similarities suggest that the even code model could potentially serve as a new representation for neurons in early visual systems.

C

There are several intriguing directions for future research. First, the even code model has been 271 applied to inputs ranging from as simple as one pixel to more complex color image patches. Can we 272 extend its application beyond the early stage of visual information processing? Second, the even code 273 model could be extended to videos by incorporating an additional time dimension alongside color, 274 width, and height dimensions. Investigating time-varying inputs, which produce spike train-like 275 outputs, and conducting an in-depth comparison with early visual systems would be very interesting. 276 Third, the even code model can also be extended to binocular vision data by adding another input 277 dimension of size two. Whether the model with binocular and/or video data can construct a 3D 278 model of the world based on data of two spatial dimensions is an intriguing question. Fourth, while 279 this paper focuses on visual information, the even code model is a general method that could be 280 applied to model other multivariate probability distributions as well. Lastly, on the application 281 side, the even code model has potential in various areas, including local edge detection, image and 282 video compression/denoising/retrieval, texture classification, and multispectral/hyperspectral image 283 processing. 284

285

286

287

288

289

290

References

Joseph J. Atick. Could information theory provide an ecological theory of sensory processing? Network: Computation in neural systems , 3(2):213-251, 1992.

Tom Baden, Philipp Berens, Katrin Franke, Miroslav Román Rosón, Matthias Bethge, and Thomas Euler. The functional diversity of retinal ganglion cells in the mouse. Nature , 529(7586):345-350, 2016.

Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. 291 Sensory communication , 1(01):217-233, 1961. 292

Michael Beyeler, Emily L. Rounds, Kristofor D. Carlson, Nikil Dutt, and Jeffrey L. Krichmar. 293 Neural correlates of sparse coding and dimensionality reduction. PLOS Computational Biology , 294 15(6):e1006908, jun 2019. 295

John Canny. A computational approach to edge detection. IEEE Transactions on pattern 296 analysis and machine intelligence , (6):679-698, 1986. 297

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large298 scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern 299 recognition , pages 248-255. Ieee, 2009. 300

James J DiCarlo, Davide Zoccolan, and Nicole C Rust. How does the brain solve visual object 301 recognition? Neuron , 73(3):415-434, 2012. 302

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, 303 Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 304 An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 305 arXiv:2010.11929 , 2020. 306

David J. Field. What Is the Goal of Sensory Coding? Neural Computation , 6(4):559-601, jul 307 1994. 308

Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables. arXiv preprint 309 arXiv:1604.06737 , 2016. 310

Torkel Hafting, Marianne Fyhn, Sturla Molden, May Britt Moser, and Edvard I. Moser. Mi311

crostructure of a spatial map in the entorhinal cortex. Nature 2005 436:7052 , 436(7052):801312

806, jun 2005. 313

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image 314 recognition. arxiv 2015. arXiv preprint arXiv:1512.03385 , 14, 2015. 315

G. E. Hinton and R. R. Salakhutdinov. Reducing the Dimensionality of Data with Neural 316 Networks. Science , 313(5786):504-507, 2006. 317

Simon Laughlin. A Simple Coding Procedure Enhances a Neuron's Information Capacity. 318 Zeitschrift für Naturforschung C , 36(9-10):910-912, oct 1981. 319

Minh Le and Subhradeep Kayal. Revisiting edge detection in convolutional neural networks. In 320 2021 International Joint Conference on Neural Networks (IJCNN) , pages 1-9. IEEE, 2021. 321

Ann B Lee, Kim S Pedersen, and David Mumford. The Nonlinear Statistics of High-Contrast 322 Patches in Natural Images. International Journal of Computer Vision , 54(5413):83-103, 2003. 323

William R Levick. Receptive fields and trigger features of ganglion cells in the visual streak of 324 the rabbit's retina. The Journal of physiology , 188(3):285, 1967. 325

Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, 326 Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common 327 objects in context, 2014. 328

Ralph Linsker. Self-organization in a perceptual network. Computer , 21(3):105-117, 1988. 329

David R Martin, Charless C Fowlkes, and Jitendra Malik. Learning to detect natural image 330 boundaries using local brightness, color, and texture cues. IEEE transactions on pattern analysis 331 and machine intelligence , 26(5):530-549, 2004. 332

Richard H. Masland. The Neuronal Organization of the Retina, oct 2012. 333

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word 334 representations in vector space. arXiv preprint arXiv:1301.3781 , 2013. 335

Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the Number of 336 Linear Regions of Deep Neural Networks. Advances in Neural Information Processing Systems , 337 4(January):2924-2932, feb 2014. 338

Jean Pierre Nadal and Nestor Parga. Nonlinear neurons in the low-noise limit: a factorial code 339 maximizes information transfer. http://dx.doi.org/10.1088/0954-898X_5_4_008 , 5(4):565-581, 340 2009. 341

Benjamin Odermatt, Anton Nikolaev, and Leon Lagnado. Encoding of Luminance and Contrast 342 by Linear and Nonlinear Synapses in the Retina. Neuron , 73:758-773, 2012. 343

Bruno A Olshausen. sparse codes and spikes. Probabilistic models of the brain , page 257, 2002. 344

Bruno A Olshausen and David J Field. Sparse coding of sensory inputs. Current opinion in 345 neurobiology , 14(4):481-487, 2004. 346

Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by 347 learning a sparse code for natural images. Nature , oct 2015. 348

Marc'Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann Cun. Efficient learning of 349 sparse representations with an energy-based model. Advances in neural information processing 350 systems , 19, 2006. 351

Joshua R. Sanes and S. Lawrence Zipursky. Design Principles of Insect and Vertebrate Visual 352 Systems, 2010. 353

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale 354 image recognition. arXiv preprint arXiv:1409.1556 , 2014. 355

JH van Hateren. A theory of maximizing sensory information. Biol. Cybern , 68:23-29, 1992. 356

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This study proposes a new framework to combine neural coding concepts of information transmission and probability density modeling.\n- This framework is based on an even code principle where the output response density strives to be even, given some arbitrary input density.\n- The authors show that this coding principle produces sensible bases for low-dimensional inputs, and orientation-tuned filters for natural image patches.\n- While conceptually straightforward, it is unclear to me whether this study provides unique insight into sensory coding in neural populations.\n\nUPDATE: Sep 1, 2023. I have read the rebuttal, and maintain my score (see details below).\n\nSTRENGTHS:\n- The study is clearly presented.\n- The concepts of max/min entropy on the output and input densities are conceptually simple to follow.\n- The numerical experiments are reasonable.\n\nWEAKNESSES:\n- This paper begins with what seems like a false dichotomy of information transmission vs. sensory probability density modeling. Indeed, from a pragmatic point of view, how can one guarantee optimized information transmission without having a good density model of the signal to be transmitted? There exists literature in this area (see questions section), and the motivation/framing of this present study is concerning.\n- There is a bit of a conceptual leap from 1 or 2 pixels to full image patches, with additional complexity and machinery introduced. The described rationale seems reasonable enough, but it is unclear whether the two-pixel orthogonal case can provide adequate intuition for the multi-dim case. Would a 2D non-orthogonal example be illuminating at all?\n- Unclear to me whether these results, which rely on binary coding provides theoretical insight for real neural coding. Spikes are inherently binary, yes, but typically spike counts/rate are what is considered the informative variable in neural coding.\n\nQUESTIONS:\n- I'm surprised there's no mention of the work by Ganguli and Simoncelli (\"Efficient sensory encoding and Bayesian inference with heterogeneous neural populations\", Neural Comp. 2014), which directly integrates density modeling with information transmission in neural populations. These authors showed that synaptic weighting functions and neural population responses are arrange so as to implicitly encode the probability of the sensory signal. There have also been attempts at generalizing these concepts to multi-dimensional stimuli by Yerxa et al (\"Efficient sensory coding of multidimensional stimuli\"; PLOS Comp Biol. 2020). At the very least I believe the present study should discuss how their approach fits in with this existing literature. \n- Minor point: is \"even code\" a standard information theory term? If it is then that's fine, but it seems like equalized probability would be a more descriptive term.\n\nLIMITATIONS:\nThere was no discussion of limitations. Unclear to me what the drawbacks are of this approach compared to existing literature.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for the representation of elementary natural images, based on the observation that classical studies in computational neuroscience focus mainly on methods to improve code efficiency, but that this could be complemented by a study of probability density modeling between neighboring pixels to improve image representation. This work consists in studying a coding principle based on a probabilistic representation and its formalization in a form of variational optimization. The paper presents the elementary method for a single pixel, then extends it to two pixels, and applies it to  small images extracted from natural images. This method is enhanced by a heuristic that allows  to formulate a cost function and thus derive an optimization algorithm. The results allow  to numerically validate this principle by deducing output statistics, as well as the emergence of local contour detectors.\n\nSTRENGTHS:\nA major strength of this paper is that it derives the image representation algorithm from fundamental principles of machine learning, particularly probabilistic representations. In this way, it rigorously defines the problem of establishing dependencies between the luminance values of neighboring pixels.\n\nWEAKNESSES:\nThe first limitation of this paper is that it applies to very elementary signals, i.e. a pixel, a pair of pixels, or small images of dots. As the initial aim of the paper is to understand the computational functioning of the biological networks that underlie the efficiency of vision, this approach is extremely caricatural, and dismisses many fundamental aspects, such as the largely parallel processing of large images, the use of large neural networks, or the ability to process multimodal images, in color or in motion, or more generally hierarchical processing that can be forward, but also modulated by feedback signals. Finally, the results that have been obtained, for example for the detection of local elementary contours, are difficult to interpret quantitatively and seem very preliminary.\n\nQUESTIONS:\nI have a number of questions arising from reading the paper.\n\nFirst of all, it seems that the principle of optimization, and in particular its derivation for elementary changes in probability, is widely known in the literature. Can you highlight the originality of your approach compared with other probabilistic optimization methods, in particular all variational optimization problems? Figure 1 shows a dependency between pixel values. What is the relationship between your study and studies that have been carried out for many years, for example by Eero Simoncelli on divisive normalization? Also, the result shown in figure 2 shows an adaptation of the code according to the probability density. Are these results compatible with the homeostasis phenomena highlighted in biological neural systems? Finally, figure 5 shows the emergence of a representation of contours in an image, reminiscent of that obtained in studies of sparse coding, for example via Bruno Olshausen's framework. What is the relationship between your principle and these algorithms?\n\nLIMITATIONS:\nFinally, these questions about the paper reveal the main limitations of this work.\n\nIn particular, the introduction to the paper presents at length principles that seem very general, such as Shannon entropy, and the rest of the paper does not sufficiently highlight the novelties that are brought forward. This brings to light a main limitation of the paper, which is the fact that the propositions that are put forward are very ambitious, but the results are applied to very limited situations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the relationship between the information theory approach and the probabilistic generative model approach in the context of understanding neural coding. The author suggests that maximizing the information-carrying capacity of output channels and modeling the input probability distribution can be pursued as independent dual objectives. To investigate this hypothesis, the author begins by examining a one-pixel system, followed by a two-pixel system, gradually progressing to 2D image patches. The resulting codes obtained for the images exhibit similarities to edge detectors and orientation-selective neurons in V1, akin to many efficient coding models developed over the past two decades.\n\nSTRENGTHS:\nThe presentation is reasonably clear.  It is rather interesting that the author begins by examining a one-pixel system, followed by a two-pixel system, gradually progressing to 2D image patches.\n\nWEAKNESSES:\nWhile the idea that both information transmission and probabilistic modeling of the images should be taken into consideration simultaneously might be new,  and is sufficient to learn edge detectors and orientation-selective neurons, the author has not established it is a necessary condition. In fact, literature in the last thirty years (from Law and Cooper's to Olshausen and Field and many others)  that such codes can be learned based on either one of the criteria. \n\nIt is surprising that the V1 neural codes were assumed to be sparse binary codes. What is the evidence?  The distribution of output values as shown in Figure 2a has not been observed biologically.  This brings the Even Code hypothesis into serious question.\n\nQUESTIONS:\nWhat is the evidence for  sparse binary codes?  The distribution of output values as shown in Figure 2a has not been observed biologically.  This brings the Even Code hypothesis into serious question.\n\nLIMITATIONS:\nSocietal impact not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors studies simple of neural encoding. The question is whether two\ndistinct goals, accurate transmission of information and learning the\ndistribution of environmental stimuli can be achieved simultaneously. The\nauthors argue that yes, it can, using the key assumption of a uniformly\npartitioned input space. The coding principle of the authors is finally applied\nto image patches, where it yields edge-like features.\n\nSTRENGTHS:\n- The author studies an important question, namely simple coding schemes that\n  reproduce filters that resemble those of deep convolutional networks or parts\n  of the visual system.\n- The author develops intuitions in simple toy models before moving to\n  applications on real images.\n- The filters shown in Fig. 3 bear a striking resemblance to the filters of a\n  trained VGG model (although I have some questions on the methodology, see\n  below)\n\nWEAKNESSES:\nI found the article confusing to read in a few places. For example, early on,\nthe author states that \"maximising the rate of transmission\" is equivalent to\nmaximising the entropy of the output distribution $H_Q$. I would think that what\nyou transmission of information requires maximising the mutual information $I(X;\nQ)$ between the distribution over inputs and outputs. (around eqs 1 + 2; note\nthat the notation is rather confusing here, using lower-case $p$ for the\ndistribution over input stimuli $x$, and capital $Q$ for the distribution over\noutput states $y$). Why are you maximising simply the entropy of the\ndistribution over outputs?\n\nSimilarly, in the section on the even code principle, I'm confused by the\nquestion of how the IPU models the input distribution. The way I read Sec. 2,\nthe IPU is considered a function of the stimuli $y=f(x)$ -- in that sense, it\ndoesn't model the input distribution, we cannot sample from it. It can give a\nmore or less faithful representation of $x$, as measured for example by mutual\ninformation if the mapping is probabilistic, \n\nAs you then move on to learn two pixel distributions, I'm confused about your\nuse of MLPs. MLPs are powerful neural networks, but you seem to use them to\n\"learn\" to partition the input space into equal partitions - is this not\npossible by just writing down a simpler model?\n\nGiven my trouble understanding the first few sections, I cannot competently\ncomment on the experimental results - while the filters obtained by the authors\ndo bear a striking resemblance to the filters of a VGG network, I don't really\nunderstand how the author obtained them. Some additional clarifications would\ntherefore be more than welcome.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}