A Computational Framework for Modeling Emergence of Color Vision in the Human Brain
Authors: Atsunobu Kotani
Ren Ng
Venue: ICLR 2025 Oral (2025)
Keywords: color vision
computational neuroscience
retina simulation
cortical learning
self-supervised learning
color blindness
Primary area: applications to neuroscience & cognitive science
TL;DR: This paper introduces a novel computational framework for modeling the emergence of human color vision by simulating the eye and the cortex.

Abstract: It is a mystery how the brain decodes color vision purely from the optic nerve signals it receives, with a core inferential challenge being how it disentangles internal perception with the correct color dimensionality from the unknown encoding properties of the eye. 
In this paper, we introduce a computational framework for modeling this emergence of human color vision by simulating both the eye and the cortex. Existing research often overlooks how the cortex develops color vision or represents color space internally, assuming that the color dimensionality is known a priori; however, we argue that the visual cortex has the capability and the challenge of inferring the color dimensionality purely from fluctuations in the optic nerve signals. To validate our theory, we introduce a simulation engine for biological eyes based on established vision science and generate optic nerve signals resulting from looking at natural images. Further, we propose a bio-plausible model of cortical learning based on self-supervised prediction of optic nerve signal fluctuations under natural eye motions. We show that this model naturally learns to generate color vision by disentangling retinal invariants from the sensory signals. When the retina contains $N$ types of color photoreceptors, our simulation shows that $N$-dimensional color vision naturally emerges, verified through formal colorimetry. Using this framework, we also present the first simulation work that successfully boosts the color dimensionality, as observed in gene therapy on squirrel monkeys, and demonstrates the possibility of enhancing human color vision from 3D to 4D.

Published as a conference paper at ICLR 2025
A COMPUTATIONAL FRAMEWORK FOR MODELING EMERGENCE OF COLOR VISION IN THE HUMAN BRAIN
Atsunobu Kotani & Ren Ng
Department of Electrical Engineering and Computer Sciences University of California, Berkeley (akotani, ren) @berkeley. edu
ABSTRACT
It is a mystery how the brain decodes color vision purely from the optic nerve signals receives, with a core inferential challenge being how disentangles inter- nal perception with the correct color dimensionality from the unknown encoding properties of the eye. In this paper, we introduce a computational framework for modeling this emergence of human color vision by simulating both the eye and the cortex. Existing research often overlooks how the cortex develops color vision or represents color space internally, assuming that the color dimensionality is known a priori; however, we argue that the visual cortex has the capability and the challenge of inferring the color dimensionality purely from fluctuations in the optic nerve signals. To validate our theory, we introduce a simulation engine for biological eyes based on established vision science and generate optic nerve signals resulting from looking at natural images. Further, we propose a bio-plausible model of corti- cal learning based on self-supervised prediction of optic nerve signal fluctuations under natural eye motions. We show that this model naturally learns to generate color vision by disentangling retinal invariants from the sensory signals. When the retina contains N types of color photoreceptors. our simulation shows that N di-dieenioonal color vision naturally emerges, verified through formal colorimetry. Using this framework, we also present the first simulation work that successfully boosts the color dimensionality, as observed in gene therapy on squirrel monkeys, and demonstrates the possibility of enhancing human color vision from 3D to 4D.
INTRODUCTION
"Color is the place where our brain and the universe meet." Paul Klee
We experience colors in everyday life so effortlessly that it is easy to take the underlying neural computations for granted. In fact, the sensory signals exiting our eye, called optic nerve signals (ONS), are nothing like our color vision (see Fig. and Supplementary Video 0:30 For example, ONS are spatially warped, akin to image taken with a fish-eye lens, due varying densities of photoreceptor cells in the retina (Curcio et al. 1990). ONS does not come in color either colors of the scene are spectrally sampled by different types of color sensitive cells (cone cells) in the retina, appearing as a layer of spatial noise in the ONS. Furthermore, other processes, such as lateral inhibition and action potentials, render the image structure barely recognizable, in gradient domain where spatiotemporal "edges" dominate. Now, the question is: how are we still seeing colors?
Specifically, this paper introduces novel computational framework for modeling the emergence of human color vision by simulating the eye and the cortex. For the eye, we present biophysically accurate implementation of textbook scientific model of the retinal neural circuitry. For the brain, we hypothesize low-level, self-supervised learning mechanism in the cortex that operates purely on the optic nerve signal stream. For color representation in the brain, we propose modeling color the brain as a nigh-dimensional vector, rather than assuming any specific color dimensionality and show that the correct color dimensionality emerges naturally through the proposed learning.
For eye simulation. goal is to create computational engine that takes in any spectral image of the world and outputs the corresponding optic nerve signal stream, based on established vision science.
Code, video, dataset, and tutorials available our project website: https://maaisse..ces..rekeeee

---

Published as a conference paper at ICLR 2025
Scene Stimulus (typerspectral)
Optic Nerve Signals (ONS)
Decoded Color Vision
Quantitaive CMF-SIM
3D
Simulation
Simulation Cortical
Analysis of Emergent Neural Color
Engine
Qualitative NS
Biological
Eyes
Learning
N-dim
400-700mm
Section
Section 4
Section
Input Imagery
Recina Model
Optic Nerve Signal
Cortical Learning
Emergeent Vision
Analysis of Viiisnn
CMF-SIM, NS
Database of
Textbook vision
Biologcal realism
Core typothesis learning by setf.
Emergent color vision in cortex
Emergent color
CMF-SMM SMMF-IMM (Color
hyperspectral
science model of
Markeed difference
vision analyzed
Masching Function
photographs
foveal optic
from color vision
supervied prediction
Cortical color represeritd as high dim vector Brain must infer the correct color
rigorously black-bow, to
Sim.) quantfies
Brain never receives this
signal generation
Eye motoon, scalar,
dimension
Retina celle cones,
spacialy warpact
ONS flucturtions under natural eye motion
aralyang another human's vision
(Neural-Scope)
ground cruth
horzoonzal cells,
parchy spectral
viaualizes 12" color
dara, only optic nerve signals
tipolar, ganglion
sampling lateral
Rigor of Maxwwells color matching
by linear projection
cells (apric
inhibition, spikin
Bic-plausible
dimensionality
RGB colorspace
Figure I: Overview of our proposed framework for modeling the emergence of human color vision. Our simulation engine of biological eyes converts scene stimulus (hyperspectral image) stream of optic nerve signals (Section 3 & Video 0:30). We simulate cortical learning purely from these optic nerve signals (Section 4 and show the emergence of color vision. We show how to analyze the emergent neural color quantitatively with Color Matching Function test Simulator (CMF-SIM) and qualitatively with Neural Scope (NS) (Section 5
Our model captures how scene form, color, and motion become spatiotemporally entangled in the optic nerve signal stream. Our simulation is based on the "textbook" model of vision science (Rodieck) 1998) for midget, private-line visual pathways, detailed in Section and comprising fixational eye motion, spectral encoding by cone cells, foveation, and lateral inhibition.
For brain simulation, we hypothesize that the cortex could disentangle the optic nerve signal from the invariant retinal properties to generate color vision - purely through self-supervised learning process that aims to predict the constant fluctuations in cellular-level activations of the optic nerve signal during small eye movements. The neural conditions for such self eeff-supervised learning are biologically plausible in the sense that the cortex continuously receives optic nerve signals under the tiny gaze movements of fixational eye drift. The theoretical intuition for why such learning might succeed is that eye motion repeatedly draws a static scene image across the invariant spectral and spatial sampling properties of the retina, potentially enabling the retinal properties and scene images to be mutually filtered out of the optic nerve signal where they are entangled, We show that this simple learning mechanism succeeds at discovering color vision with the correct dimensionality.
But first, a somewhat esoteric yet technically critical feature of the modeling framework needs discussion: color representation in the brain. Existing research often overlooks how the cortex develops color vision or represents color space internally, assuming that the color dimensionality is known a priori, e.g. RGB. We argue that the cortex has the capability and the task of inferring color dimensionality, purely from fluctuations in the optic nerve signals. Therefore, we propose representing color in the cortex space and color dimensionality emerge naturally as geometric properties of the hypothesized learning mechanism. We show how to formally quantify and visualize the emergent color space (Section (5).
nigh-dimensional vector in R and find that the correct color
Remarkably, the hypothesized learning mechanism results in clear color vision of the correct color dimensionality. When the retina contains 1, 2, 3 or 4 types of color photoreceptors (cones), correct color dimensionality emerges: respectively, 1D mono-, 2D di-, 3D tri- or 4D tetra-chromatic vision. In fact, is an esoteric but well-known fact vision science that the three types 2D dichromacy result in highly specific color-spaces (bluc-yellow, bluc-orange, and teal-pink), which naturally, Even more, the simulation presents a model of how color dimensionality boosting occurred in the squirrel monkeys (Mancuso al. 2009). We simulate injection of the third cone pigment virus into the dichromat retina, which results in boosting from dichromacy to trichromacy. Intriguingly, the model also shows the possibility that normal human trichromacy could be boosted to tetrachromacy.
emerge
In sum, our proposed framework formulates the emergence human color vision in a computational manner. The simulation engine of realistic optic nerve signals generates training data to this problem,
2

---

Published as a conference paper at ICLR 2025
and intentional and challenging constraint is that the cortical model must strietly learn only from the optic nerve signal with no auxiliary information. This paper presents the first, simple and yet complete existence proof that such cortical inference is possible, and we show that this learning simulation is consistent with various surprising and unexplained vision science phenomena.
RELATED WORK
2.1 VISION SCIENCE ON OPTIC NERVE SIGNAL ENCODING
In the "textbook" model, the retina transforms light into electrical signals through three primary functions: color sampling via cone cell spectral response functions, lateral inhibition via horizontal neural connections, and spatial sampling via cell positioning (Fig. 2 Under daylight, this process begins when light activates cone cells, which are of three types for most humans (Young) T801]
1802 Von Helmholtz (867). ) each sensitive to different wavelengths (Stockman & Sharpe, 2000).
These signals are then modulated by horizontal cells that enhance visual contrast through lateral inhibition (Rodieck 1965 Dacey et al. 1996. Verweij et al. 2003). The signals continue to bipolar and retinal ganglion cells (RGCs), with direct connection the fovea via the midget private-ling line pathway (Dowling & Boycott 1966; McMahon et al.] 2000; Wool et al.) 2018). vital for high- resolution color vision. Notably, the cone cell density varies, reaching its peak the fovea (Osterberg)
1935 Curcio et al.) 1990). The axons of the ganglion cells bundle to form optic nerve signals, maintaining their spatial arrangement, which results in retinotopy in the visual cortex (Holmes) 1918: Tootell et al. 1982 Dougherty et al., (203). Fixational eye movements cause dynamic photoreceptor activation at all times by constantly shifting the gaze, even when focusing on static
objects (Rucci & Victor 2015 Young & Smithson 2021, Martinez-Conde et al. 2013
Recordings of real optic nerve signals exist, but cannot be used for our cortical simulations because they are orders of magnitude too low resolution (from only a few thousand cells) and only in response
to grayscale rather than color imagery (Litke et al. [2004] Brackbill et al. 2020 Marre al. 2017]
Liu et al. 2022). To overcome these data limitations, we present a simulation engine for the midget private-line pathway in the fovea in Section 3
2.2 COLOR REPRESENTATION IN COMPUTATIONAL NEUROSCIENCE
It is an open question how to meaningfully model neural representations of color in simulations of visual perception. Most computational neuroscience sidesteps this issue, "hard-coding" a dimensional- ity of 3, representing and constraining cortical color to tristimulus values, such as RGB (Parthasarathy
jtt al. 2017: Botella-Soler et al.,
2018 Brackbill et al. 2020 Kim et al. 2021 Zhang et al. 2022
Wu et al. 2022). LMS (Young) 1802; Von Helmholtz, 1867) or oppenopponent space (Derrington
et al. 1984; Macl & Boynton 1979). Constraining cortical models to such a ceiling of three for
dimensionality clearly conflicts with the study of functional tetrachromat observer with 4D color vision (Tordan et al., (2100) Rezeanu et al. (00TT. Instead, we model the brain with the capability and the challenge of deducing the inherent color dimensionality encoded in the optic nerve signals.
2.3 THEORY ON HOW VISION EMERGES IN THE BRAIN
In computational vision science modeling, it is often overlooked that the cortex relies solely on a stream of optic nerve signals to discover color vision, with no access a teacher signal or perceptual ground truth. Rather, various efforts have been made to reconstruct visual stimuli from neural
responses by giving the cortical model access to ground truth stimulus image (Naselaris et al. 2009
Nishimoto et al. 2011; Paattassartty et al., 2017 Botella-Soler et al.] 2018) Brackbill et al. 2020
Kim et al. (021), ). but the neural reality that the brain never has direct access the visual scene. This
characteristic makes the human visual system quintessential example of self-supervised learning.
which is a growing field in computer vision (De Sa 1993, Chen et al. 2020 He et al., 2022). In this work, we propose a learning principle in the cortex which aims to predict the cellular-level fluctuations in activation that occur during small eye movements, which is associated with the idea of temporal prediction (Palmer et al. [2015] Lotter et al. 2016,; Singer al. 2018 2023b), as well the
broader concept of predictive coding (Rao & Ballard 1999, Srinivasan et al. 1982). It is also closely
linked to the sensorimotor contingency theory from cognitive science (0 Regan & Noë 2001) and
3

---

Published as a conference paper at ICLR 2025
slow feature analysis (Hinton 1990: Földiák 1991) which suggests that the brain learns to anticipate the sensory outcomes of motor actions (e.g. eye movements) and filters out invariant (slow) features.
2.4 THEORY ON How BRAINS INFER COLOR DIMENSIONALITY
Previous studies has investigated inference of invariant retinal properties from sensory signals. For instance, research has demonstrated the ability to deduce cone cell types (Wachtler et al., |007]
Brainard et al. 2008 Benson et al. 2014) and the positions of photoreceptors (Maloney & Ahumada
1989, Brainard et al.
2008) via statistical methods from sensory signals. [Brainard al. (2008) hinted at analyzing sensory inputs from different time points during fixational drift as a way to reveal retinal features, and in this paper we provide computational realization of this idea with a specific learning mechanism that achieves complete disentanglement of retinal invariants from optic nerve signals.
2.5 MEASUREMENT OF COLOR DIMENSIONALITY IN HUMAN COLOR PERCEPTION
In this paper, we need to rigorously measure the color dimensionality of the emergent color from cortical simulation. To do so, we adapt Maxwell's famous color-matching experiments (Maxwell 1856), which laid the foundation of colorimetry that remains at the heart of all color reproduction technology today. Color matching experiments confirmed the trichromatic theory (Young 1802]
Von Helmholtz 1867 Grassmann 1853) in which the 3-dimensional nature of human color vision has its basis in the three different cone types in the human retina. Jacobs's 's recent review (Jacobs) 2018) of color dimensionality in animal vision, however, reminds the community that the dimensionality of color vision does not automatically equal the number of cone types, and that the most rigorous way measure it remains Maxwellian color matching - as we follow in this work.
2.6 COMPLEXITY OF HUMAN COLOR VISION
This paper focuses on color dimensionality because it is the foundational characteristic of an ob- server's color experience, but many layers of further perceptual complexity exist atop that foundation. Examples include chromatic adaptation (color constaney) (Von Kries
1902.
Land 1977 Foster
2011), perceptual nonuniformity across colorspace (CIE) 1976; MacAdam 1942). complex contextual
interactions (Fairchild 2013), and even surprising flood-fill features (Pinna 1987 Pinna et al. 2001).
Parts of these perceptual phenomena are scientifically mapped to neural correlates, such nonlinear
photoreceptor responses (Krauskopf& & Karl 1992 Angueyya et al. (022), or parvocellular pathway
and neural processes spanning V1, V2, and V4 (Livingstone & Hubell 1987 Zeki 1978 Li et al.
2014: Liu et al., (2000) (Angueyra et al. (022). This paper leaves these additional layers of perceptual complexity as future modeling work.
SIMULATION ENGINE FOR BIOLOGICAL EYES AND OPTIC NERVE SIGNALS
We model the primary functions of the human retina based on the known science. The inputs to our retina model are hyperspectral images A: details in Appendix A. A model of fixational eye drift (Rucci & Victor 2015) Young & Smithson 2021 Martinez-Conde et al.) 2013) generates a sequence of frames that sample different parts of the image. Each frame is projected on the retina, stimulating a randomized array of cone cells, according to known cell density variation as a function of eccentricity (distance from the fovea) (Curcio et al.) 1990) and known statistics of different cone cell types (Carroll et al., 2000) (Fig. B). Each cone cell converts the scene light into photoreceptor activations based on cone type spectral response functions (Stockman et al. 1999 Stockman &
Shharpe| 2000) (Fig. C), followed by lateral inhibition from horizontal cells (Wool et al.) 2018
Rodieck 1965) (Fig. D). These signals are transformed into spike trains (Fig. 2 F), then bundled
into optic nerve signals (ONS), resulting in spatial distortion (Fig F), turning color scene stimulus into a noisy, spatially distorted ONS. Additional details are in Appendix A and Video 1:35.
One observation to make is that there are three invariant properties of the simulated retina (i.e. retinal invariants): cell positions, cell types, and horizontal cell connections. These properties are held constant during generation of ONS for particular eye, but we use the engine with different values for these properties to generate ONS datasets for a diverse set of eyes. For example, we create datasets with different cone types, including monochromatic (I., M, S), dichromatic (L.M, L.S.
4

---

Published as a conference paper at ICLR 2025
SiSmullttion Engine of Biokogical Eyes
2. Engine Input Paramecers
Scene Stimulus
Eye motion simulator
Example Redina
Center
Incoming Spectra
Received Light
Gone Cell
Bipolar Signals
RGC Spikes
Optic Nerve Signals
OF EXTI
Activations
Scene image datasec
Cone cell types
Cone cell positions
Motion
Spazial Sampling
Spectral Sampling
Lazeral Inhibition
Spike Conversion
Paciing/
Cone spectral responses
Bundling
typerspactral
Lateral inhibition lerrnel
RRGC: Retinal Ganglon Cels
Example Output Our Simulation Engine
Varying Input "Cone cell types" Simulation Engine
Incomiing spoctra
Recehed
light
C
actiaaions
Cone cal
Number cone call types
(M only)
(MS)
(LMS)
(LMQS)
GM
Time
Time
Time
Cone cell activations
Bepohr
E
RGC firige
Bipolar signalss
Time
Figure Overview of our simulation engine of biological eyes. 1. This engine takes scene stimulus as input, processes it through "textbook" model of eye motion and retinal neural circuitry, to generate stream of optic nerve signals. 2. This engine accepts custom eye and retina parameters. It allows visualization of neural signals in steps (A-F), illustrating the progressive entanglement of scene imagery with retinal properties. 4. Visualization of changing one the input parameters, number of cone cell types showing that the signals become noisier the number increases.
MS), trichromatic (LMS) and tetrachromatic (LMSQ) configurations (Fig. 4).. The generated ONS becomes spatially noisier as the number of cone types increases, and we study how the cortex infers the inherent color dimensionality of these eyes purely from the differences in their respective ONS.
SIMULATING CORTICAL LEARNING AND EMERGENCE OF COLOR VISION
Our cortical model is structured to learn three functions in pipeline (Fig. 1): 1. d that decodes the optic nerve signal at time i into its internal percept; 2. f2 that translates this percept according to eye motions inferred from the signal over short time interval, dt; and 3. V that re-encodes the translated percept back into predicted optic nerve signal at time t + dt, which is compared against the real signal received that time. Mathematically, given the optic nerve signal O, at time
Ô (S()(()))
where Óctat the predicted ONS time ++ dt. Here the task of decoder analogous to inverting the retinal processes to transform O, into the visual percept image V, (i.e. V, + ((O)). Likewise, the re-encoder function resembles the retinal processes, as it aims to reproduce an optic nerve signal from the visual percept (i.e. O (VV)). Therefore, & and v are pseudo-inverses.
The learning objective is to minimize the prediction error Eprediction the difference between predicted and real optic nerve images at time t dt,, such that:
Fsprediction 10 Ô l1 ||O ((((
where Ot+dt is the real observed ONS at time t + dt.
Three main ideas drove the evolution of our selected model features. First, reasoned that it would be a big step towards successful decoding and re-encoding if the cortex could infer the key encoding properties of each cone cell. We gave the cortical model sets of learnalee parameters, which

---

Published as a conference paper at ICLR 2025
Hypothesized Cortical Learning Mechanism
Example Measurement of Color Dimensionality
True ONS at dime
Initial state
700 steps
1,200 steps
100K steps
1, V are learnable functions
Internal Percept
CMF- SIM
ID
ID
2D
3D
Decade
01234
01234
0 2323
N-dim
ONS at
Predicted
Neural Buckets
Translate
a Ar
Ω
NS
RexEncode
N-dim
:
Prediction Error
linearly projects N-dim RGB color space
Exammle Leaming Evolution Trichromar ONS Stream
Inferred Retinal Invariants with Neural Buckers
Inicial state
700 steps
1,200 steps
100K steps
Log ONS prediction error (a.u)
Positions
P
700 1,200
Cone cypes
Ground-Truch
Initial
700 steps
1,200 steps
100K steps
High
Ul weights
W
Error
+ O
Figure 3: Overview of our hypothesized cortical learning mechanism and exclusive study of the learning behavior of the cortical model with trichromatic retina. Given the stream of optic nerve signals as the only input data, the cortical model aims to predict the next ONS from the current one with learnable functions, decoder ④ translation operator and re-encoder error decreases learning progresses, converging after 100K learning steps. 3. During learning, the color dimensionality the internal percepts transition from ID, 2D to 3D, formally measured by CMF-SIA -SM and visualized by NS (Fig. 4 & Section 5. 4. The cortical model infers the retinal invariant properties during learning: cell positions P (higher density in fovea), cone cell types C, and lateral inhibition weights W (center-surround receptive field).
Prediction
call "neural buckets", in which buckets contain the following information for each cone cell: 2D position in visual space (P), cone spectral identity (C), and lateral inhibition weights (W) to neighboring cones.
and update guesses these properties during learning. The
The second main idea was the observation that decoder & and encoder V are mathematically factorizable into a pipeline of subfunctions, such that:
Φ dp 2c Pw
V Vw Ve Vp-
Each sub-function is an operator conditioned on its corresponding neural bucket, cell positions P. cone spectral types C, and lateral inhibition weights to neighboring cells W. In case of decoder é it first executes inversion of lateral inhibition using 4w in order to estimate the activations of each photoreceptor associated with an optic nerve axon; second, projects scalar cone activations into R by Pe using inferred spectral identities in C' (and interpolating color across space the third main idea below); and finally inverting the spatial distortion of foveation via &p. The re-encoding function is a pipeline of analogous subfunctions in reverse order: re-applying spatial warping with Vp: re-projecting color into scalar photoreceptor activations with Ve; and re-applying lateral inhibition with W. Further implementation details provided in Appendix B
The third main idea was that, order to accurately re-encode after translation, the model needs to learn to interpolate color information spatially, because there is only one cone type at each point on the retina. This need to interpolate analogous to the situation cameras with image sensors (Bayer) 1976; Fossum 1997 Kimmel 1999) that physically sample only one of the R.G.B channels each pixel, and fundamentally require demosaicking algorithms to interpolate full R.G.B values at all pixels. The required interpolation function in the cortical model complex because the spectral

---

Published as a conference paper at ICLR 2025
1. CMF-SIM: Quantitative Color Matching Function Test Simulator
NS: Qualitative Neural Scope
Coefficients
Eye model
Cortical model
Cortical model
on primaries
Color percept
RGB-prajected Visual Percept
(Knobs)
Visual Percept
Primary lights
Match
Decode
Decode
S
difference
Neural
Nidim
Scope
Color percept
Target light
Decade
Linear transform from R to R²
Target color
NS learnable macrix.
Update the coefficients to minimize the error until convergence
Perceptual
Optimized visualization parameter,
indebendent of the main cortical loop.
Example Output: Full CME-SIM Result the Cortical Model Trained with Trichromat Retina
Il primaries
0
2
3
Summary
Coefficients
N/A
Area under the curve f of primaries
I 2
convergance
Perceptual error convergence
O area the curve
area under the curve
Error first goes zero with primaries. measurement:
vision.
Figure 4: Overview of our measurement methods of emergent color dimensionality, Color Matching Function Test Simulator (CMF-SIM) SIM) Neural Scope (NS). CMF-SIM determines the mini- mum number of primary colors needed to match any target color by iteratively updating coefficients minimize perceptual error. NS visualizes visual percepts independently of the cortical learning loop, optimized learnable N x 3 matrix to minimize projection error to the target RGB image. 3. Example CMF-SIM output for a trichromat retina-trained cortical model shows matching errors for 400-700nm spectral light converging zero with three primaries, confirming 3D color vision.
sampling pattern is random, so learning this function is entangled with correctly resolving C. We enable and force the cortical model to learn the color interpolation function by representing as convolutional neural network with neural bucket parameters D (Appendix B.2.2)
important detail the handling of eye motion the simulation. In one experiment we show that the model can learn subfunction that estimates the eye motion translation between times and t + dt purely from the optic nerve signals at those times. The main computational challenge here is the spatial warp in optic nerve signals. The uniform translation in stimulus space corresponds to non-uniform translation the optic nerve signal space, which makes the prediction of the eye motion dependent the inference of cell positions P. We find that the cortical model iteratively updates the neural bucket from imperfect initial eye motion estimate, which helps to improve the prediction of eye motion, and vice versa. This inference converges the correct eye motion estimate, the model learns to minimize prediction eтror (Appendix B.3
With this pipeline of subfunctions and associated neural buckets, the hypothesized learning is equivalent to parallel numerical optimization of all neural buckets in striving to minimize prediction error. We simulate learning using stochastic gradient descent (Kingma & Ba, 2014).
NEURAL REPRESENTATION OF COLOR SPACE AND ANALYSIS OF EMERGENT COLOR DIMENSIONALITY
We model color in the cortex as a vector in high-dimensional space, R This decision represents our view that the brain has both the freedom and the challenge of somehow inferring the intrinsic color dimensionality of the visual signals it is receiving along the optic nerve. Specifically, we define a cortical decoding function that takes the optic nerve signal O, at time and transforms it into its internal visual percept V, (Fig. 3). such that:
V, (O.)
where each pixel in V; is N N-dimensional vector. Inside R assume that cortical color space emerges as a K-diimensional manifold. To measure this intrinsic dimensionality K, we introduce two methods: formal, numerical CMF-STM; and intuitive, visual NS.
7

---

Published as a conference paper at ICLR 2025
CMF-SIM (Color Matching Function Test Simulator) is tool to formally quantify the color dimensionality that has emerged in the RV color space of the cortical model (Fig. 41. CMF-SIM treats the retina model and cortical model as black-box color observer. Specifically, we limit ourselves to showing the model two patches of scene color at random, distinct locations on the retina to reflect the effect of real-world viewing conditions, obtaining only scalar score as feedback to indicate the difference in color appearance betweem the two patches (with zero representing a color match). This interface is intentionally limited and identical to color matching experiments with human subjects. Then, resting on the formal, technical bedrock for colorimetry established by [Mxxwell (1866) Grassmann (1853), we exhaustively probe to determine the minimum number of color primaries needed to match any test color through linear combination (Fig. 4 & Appendix ( For example, Maxwell found that most humans require primaries and are formally trichromatic, but red-green colorblind persons require only 2 primaries and are dichromatic. We model diverse color observers and measure the color dimensionality of their emergent vision.
NS (Neural-Scove -coppe) is our tool to display emergent colors in IN by projecting them into RGB color space (Fig. 2). NS is a learnable N x 3 matrix mapping the emergent cortical color manifold in RN linearly conventional RGB color, enabling visual inspection of emergent color vision. To compute NS, we first project our hyperspectral image dataset in two ways: (1) using the retina and learned cortical model to map hyperspectral images into the R-V cortical colorspace, and (2) using conventional color processing to convert hyperspectral data to RGB via CIEXYZ (CIE) 1931). NS then determined as the least squares transform from the former data in RV the latter in RGB, Importantly, NS is independently optimized as a visualization parameter, separate from the main cortical loop, ensuring that the cortex processes ONS without any exposure input images in either hyperspectral or RGB form. NS provides striking color visualizations and complementary intuition visually, which are fully consistent with formal CMF-SIM results. For example, NS allows us to compute and contrast the color palettes of the three different types of human dichromacy (Fig. 5))
In sum, we model cortical color as vectors in RR.V allow color space to emerge naturally through the hypothesized learning mechanism, and measure the emergent color space's dimensionality quantitatively with CMF-SIM and analyze it visually with NS.
6 SIMULATION RESULTS EMERGENCE OF COLOR VISION
Figure 3 begins with a model of typical human retina containing L, M and S cones, and illustrates the time-varying behavior of the cortical model as it learns color vision. The visualized prediction error decreases as the training progresses, and the internal percept converges to 3-D color vision both formally (CMF-SIM and visually (NS imagery), shown in Figure 3 3. Notably, the visual timeline highlights that the cortical model learns color vision one dimension at a time: achieving monochromacy at 700 learning steps, dichromacy at 1,200 steps, and converging to trichromacy after 10 steps. At convergence, the cortical model has accurately inferred all retinal properties: spectral identity of each cell, cell positions and lateral inhibition neighbor weights (Fig. (F)..344). For a cortical model trained with a trichromat retina, CMF-S7 SSM results closely align with human psychophysical data (Stiles & Burch 1955) (Appendix G.I and the model consistently demonstrates 3D color vision across different noise initializations (Appendix G.2), highlighting our model's 's validity and robustness.
The remainder of this section shows CMF-SIM and NS results at convergence for a diversity of simulation scenarios. 4 dissects the CMF-SIM analysis for the case where the retina contains 3 cone types. The graphs show color matching function results using optimal sets of color primaries from 0 to 4 primaries, along with the residual perceptual error as function of wavelength. The area under the curve (AUC) for each error graph suggests that the error falls to near-zero only with at least 3 primaries - this formally proves that the color dimensionality is 3.
Figure | presents results of the hypothesized cortical learning in a diversity of color observers where the retina contains different numbers of cone cell types. This shows that the model learns K -ddimensional color vision when the retina contains K cone types. That is, when the retina contains 1, 2, 3 or 4 cell types, the cortical model converges mono, di, tri, or tetrachromat color vision, formally quantified with CMF-SIM (further analysis of tetrachromat models in Appendix F). And qualitatively, we observe that the NS images grayscale for K = 1. colorblind with only shades of blue and yellow for K- = 2. and full color with all trichromatic hues only with K = 3 (Fig. I).
Use of NS is limited to color dimensionality up to 3, and is not applicable K = 4 case, but
8

---

Published as a conference paper at ICLR 2025
1.
Emerged Color Dimensionality Equals Number of Cone Types
2. Qualitative Colors of Dichromat
3. Control
# of cone cell types
4 LMQS
2 protan
2 deutan
2 tritan
M
MS
LMS
MS
LS
LM
LMS
Percept
Neural Scape Not Applicable
Input ONS
CMF- SIM
D
2D
3D
4D
2D
2D
2D
D
Figure 5: Results of simulating emergence of color vision from various retinas, with analysis of learned color dimensionality using qualitative visualization (NS) and formal methods (CMF-SIM) Cortical models trained with dataset generated with retinas containing 1, 2, 3, 4 cone types result in mono-, di-, tri, tetrachromatic color vision, respectively. 2. Qualitative color of dichromat variants consistent with known vision science on color vision deficiency. 3. Control experiment with a trichromat retina, but with cortical learning deliberately removed: CMF-SIM measures color as I-D, highlighting that cortical learning necessary for emergence of correct color dimensionality.
CMF-SIA formally confirms that 4- 4-dimensional color emerges there. All variants of retinas with two cone types (protanopia, deuteranopia and tritanopia), converge to 2D color vision. But more striking, NS reveals hue shifts among these models that consistent with color vision deficiencies studies (Brettel et al. 1997) (yellow-blue hues for deuteranopia protanopia (Judd) 1948) Graham &
Hsia 1959) and teal-pink hues for tritanopia (Alpern al. 1983 (Fig. 2)
In control experiment, we verify that cortical learning is essential for color vision by comparing two scenarios: (1) baseline where optic nerve signals directly form percepts, with all cortical learning deliberately removed (Fig. 3), and (2) the proposed model including cortical learning (Fig. 5 LMS case). The control baseline experiment fails standard color-matching tests. reducing color dimensionality to 1-D despite 3 cone types, demonstrating that cortical learning is indeed necessary for emergence of correct color dimensionality (details in Appendix D}. In contrast, the proposed learning-based model results in correct 3-D color, as expected.
Figure 6 simulates boosting of color dimensionality in adulthood by gene therapy. Previous genetic studies (Jcobss et al.) 2007 Mancuso al. 2009 Zhang et al.) (017)) demonstrated that the intro- duction an additional class of photopigments in the mammalian cone mosaic, even in adulthood via gene therapy. resulted in new dimension of chromatic sensory experience. Here, simulate the experiment performed in squirrel monkeys (Mancuso et al. 2009), finding simulation results consistent with the noted boost in color dimensionality (Fig. (T) First, model the vision of an adult male squirrel monkey with a protanopic retina (M and S cones only) and a cortical model that has converged. Next, we simulate the effects of gene therapy, by modifying the retina model so that random subset of cones begin to express L opsin. Immediately after this retinal change, NS continues to show blue-yellow dichromatic vision. However, if we allow the cortical model to continue the hypothesized self- self-supervised learning, vision re-converges to boosted, 3-dimensional color (Fig. 6 1). As an aside, the real gene therapy experiment (Mancuso et al., 2009) resulted in expression of both M and I. photopigment in affected cones, but the relative amounts are difficult to ascertain. We additionally model scenarios in which equal and expression occurs or variable amount of relative to M at each cell, and both scenarios converge to trichromacy well (Appendix (). These result indicate that the hypothesized self-supervised learning can explain experimental boosting of color dimensionality in adulthood consistent with Mancuso al. 2009). if the hypothesized learning is assumed to occur continuously even in adulthood.
Figure 2 further demonstrates that simulations of boosting color dimensionality succeed even in 3D-+4 cases by the addition of a fourth cone between S and M cones. This simulation results present the first theoretical work that highlights the possibility of boosting humans with trichromatic
9

---

Published as a conference paper at ICLR 2025
Gene Therapy Squirrel Monkeys to Cure Color Blindness
Gene Therapy Humans Boost Trichromatic to Tetrachromatic Vision
Seraitvity
MS
Sansitweyy
LMS
Sensiowity
LMS
Sunsitwiy
LMQS
Bm
Fm
Pre- Pre-Therapy
Tost-Therapy
Pre- Pre-Therapy
Post- Post-Therapy
Colorblind
Initial
Converged
Trichromat
Initial
Converged
Neural Scope Not
2D
2D
3D
3D
3D
01234
2
Figure 6: Simulated experiments for boosting color dimensionality via gene therapy. 1. 2D dichromat model boosted to 3D trichromacy by mutating some M cones to express L opsins, with cortical learning re-converging to 3D color. 2. 3D trichromat model is boosted to 4I) tetrachromacy by adding a fourth cone type between M and I. cones, with cortical learning re-converging to 4D color.
vision to tetrachromat one by addition of a fourth photopigment. Further details, including the study of human tetrachromat observer model, are described in Appendix F
CONCLUSION
In this work, we presented framework for modeling the emergence of color vision in the human brain. We introduced simulation engine for optic nerve signals, simulated cortical learning purely from such input signals, and measured the emergent color dimensionality qualitatively with NS and quantitatively with CMF-STM.
We believe that the critical contribution of this work actually lies in the computational formulation of the problem in human perception itself given access to only a stream of optic nerve signals, we probe how to meaningfully model the emergence of color vision in the cortex via simulated learning. Once the problem formulated this way, it may not be entirely surprising to machine learning researchers that color with the correct dimensionality can be inferred; however, from the vision science perspective, this new approach is in many ways foreign way to formulate the problem, because it is more common to think color comes from hardwired neural cireuits (Appendix D).
The connection to camera imaging systems is noteworthy as well. The proposed computational framework is akin camera processing pipeline attached to unknown, random color filter array pattern, where even the number of different color filters is a mystery. This is related to branch of computational imaging called "auto calibration" that jointly solves for the scene and unknown system calibration parameters from measurements. One could imagine a new class of engineered sensing systems that have general processing units along the lines of the learning mechanism in this paper, that enables perceptual inference from a broader range of sensory streams that do not require the precision manufacturing common to many current sensors and cameras.
The learning notion of perception emerging from process of disentangling it from encoded sensory streams an interesting intellectual view of how perception may generally emerge in the brain. Any sensory stream going to the brain encodes information from the world entangled with sensing organ characteristics. The self-supervised learning mechanism proposed in this paper might be abstracted into general neural process of learning to predict fluctuations in sensory stream due to ego perturbation, with perception for that sense emerging neurally as the optimal internal representation and associated decoder/encoder pair that enables accurate prediction.
To conclude, we invite the research community to build on the computational framework proposed in this paper, by improving the visual system components (e.g. more accurately modeling details in the eye model, or replacing back prop-ppopngttion the cortical model with more bio- plapsuuueible learning rules (Lillicrap al. 2020 Hinton 2022)),
applying the framework other sensory modalities.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This project was supported by Air Force Office of Scientific Research's MURI grant (FA9550-20-1 0195 and FA9550-21-1-0230; We grateful everyone involved in the MURI project for their support and feedback. Special thanks to Bruno Olshausen, Jessica Lee, and Alexander Belsten for their valuable discussions and insights. We also appreciate Fred Rieke, Lawrence Sincich, and David Brainard for their thoughtful input and engaging conversations.
REFERENCES
Colorimetry. Number 15.2 in CIE Publication. Commission Internationale de l'éclairage, Vienna, Austria, 2nd edition, 1976.
Larry F Abbott. Lapicque's introduction of the integrate-and-Ff model neuron (1907). Brain research bulletin, 50(3-6))303-304, 1999.
Russell J Adams, Daphne Maurer, and Margaret Davis. Newborns' discrimination of chromatic from achromatic stimuli. Journal of Experimental Child Psychology, 41(2):267-25 1986.
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Albert Ahumada. Learning receptor positions. Computational models of visual processing, pp- 23-34, 1991.
M Alpern, K Kitahara, and D H Krantz. Perception of colour in unilateral tritanopia. The Journal of Physiology, 335(():683-697, 1983.
Alexander Anderson, Kavitha Ratnam, Austin Roorda, and Bruno A Olshausen. High-acuity vision from retinal image motion. Journal of vision, 20(7):34-34, 2020.
Juan M Angueyra, Jacob Baudin, Gregory W Schwartz, and Fred Rieke, Predicting and manipulating cone responses to naturalistic inputs. Journal of Neuroscience, 12(7):1254-1274 2022.
Boaz Arad, Radu Timofte, Rony Yahel, Nimrod Morag, Amir Bernat, Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Zhang, et al. Ntire 2022 spectral recovery challenge and data set. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 863-881, 2022.
Claudia Diaz Araya and Jan M Provis. Evidence of photoreceptor migration during early foveal development: a quantitative analysis of human fetal retinae. Visual neuroscience, 8(6):505-55 514, 1992.
Michael A Arbib. The handbook of brain theory and neural networks. MIT press, 2003,
Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning by looking at noise. Advances in Newral Information Processing Systems, 2021.
Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01):217-233, 1961.
Bryce Bayer. Color imaging array. United States Patent, no. 3971065, 1976.
Noah C Benson, Jeremy Manning, and David Brainard. Unsupervised learning cone spectral classes from natural images. PLoS Computational Biology, 1066):e1003652, 2014.
Michael Berry, David K Warland, and Markus Meister. The structure and precision of retinal spike trains. Proceedings of the National Academy of Sciences, 94(10):5411-541 1997.
Vicente Botella-Soler, Stéphane Deny, Georg Martius, Olivier Marre, and Gašper Tkačik. Nonlinear decoding of a complex movie from the mammalian retina. PLoS computational biology, 14(5): e1006057, 2018
11

---

Published as a conference paper at ICLR 2025
Nora Brackbill, Colleen Rhoades, Alexandra Kling, Nishal P Shah, Alexander Sher, Alan M Litke, and EJ Chichilnisky. Reconstruction natural images from responses of primate retinal ganglion cells. Elife, 9cc58516, 2020.
David Brainard, Haomiao Jiang, Nicolas P Cottaris, Fred Rieke, EJ Chichilnisky, Joyce Farrell, and Brian A Wandell. Isetbio: Computational tools for modeling early human vision. In Imaging Systems and Applications, pp. IT4A-4. Optica Publishing Group, 2015.
David H Brainard. Color and the cone mosaic. Annual Review of Vision Science, 1:519-546, 2015.
David H Brainard, David R Williams, and Heidi Hofer. Trichromatic reconstruction from the interleaved cone mosaic: Bayesian model and the color appearance of small spots. Journal of vision, 8(5):15-15, 2008
Hans Brettel, Françoise Viénot, and John Mollon. Computerized simulation of color appearance for dichromats. Josa 14(10):2647-2555, 1997.
Gijs Joost Brouwer and David J Heeger. Decoding and reconstructing color from responses in human visual cortex. Journal of Neuroscience,
2009.
Angela M Brown. Development of visual sensitivity to light and color vision in human infants: critical review. Vision Research, 30(8): 159-1188. 1990.
Yoram Burak, Uri Rokni, Markus Meister, and Haim Sompolinsky. Bayesian model of dynamic image stabilization in the visual system. Proceedings of the National Academy of Sciences, 107 (45):19525-19530, 2010.
Joseph Carroll, Carrie McMahon, Maureen Neitz, and Jay Neitz. Plicker-photomeete electroretino- gram estimates of I: M cone photoreceptor ratio in men with photopigment spectra derived from genetics. JOSA A, 17(3):499-500 2000.
Joseph Carroll, Jay Neitz, and Maureen Neitz. Estimates of I: M cone ratio from erg flicker photometry and genetics. Journal of vision, 2(8): 2002.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
CIE. Commission internationale de l'eclairage proceedings, 1931.
Bevil R Conway. The organization and operation of inferior temporal cortex. Annual review of vision science, 4:381-402, 2018
Bevil R Conway, Saima Malik-Moraleda, and Edward Gibson. Color appearance and the end of hering's 's opponent-colors theory. Trends in Cognitive Sciences, 2023.
Frans W Cornelissen and Eli Brenner. Is adding a new class of cones the retina sufficient cure color-blindness? Journal of vision, 15(13):22-2 22, 2015.
Nicolas Cottaris, Haomiao Jiang. Xiaomao Ding, Brian A Wandell, and David H Brainard. A computatioonal-ossereee model of spatial contrast sensitivity: Effects of wave-front based optics, cone-mosaic structure, and inference engine. Journal of vision, 19 2019.
Lisa J Croner and Ehud Kaplan. Receptive fields of p and m ganglion cells across the primate retina. Vision research, 35(1):7-24, 1995.
Joanna D Crook, Orin S Packer, John B Troy, and Dennis M Dacey. Synaptic mechanisms of color and luminance coding: rediscovering the xy-cell dichotomy primate retinal ganglion cells. In The new visual neurosciences, pp. 144. The MIT Press, Cambridge, MA, 2013.
Christine A Curcio, Kenneth Sloan, Robert E Kalina, and Anita Hendrickson. Human photore- ceptor topography. Journal of comparative neurology, 292(4):497-523, 1990.
12

---

Published as a conference paper at ICLR 2025
Dennis Dacey, Orin S Packer, Lisa Diller, David Brainard, Beth Peterson, and Barry Lee. Center surround receptive field structure of cone bipolar cells in primate retina. Vision research, 40(14): 1801-1811, 2000.
Dennis M Dacey. The mosaic of midget ganglion cells in the human retina. Journal of Neuroscience, 13(12):5334-533 -3355 1993.
Dennis M Dacey and Barry B Lee. The theeboue-on' opponent pathway in primate retina originates from distinct bistratified ganglion cell type. Nature, 367(6465)731-735, 1994.
Dennis M Dacey, Barry B Lee, Donna K Stafford, Joel Pokorny, and Vivianne C Smith. Horizontal cells of the primate retina: cone specificity without spectral opponency. Science, 271(924 656-659, 1996.
Virginia De Sa. Learning classification with unlabeled data. Advances in neural information processing systems, 6, 1993.
Russell L De Valois and Karen K De Valois. multi-stage color model. Vision research, 33(8): 1053-1065, 1993.
Andrew M Derrington, John Krauskopf, and Peter Lennie. Chromatic mechanisms in lateral geniculate nucleus of macaque. The Journal of physiology, 357(1):241-265, 1984.
René Descartes. Descartes: Selected philosophical writings. Cambridge University Press, 1988.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08800 2016.
Robert F Dougherty, Volker M Koch, Alyssa A Brewer, Bernd Fischer, Jan Modersitzki, and Brian A Wandell. Visual field representations and locations of visual areas v1/2/3 in human visual cortex. Journal of vision, 3(10):1-1, 2003.
John E Dowling and Brian Blundell Boycott. Organization of the primate retina: electron microscopy. Proceedings of the Royal Society of London. Series Biological Sciences, 166(1002):80-111, 1966.
Ch Enroth- Cugeel, JG Robson, DE Schweitzer-Tong and AB Watson. Spatio-temporal interactions in cat retinal ganglion cells showing linear spatial summation. The Journal of Physiology, 341(1): 279-307, 1983.
Mark Fairchild. Color appearance models. John Wiley & Sons, 2013.
Greg D Field, Jeffrey L Gauthier, Alexander Sher, Martin Greschner, Timothy A Machado, Lauren H Jepson, Jonathon Shlens, Deborah E Gunning, Keith Mathieson, Wladyslaw Dabrowski. al. Functional connectivity in the retina at the resolution of photoreceptors. Nature, 467(7316): 673-677, 2010.
Peter Földiák. Learning invariance from transformation sequences. Neural computation, 3(2): 194-200, 1991.
Eric R Fossum. Cmos image sensors: Electronic camera-on-a-chip. IEEE transactions electron devices, 44100)1689-1698, 1997.
David H. Foster. Color constancy. Vision Research, 51(7):674-700, 2011. ISSN 0042-6989. doi: https://oi..rr//0. 101 v0116/.visres.200..999
Vision Research 50th Anniversary Issue: Part 1.
Julian Freedland and Fred Rieke. Systematic reduction of the dimensionality of natural scenes allows accurate predictions of retinal ganglion cell spike outputs. Proceedings of the National Academy of Sciences, 1 119(4))e2121144111 2022.
Tyler Godat, Nicolas Cottaris, Sara Patterson, Kendall Kohout, Keith Parkins, Qiang Yang, Jen- nifer M Strazzeri, Juliette E McGregor, David H Brainard, William H Merigan, et al. In vivo chromatic and spatial tuning of foveolar retinal ganglion cells in macaca fascicularis. Plos one, 17 ())e0272261 2022.
13

---

Published as a conference paper at ICLR 2025
CH Graham and Yun Hsia. Studies of color blindness: unilaterally dichromatic subject, 1959.
Hermann Grassmann. Zur theorie der farbenmischung. Annalen der Physik, 165(5):69-88) 1853.
John Guild. The colorimetric properties of the spectrum. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, 230
(681-693)114--187, 1931.
S Leee Guth. Model for color vision and light adaptation. JOSA A, 8(6):966-99) 993, 1991.
Theodros M Haile, Kaitlin S Bohon, Maria Romero, and Bevil Conway. Visual stimulus-driver functional organization of macaque prefrontal cortex. Neuroimage, 188:427-444, 2019.
KEFFER Hartline and Floyd Ratliff. Spatial summation of inhibitory influences in the eye of limulus, and the mutual interaction of receptor units. The Journal of general physiology, 41(5): 1049-1066, 1958.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Ewald Hering. Zur Lehre vom Lichtsinne: sechs Mittheilungen an die Kaiserl. Akademie der Wissenschaften in Wien. C. Gerold's Sohn, 1878.
Geoffrey Hinton. The forward-forwart algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.133355 2022.
Geoffrey E Hinton. Connectionist learning procedures. In Machine learning, pp. 555-610. Elsevier, 1990.
Hajime Hirasawa and Akimichi Kaneko. pH Changes in the Invaginating Synaptic Cleft Mediate Feedback from Horizontal Cells to Cone Photoreceptors by Modulating Ca2+ Channels. Journal of General Physiology, 1226)))65-667,, 11 2003. ISSN 0022-1295.
Heidi Hofer, Joseph Carroll, Jay Neitz, Maureen Neitz, and David R Williams. Organization of the human trichromatic cone mosaic. Journal of Neuroscience, 25(42):9669-9777 2005.
Gordon Holmes. Disturbances of vision by cerebral lesions. The British Brutiah of ophthalmology, (7):353, 1918.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architec- ture the cat's visual cortex. The Journal of physiology. 160(1):106, 1962.
Gerald H Jacobs. Photopigments and the dimensionality of animal color vision. Neuroscience & Biobehavioral Reviews, 86: 1000-30, 2018
Gerald H Jacobs, Gary A Williams, Hugh Cahill, and Jeremy Nathans. Emergence of novel color vision in mice engineered to express a human cone photopigment, science, 31535119): 2007.
Sandra Jiménez and Jesús Malo. The role of spatial information in disentangling the irradiance- reflectance tra.entttttammmmttneeeee ambiguity. IEEE transactions on geoscience and remote sensing. 52(8): 4881-4894, 2013.
Gabriele Jordan, Samir Deeb, Jenny M Bosten, and John Mollon. The dimensionality of color vision in carriers of anomalous trichromacy. Journal of vision, 12- 2010.
Deane Judd. Color perceptions deuteranopic and protanopic observers. Journal of Research of the National Bureau of Standards, 41:247-271,
1948. 1948
Deane B Judd. Response functions for types of vision according to the Müller theory. US Govermment Printing Office, 1949.
14

---

Published as a conference paper at ICLR 2025
Young Joon Kim, Nora Brackbill, Eleanor Batty, JinHyung Lee, Catalin Mitelut, William Tong, EJ Chichilnisky, and Liam Paninski. Nonlinear decoding of natural images from large-scale primate retinal ganglion recordings. Neural Computation, 33(7):1719-111 2021.
Ron Kimmel. Demosaicing: Image reconstruction from color ced samples. IEEE Transactions on image processing, 8(9):1221-1288, 1999.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
John Krauskopf and Gegenfurtner Karl. Color discrimination and adaptation. Vision research, 32 (11):2165-2175, 1992.
Stephen W Kuffler. Discharge patterns and functional organization of mammalian retina. Journal of newrophysiology, 16(1):37-68. 1953.
Rosa Lafer-Sousa and Bevil Conway. Parallel, multi-stage processing of colors, faces and shapes in macaque inferior temporal cortex. Nature neuroscience, 16(12):1870-1875 2013.
Edwin H Land. The retinex theory of color vision. Scientific american, 237(6):088-129, 1977. 1977.
Michael F Land and Dan-Eric Nilsson. Animal eyes. OUP Oxford, 2012.
Lapicque. Recherches quantitatives sur 620 635, 1907.
I l'excitation electrique des nerfs. Physiol. Paris,
Jessica Lee, Nicholas Jennings, Varun Srivastava, and Ren Ng. Theory of human tetrachromatic color experience and printing. ACM Trans. Graph, 43(4), July 2024.
Peter Lennie. P William Haake, and David R Williams. The design of chromatically opponent receptive fields. Computational models of visual processing, pp. 71-82, 1991.
Ming Li, Fang Liu, Mikko Juusola, and Shiming Tang. Perceptual color map macaque visual area
v4. Journal of Neuroscience, 34(1):202-211 217, 2014.
Trisha Lian, Kevin J MacKenzie, David H Brainard, Nicolas Cottaris, and Brian A Wandell. Ray tracing 3d spectral scenes through human optics models. Journal of vision, 19(12):23-2 23, 2019.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, 2 (1))3335-334, 2020.
AM Litke, N Bezayiff, EJ Chichilnisky, W Cunningham, W Dabrowski, AA Grillo, M Grivich, Grybos, P Hottowy, Kachiguine, al. What does the eye tell the brain?: Development of system for the large-scale recording of retinal output activity. IEEE Transactions on Nuclear Science, 51(4):1434 1440, 2004.
Jian K Liu, Dimokratis Karamanlis, and Tim Gollisch. Simple model for encoding natural images by retinal ganglion cells with nonlincar spatial integration. PLoS computational biology, 18(3): el e1009925, 2022.
Jianfei Liu, Hae Won Jung, Alfredo Dubra, and Johnny Tam. Cone photoreceptor cell segmentation and diameter measurement on adaptive optics images using circularly constrained active contour model. Investigative ophthalmology & visual science, 59(1 16:499 4652, 2018.
Ye Liu, Ming Li, Xian Zhang, Yiliang Lu, Hongliang Gong, Jiapeng Yin, Zheyuan Chen, Liling Qian, Yupeng Yang, Ian Max Andolina, et al. Hierarchical representation for chromatic processing across macaque vI, v2, and v4. Neuron, 108(3):538-555 2020.
Margaret S Livingstone and David H Hubel. Psychophysical evidence for separate channels for the perception of form, color, movement, and depth. Journal of Neurosceence, 7:1)::4411111111 7(11):31163448 1987.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. In International Conference on Learning Representations, 2016.
15

---

Published as a conference paper at ICLR 2025
David L MacAdam. Visual sensitivities to color differences in daylight. Josa, 32(5):247-274, 1942.
Donald IA MacLeod and Robert M Boynton. Chromaticity diagram showing cone excitation by stimuli of equal luminance. JOSA, 69(8): 183-1186, 1979.
Walter Makous. Comment on" emergence of novel color vision in mice engineered to express human cone photopigment". Science, 3 8658488) 2007.
Laurence T Maloney and Albert Ahumada. Learning by assertion: Two methods for calibrating a linear visual system. Neural Computation, 1(3):392-40,, 1989.
Katherine Mancuso, William Hauswirth, Qiuhong Li, Thomas Connor, James A Kuchenbecker, Matthew C Mauck, Jay Neitz, and Maureen Neitz. Gene therapy for red-green colour blindness in adult primates. Nature, 461(7765):784-535 2009.
Olivier Marre, Gasper Tkacik, Dario Amodei, Elad Schneidman, William Bialek, and Michael Berry. Multi-electrode array recording from salamander retinal ganglion cells. 2017.
Susana Martinez-Conde, Jorge Otero-Millan, and Stephen L Macknik. The impact of microsaccades on vision: towards a unified theory of saccadic function. Nature Reviews Neuroscience, 14(2): 83-96, 2013.
James Clerk Maxwell. Theory of the perception of colors. Trans. R. Scottish Soc. Arts, 4:394-400, 1856.
W. McClurkin and L. M. Optican. Primate striate and prestriate cortical neurons during discrim- ination. i. simultaneous temporal encoding of information about color and pattern. Journal of Neurophysiology, 75(1):481-4955 1996.
W. McClurkin, J. A. Zarbock, and L. M Optican. Primate striate and prestriate cortical neurons dur- ing discrimination. ii separable temporal codes for color and pattern. Journal of Neurophysiology. 75(1):4965507, 1996.
K McLaren. Xiii-th development of the cie 1976 (1* a* b*) uniform colour space and colour- difference formula. Journal of the Society of Dyers and Colourists, 92(9):338-344 1976.
Matthew J McMahon, Martin JM Lankheet, Peter Lennie, and David Williams. Fine structure of parvocellular receptive fields in the primate fovea revealed by laser interferometry. Journal of Neuroscience, 20(5):2043-203 2000.
JD Mollon and JK Bowmaker. The spatial arrangement of cones in the primate fovea. Nature, 360 (6405):677-679, 1992.
Thomas Naselaris, Ryan Prenger, Kendrick N Kay, Michael Oliver, and Jack L Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6):902-915, 2009.
J Neitz and Neitz. Evolution of the circuitry for conscious color vision in primates. Eye, 31(2): 286-300, 2017.
Jay Neitz and Maureen Neitz. The genetics of normal and defective color vision. Vision Research, 51 (7):633-651, 2011.
Isaac Newton. Opticks: or, A Treatise of the Reflexions, Refractions, Inflexions and Colours of Light. Courier Corporation, 1704.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L. Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19):1641-1446 2011.
J Kevin O' O'Regan and Alva Noë. A sensorimotor account of vision and visual consciousness. Behavioral and brain sciences, 24(5):939 2001.
Gustav A Osterberg. Topography of the layer of rods and cones in the human retina. Acta ophthalmo- logica, 1935.
16

---

Published as a conference paper at ICLR 2025
Orin S Packer and Dennis M Dacey. Receptive field structure of hl horizontal cells in macaque monkey retina. Journal of Vision, 2(4):1-1, 2002.
Stephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in sensory population. Proceedings of the National Academy of Sciences, 1 12(22))69088 6913, 2015.
Nikhil Parthasarathy, Eleanor Batty, William Falcon, Thomas Rutten, Mohit Rajpal, EJ Chichilnisky, and Liam Paninski. Neural networks for efficient bayesian decoding of natural images from retinal neurons. Advances in Neural Information Processing Systems, 30, 2017.
Ken Perlin, An image synthesizer. ACM Siggraph Computer Graphics, 19(3):277-266, 1985.
Baingio Pinna. Un effetto di colorazione. In I/ laboratorio e la città. XXI Congresso degli Psicologi Italiani, volume 158. Edizioni SIPs, Societá Italiana di Psiocologia, Milano, 1987.
Baingio Pinna, Gavin Brelstaff, and Lothar Spillmann. Surface color from boundaries: a new 'watercolor illusionn Vision research, 41(20))2669-2677 2001.
Jonathan R Polimeni, Bruce Fischl, Douglas N Greve, and Lawrence Wald. Laminar analysis of t bold using imposed spatial activation pattern in human vl. Neuroimage, 52(4):1334-134, 2010.
Rajeev Ramanath, Wesley E Snyder, Griff L Bilbro, and William A Sander III. Demosaicking methods for bayer color arrays. Journal of Electronic imaging, 11(3):36--155, 2002.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
Katja Reinhard and Thomas A Münch. Visual properties of human retinal ganglion cells. PLoS One, 16(2):e0246952, 2021.
Ilias Rentzeperis, Andrey R Nikolaev, Daniel C Kiper, and Cees van Leeuwen. Distributed processing of color and form in the visual cortex. Frontiers psychology, 5:932, 2014.
Dragos Rezeanu, James Kuchenbecker, Rachel Barborek, Mareus Mazzaferri, Maureen Neitz, and Jay Neitz. Explaining the absence of functional tetrachromacy in females with four cone types. Investigative Ophthalmology & Visual Science, 6278):527, 2021.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pp. 1530-1538. PMIR, 2015.
Austin H Riesen, Robert L Ramsey, and Paul D Wilson. Development of visual acuity in rhesus monkeys deprived of patterned light during early infancy. Psychonomic Science, 1(1-12):33-4, 1964.
Mattia Rigotti, Omri Barak, Melissa R Warden, Xiao-Jing Wang, Nathaniel D Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497 (7451))585-590 2013.
Robert W Rodieck. Quantitative analysis of cat retinal ganglion cell response to visual stimuli. Vision research, 5(12):533-601 1965.
Robert W Rodieck. The first steps in seeing. 1998.
Martin Rolfs. Microsaccades: small steps long way. Vision research, 49(20))2415-24M) 2009. 2009.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention -MICCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234-241. Springer, 2015.
Austin Roorda, Fernando Romero- Romero-Borja, William J Donnelly III. Hope Queener, Thomas J Hebert, and Melanie CW Campbell. Adaptive optics scanning laser ophthalmoscopy. Optics express, 10 19::405-112. 2002.
17

---

Published as a conference paper at ICLR 2025
Michele Rucci and Jonathan D Victor. The unsteady eye: information cetoomatioo-procsssing stage, not bug. Trends in neurosciences, 38(4): (914)):95-2000 2015.
Michele Rucci, Ehud Ahissar, and David Burr. Temporal coding of visual space. Trends in cognitive sciences, 22((0):883- 895, 2018.
Ramkumar Sabesan, Heidi Hofer, and Austin Roorda. Characterizing the human cone photoreceptor mosaic via dynamic photopigment densitometry. PloS one, 10(12))e0144811 2015.
Ramkumar Sabesan, Brian P Schmidt, William S Tuten, and Austin Roorda. The elementary representation of spatial and color vision in the human retina. Science advances, 2(
2016.
Jared M Salisbury and Stephanie Palmer. Optimal prediction in the retina and natural motion statistics. Journal of Stafistical Physics, 16265):139--132, 2016.
Elad Schneidman, Michael J Berry, Ronen Segev, and William Bialek. Weak pairwise correlations imply strongly correlated network states in neural population. Nature, 440(7087):1007-1111 2006.
Manuel Schottdorf and Barry B Lee. A quantitative description of macaque ganglion cell responses to natural scenes: the interplay of time and space. The Journal of physiology, 599(12):316-3199) 2021.
Jérémie Sibille, Carolin Gehr, Jonathan Benichov, Hymavathy Balasubramanian, Kai Lun Teh, Tatiana Lupashina, Daniela Vallentin, and Jens Kremkow. High-density electrode recordings reveal strong and specific connections between retinal ganglion cells and midbrain neurons. Nature communications, 13(1):5218, 2022.
Yosef' Singer, Yayoi Teramoto, Ben DB Willmore, Jan WH Schnupp. Andrew J King, and Nicol S Harper. Sensory cortex optimized for prediction of future input. elife, 7:e31557, 2018
Yosef' Singer, Luke Taylor, Ben DB Willmore, Andrew King, and Nicol S Harper. Hierarchical temporal prediction captures motion processing along the visual pathway. eLife, 12:e52599, 2023a.
Yosef' Singer, Luke Taylor, Ben DB Willmore, Andrew King, and Nicol S Harper. Hierarchical temporal prediction captures motion processing along the visual pathway. Elife, 12:e52599, 2023b.
Barun Singh, William T Freeman, and D Brainard. Exploiting spatial and spectral image regularities for color constancy. Workshop on Statistical and Computational Theories of Vision. Nice, France. Citeseer, 2003.
Florentina Soto, Cen-Chun Hsiang, Rithwick Rajagopal, Kisha Piggott, George J Harocopos, Steven M Couch, Philip Custer, Josh L Morgan, and Daniel Kerschensteiner. Efficient coding by midget and parasol ganglion cells the human retina. Neuron, 107(4):656-666, 2020.
Mandyam Veerambudi Srinivasan, Simon Barry Laughlin, and Andreas Dubs. Predictive coding: fresh view of inhibition the retina. Proceedings of the Royal Society of London. Series Biological Sciences, 21 216(1205):427-4555 1982.
WS Stiles and JM Burch. Interim report to the commission internationale l'eclairage, zurich, 1955, on the national physical laboratory's investigation of colour colour-matching (1955). Optica Acta: International Journal of Optics, 164) 1955.
Vincent Stimper, David Liu, Andrew Campbell, Vincent Berenz, Lukas Ryll, Bernhard Schölkopf, and José Miguel Hermández-Lobateee normflows: A pytorch package for normalizing flows. Journal of Open Source Software, 8(86):5361, 2023. doi: 10.21 15210000sss.0536111 053611 URL https: l/doi.org//0.3 1t..orrg/10..1111 105/joss.05361 //b5.oo0/002110000ss.003333
Andrew Stockman and Lindsay T Sharpe. The spectral sensitivities of the middle-and long- wavelength-senann cones derived from measurements in observers of known genotype. Vision research, 40(13): 2000.
18

---

Published as a conference paper at ICLR 2025
Andrew Stockman, Lindsay T. Sharpe, and Clemens Fach. The spectral sensitivity of the human whort-wavelength sensitive cones derived from thresholds and color matches. Vision Research, 39 (17):2901-2927, 1999. ISSN 0042-6989.
Andrew Stockman, David H Brainard, et Color vision mechanisms. OSA handbook of optics, 11-1, 2010.
Roger BH Tootell, Martin Silverman, Eugene Switkes, and Russell L De Valois. Deoxyglucose analysis of retinotopic organization in primate striate cortex. Science, 218(4575):902-994 1982.
Antonio Torralba and Aude Oliva. Statistics of natural image categories. Network: computation in neural systems, 14(3):391, 2003.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jan Verweij, Eric P Hornstein, and Julie L Schnapf. Surround antagonism in macaque cone photore- ceptors. Journal of Neuroscience, 12299111229 2003.
Hermann Von Helmholtz. Handbuch der physiologischen Optik, volume 9, Voss, 1867.
J Von Kries. Theoretische studien über die umstimmung des sehorgans. Festschrift der Albrecht- Ludwigs- Universität, 12:145-158, 1902.
Thomas Wachtler, Eizaburo Doi, Te-Won Lee, and Terrence J Sejnowski. Cone selectivity derived from the responses of the retinal cone mosaic to natural scenes. Journal of vision, 7(8):66, 2007.
GEORGE WALD. The molecular basis of visual excitation. Nature, 21965156):00-8077, 1968.
Yiyi Wang, Nicolas Bensaid, Pavan Tiruveedhula, Jianqiang Ma, Sowmya Ravikumar, and Austin Roorda. Human foveal cone photoreceptor topography and its dependence on eye length. Elife, 8: c47148, 2019.
John Werner, Brennan Marsh-Armstronn.. and Kenneth Knoblauch. Adaptive changes in color vision from long-term filter usage in anomalous but not normal trichromacy. Current Biology, 30 (33):301 2020.
David R Williams, Donald IA MacLeed, and Mary M Hayhoe. Foveal tritanopia. Vision Research, 2109):344-1356, 1981.
Lance Williams. Pyramidal parametrics. In Proceedings of the 10th annual conference Computer graphics and interactive techniques, pp. 1-11, 1983.
Lauren E. Wool, Joanna D Crook, John B Troy, Orin S Packer, Qasim Zaidi, and Dennis M Dacey. Nonselective wiring accounts for red-green opponency in midget ganglion cells of the primate retina. Journal of Neuroscience, 38(6): (520)))152--54000 2018.
William David Wright. A re-determination of the trichromatic coefficients the spectral colours. Transactions of the Optical Society, 30(4):141, 1929.
Eric Wu, Nora Brackbill, Alexander Sher, Alan Litke, Eero Simoncelli, and EJ Chichilnisky. Maxi- mum posteriori natural scene reconstruction from retinal ganglion cells with deep denoiser priors. Advances in Neural Information Processing Systems, 35:27212-27224, 2022.
Günther Wyszecki and Walter Stanley Stiles. Color science: concepts and methods, quantitative data and formulae. John wiley & sons, 1982.
Ed Yong. An immense world: How animal senses reveal the hidden realms around us. Knopf Canada, 2022.
Laura K Young and Hannah E Smithson. Emulated retinal image capture (erica) to test, train and validate processing of retinal images. Scientific Reports, 11(1):11225, 2021.
19

---

Published as a conference paper at ICLR 2025
Thomas Young. li. the bakerian lecture. the mechanism of the eye. Philosophical Transactions of the Royal Society of London, (91):23-88, 1801.
Thomas Young. Ii. the bakerian lecture. on the theory of light and colours. Philosophical transactions of the Royal Society of London, (92):12-48, 1802.
Semir M Zeki. Functional specialisation in the visual cortex of the rhesus monkey. Nature, 274 (5670):423 428, 1978.
Ling-Qt Qi Zhang, Nicolas P Cottaris, and David H Brainard. An image reconstruction framework for characterizing initial visual encoding. Elife, 11:e711322 1132, 2022.
Yuxin Zhang, Wen-Tao Deng, Wei Du, Ping Zhu, Jie Li, Fan Xu, Jingfen Sun, Cecilia D Gerstner, Wolfgang Baehr, Sanford I. Boye, al. Gene-based therapy in a mouse model of blue cone monochromacy. Scientific reports, 7(1):6690, 2017.
Zhetuo Zhao, Ehud Ahissar, Jonathan D Victor, and Michele Rucci. Inferring visual space from ultra-fine extra-retinal knowledge of gaze position. Nature communications, 14(1):269, 2023.
20

---

Published as a conference paper at ICLR 2025
Supplementary Materials
In this supplementary section, we provide additional details on our computational framework for modeling the emergence of human color vision. Section Aooerrss the simulation of optic nerve signals from hyperspectral scene images and our modeling of retina circuitry. Section details the design of our simulated cortical model. formal color matching function tests of color dimensionality, CMF-STM. Section D we probe whether the emergence of color vision depends cortical learning, by introducing comparative, baseline cortical models. Section Eisiulatess variant cell expression phenotypes after gene therapy in squirrel monkeys (Mancuso et al. 2009), showing robustness of the simulation result that addition of a new cell opsin gene enhances color dimensionality. Finally, Section discusses our tetrachromacy simulation, highlighting the differences between the human cone (between M and cones) and the pigeon Q cone (between S and M cones). A supplementary video available on our project website:
step-by-step description of how we simulate
https:/mmatisse.ecs..bee www.eeeee.....eeee..........
Table 1: Glossary of terms used in our framework
Scalars
F. y/
Time, world view coordinates Optic nerve image coordinates
u, v
N
N-dimensional vector represents cortical color
Eye Simulation Engine
S1(2, y) r
Scene image, spectrum at each pixel
Retina encoding of scene into time-averaged optic nerve image: ((s:) O
Cortical Image Functions
Orlu, v)
Time- averaaee optic nerve image
A, v)
Cone activations image
Visual percept image
Vi(2,, v)
Unwarped visual percept image
(0+de(u,))
Predicted optic nerve image, time dt
Neural Buckets (Learnable Parameters)
C
Color type of each cone
P
Position of each cone in visual field
W
Lateral inhibition weights neighboring cones Demosaicing operator parameters
D
M
Motion estimation operator parameters
Learned Cortical Function Operators + decodes O, into V,, and y re-encodes V. into O,, + = +p &c o Pw, and V = W/ o Wc Wp.
Φ, V
Pw. Vw
Transforms O₁ into A,, and vire-versa. Transforms A, into V,, and vice versa. Transforms Vi into Vz, and vice versa.
cc. Vc
pp. Up
pM O2) ((z2,)
Estimates world translation (æ, y) between two optic nerve images Translates an image laterally by (z. y). (V) Vit
A SIMULATION ENGINE OF BIOLOGICAL EYES
We simulate the optic nerve signals from hyperspectral scene images, simulate eye gaze movements, and model the retina circuitry. We describe the details of these simulations in the following.
A.1 DATABASE OF HYPERSPECTRAL SCENE IMAGES
We model the scene viewed by the eye using a dataset comprising 900 hyperspectral images from everyday scenes, captured with hyperspectral camera (Arad et al. 2022). Each image in the dataset
21

---

Published as a conference paper at ICLR 2025
is of dimension 512 pixels 482 pixels 31 spectral channels, where each pixel is represented by its spectral power distribution ("spectra below). To generate our training data set, we cropped thousands of 482x482 patches from these hyperspectral images, ensuring sufficient resolution to avoid aliasing effects, induced by varying cone cell densities (Section A.3.2).
While only hyperspectral images are used during learning simulation, RGB images be presented during test-time by simulating a projector with specific spectral power distributions (SPDs) for the R. G. and B channels (Cottaris al., 2019), effectively converting each RGB image into spectral representation before passing into the retina model. This test-time use of RGB images does not affect learning and is purely for illustrative purposes in the figures (e.g., Figure
A.2 EYE GAZE MOVEMENTS FIXATIONAL DRIFT
We simulate small eye movement as fixational eye drift with random walk (Young & Smithson 2021). This produces a stream of the current scene image translating across the retina. Over a time interval of duration dl, sample the change in eye motion from a uniform probability distribution (dX,dY)= ~ ((((11, 15, 15) x (-15, 15)] where 15 corresponds to 15 pixels in the scene image S.
The Mona Lisa video in Supplementary Video 2:08 also incorporates manually- saccades (e.g. from hand to mouth) for illustrative purposes, but the learning simulation uses only fixational drift.
A.3 RETINA CIRCUITRY MODEL
Our model of light detection and signal encoding in the retina is the textbook model (Rodieck 1998) of the photopic, midget, "private-ling line" pathway (see Figure 2). Details of our implementation follow.
A.3.1 CONE MOSAIC AND SPECTRAL SAMPLING
In our simulation, we model 256 x 256 cone cells, with the random distribution of L, M and cone cells the retina with relative probability of 0.63, 0.32 and 0.05 from (Saessan al. (2015). We model I., M and S spectral response using the template response function from Carroll et al. (2000), with spectral peaks of 560, 530, 419 nm, respectively. We model photon-shot noise in the photoreceptor activation values, simulating a signal-to-noise ratio of approximately 100. The simulation omits foveal tritanopia (Williams al. 1981).
A.3.2 FOVEATION AND CONE POSITIONS
We model cone cell spatial density on the retina in accordance with Curcio et al. (1990), while enforcing retinotopy. Starting from a regular grid of cells, we computationally perturb positions until the target spatial distribution is stochastically achieved, while constraining each cell to be always confined by its neighbors. We model biological variation in cell locations, which represents challenge for cortical learning, by adding multi-resolution positional randomness (Perlin) 1985).
A.3.3 CENTER- SUREERN LATERAL INHIBITION
We mathematically model textbook lateral inhibition in retinal signals Difference-of Gaussians (DoG) convolution kernel EEgroth-Cugell et al. 1983), with parameters fitted from electrophysiol- ogy (Wool et al. 2018). Specifically our DoG kernel has standard deviations of 0.15 and 0.9 cone diameters, respectively. for the positive center and negative surround Gaussians. Significantly. the relative amplitudes are such that the kernel has a non-zero mean of 0.09 (Lennie et al.) [991] Wool al. 2018). We convolve the array of cone activation values by this DoG kernel to compute outputs to bipolar cells.
A.3.4 ON/OFF PATHWAYS AND OPTIC NERVE SPIKING
We model textbook on- and off- connections from cone activations to bipolar cells, forwarding to and off- retinal ganglion cells (RGC)s. The on- activation is modeled by a rectified linear unit activation function, ReLU(x) = max(0, x), where x is the laterally- -nnhbbled cone output. The off- activation is modeled ReI.U(=x)) Optic nerve spikes are the outputs at RGC axons, with action
22

---

Published as a conference paper at ICLR 2025
Details of Learnable Cortical Functions
True ONS time
Laceral Inhibition
Cone Activations
Cone Identity
Visual Percept
Cell Position
Unwarped Visual Percept
(I))O)
((A)
Pp(VV
Translation
D
Compute prediction error & update cortical functions Y and accordingly
Kernel Fourier domain
WXX learnable
Demossicking CNN
Coordinate- based network
S2(V)
D+Ar
OH+AI
v (Artad
Ar+Al
V /+AI
True ONS a &r
Predicted ONS at time 2r
Predicted Cone Activation at
Predicted Visual Percept at + A
Predicted Unwarped VP at 1+ Ar
Figure 7: Pipeline of decoding, translation and re-encoding cortical functions. The optic signal image Oz decoded into visual percept image V,. The percept translated in accordance with eye motion over time dt, then re-encoded back into prediction of the optic nerve signal O at short time in the future.
potentials generated with Leaky Integrate and Fire model 2003
(Lapicque
1907
Abbott T999
Arbib
B CORTICAL MODEL
B.1 TIME-AVERAGED OPTIC NERVE SIGNALS
We model the first step of cortical processing as re-combination of the on- and off- RGC action potentials followed by time-averaging. This results in an image stream O, with real-valued pixel values that are proportional to the laterall--inhibice output from cone cells. O is the sole input to the remainder the cortical model.
B.2 PREDICTION PIPELINE AND LEARNING OBJECTIVE
Figure 7 shows an elaboration of the cortical prediction pipeline introduced in Figure 3 and summarized in Section This pipeline represents existence-proof simulation that the hypothesized self-supervised learning on O, can successfully produce: (1) emergence of vision with the correct color dimensionality, and (2) inference of invariant retinal properties, including learned neural buckets for cone spectral identities C, cell positions P and lateral inhibition weights W. Our results show that both can be simultaneously achieved by learning to minimize the error in predicting the fluctuations in optic nerve signal values.
To re-summarize Section the prediction is made by applying the learned functions for decoding l translation S and re- re-encoding V to the optic nerve signal O,, to obtain a predicted optic nerve signal Ö = ( (O,)())))))) The learning objective function is to minimize prediction error, the difference between predicted and real optic nerve images at time + dt: Eprediction = |00 Öc+dx 1 ||0 ( ((((((((O))) (O,))) The decoder & and re-encoder &i functions are each factorized into a pipeline, such that: 4p o 4c Φw and Vw Ve Vp. where each sub-function is an operator conditioned on its corresponding neural bucket, C. P and W. The following sections describe the implementation of each learnable neural bucket and its associated cortical sub-function.
B.2.1 LEARNING LATERAL INHIBITION NEIGHBOR WEIGHTS W
In the factorized function pipeline, sub-decoder &w conceptually inverts lateral inhibition to transform optic nerve image O into cone activation image A,, Conversely, sub-encoder Vw reproduces lateral
23

---

Published as a conference paper at ICLR 2025
inhibition to transform A, into O₁. These operators modeled, respectively, as inverse and forward convolution operators, implemented by pixel-wise division and multiplication the Fourier domain. The learnable parameters for this operator neural bucket W, representing the Fourier transform image pixels the lateral inhibition DoG kernel's Fourier transform. The resolution of W is set equal the image resolution of Oz.
B.2.2 LEARNING COLOR TYPES C AND INTERPOLATION D
In the factorized function pipeline, sub-decoder &e conceptually transforms cone activation image A, (scalar-valued pixels) into a full-color visual percept image V (N-dimensional vector pixels). Conversely, e re-encodes V; into A,. These functions depend on two neural buckets of learnable parameters, C and D. C is an image of N N-dimensional vectors, in which the cortical model learns to represent the spectral type of the source cone associated with each pixel in Ar. are the parameter weights of learnable function Φn that spatially interpolates color across the image, which we implement as a convolutional neural network based on the U-Net (Ronneberger et al., 2015) architecture.
Mathematically, &e and W are defined as Pc(A,) = pD(4s @ C), where @ denotes element-wise multiplication; (V,V = V, @ C, where @ denotes element- wise dot product. To maintain &c and Vc as pseudo-inverses, constrain C @ to be image with all pixels equal 1.
B.2.3 LEARNING CELL POSITIONS P
As shown in Fig. 3.FF, optic nerve images are spatially distorted due to foveation, whereas visual perception is apparently undistorted. In the factorized function pipeline, sub-decoder Dp conceptually transforms (u, v), which is foveated and spatially distorted, into the final visual percept image V1 y). which is Euclidean and undistorted. Conversely, P re-warps Vi(x, y) into Vz v).
We define a learnable spatial warping function dp v) = (x, y). implemented using normalizing flow (Rezende & Mohamed 2015) Dinh et al., 2016)) that is an invertible function. Neural bucket comprises the learnable parameters this normalizing flow network. Then 4p (Z))((z y) = Vep (x. y)), and V (M,) v) = (1, v)).
B.3 LEARNING EYE MOTION ESTIMATION M
In the learning loop, cortical sub-function Ω translates the visual percept image according to the spatial motion (dz, dy) that occurs during the brief period between and t dt. Where does (dæ, dy) come from? Here, we describe how the cortex can learn helper cortical function MM (O2, Oc+dt) that estimates (dz, dy) directly from the optic nerve stream values times t and + dt, maintaining the strict operation of the cortical learning model purely from optic nerve signals.
We model learning of (On(O, Oz+ds) with neural bucket M by optimizing:
M = argmin [B(O+ar)) Vp "PPPppasO (pp((
M.P
where B is convolution operator that low-pass filters the optic nerve images, effectively halving the highest frequencies; Pp aims to dewarp the optic nerve image into Euclidean coordinates, and vp is the inverse operator that rewarps the image. Details are discussed also in Supplementary Video 16:25.
B.4 EFFICIENT LEARNING IMPLEMENTATION & ALTERNATIVE REPRESENTATION OF INTERNAL PERCEPTS
We simulate the self-supervised learning by paraliel numerical optimization of all neural buckets by applying the stochastic gradient descent algorithm to minimize the prediction error metric. In practice, this is implemented by repeating the following step thousands of times until convergence: pick batch of different scene images from the database; generate the optic nerve signals at time t with the retina encoding model; send this into the cortical model and execute the decode / translate / re-encode functions with current neural bucket parameters to estimate the signal at + dt; compute the actual optic nerve signal at + dt with the retina encoding model; compute the prediction error
24

---

Published as a conference paper at ICLR 2025
by taking the difference between prediction and actual signal: backpropagate the prediction error to update neural bucket parameters.
For efficient processing of cortical functions, we avoid computing full-resolution undistorted visual percept images (a y) during the learning simulation. Instead, we directly compute Ve+dt (u, v) by optical flow of the pixels in (u, v),, where the optical flow map is defined by ( (VP o (O, . 2p). This is mathematically equivalent the learning implementation described in Section B.2 and the cortical model continues to learn cell positions P, and therefore decoder & can still be applied to compute undistorted visual percepts when desired.
C COLOR MATCHING FUNCTION TESTS & CMF-SIM
In the classical color matching theory of Maxwell (856) and Grassmann (1853), an observer compares a test color spectrum, f, and a set of K primary color spectra, p; for -I, ..... K and tunes real weights a for each primary until a color match is achieved. If particular weight has negative value, primary light of that magnitude is added to the test color rather than to the other primary colors. Mathematically, color match is achieved when the matching spectrum Σ at ps and test spectrum + Σ "ir)))+poo pt appear identical. Here, az is defined as max( 0).
We simulate such classical color matching by formulating the retina model plus learned cortical model as a black-box color observer in the colorimetric sense. That is, we take the test and matching spectra, simulate the stimulation different parts of the retina for test and match color patches, apply the retina encoding model Y and the learned cortical decoder & to each separately, and iteratively update the weights a; until there is further improvement in the computed perceptual difference E [ K a."p)) pt)) - (t -a,) ) 1.)) Note that the optimal E will not be close to zero if the primaries cannot match test color t.
The color dimensionality of an observer is formally equal to the minimum number of primary colors required to match any test color. CMF-SIM simulates thousands of color matches to rigorously compute this dimensionality for any given retina model plus cortical model. In CMF-SIM, we simulate classical color matching function (CMF) tests of exhaustively attempting to match test colors equal each monochromatic wavelength (100 samples from 400 to 700 nm), with linear combination of K primary colors. For the primary colors, we use distinct, monochromatic spectra. We start CMF tests with a single primary, and add primaries one-by-one until all test wavelengths in the CMF can be matched successfully. The pseudocode for CMF-SIM is described in Algorithm
For example, for our cortical model to be formally measured as trichromat, we must show that there do not exist any 2 primaries that can pass the CMF tests, and show 3 primaries that succeed. A brute-force approach would be to exhaustively test all possible sets of K primaries; for efficiency, instead perform stratified sampling randomly generate large number (e.g. 500) distinct sets of primaries for each choice of K. We require that all of these possible primary sets fail, with the aggregate perceptual error across all wavelengths being greater than threshold e, before incrementing the CMF tests to N+1 primaries.
D BASELINE RESULTS - TESTING IF CORTICAL INFERENCE IS REQUIRED FOR COLOR VISION
fundamental question about color perception whether cortical inference is necessary, or if human color perception arises directly from hardwired neural circuits. The main paper presents analysis an existence proof that cortical inference can indeed result color vision of the correct dimensionality. Here, add evidence that cortical inference of some kind is necessary, by modeling and testing three baseline cortical models with limited or no cortical processing.
For our first baseline, assume extreme case where cortical learning occurs, treating raw optic nerve signals directly as internal percepts (i.e., the cortical decoder function is an identity function). We apply CMF-SIM directly to the optic nerve signals generated by a trichromat retina. Figure 8 presents analysis that this model fails to generate vision that can be recognized as color consistent. The analysis is to perform baseline color matching experiments where the test and match
25

---

Published as a conference paper at ICLR 2025
Algorithm 1: Pseudocode for CMF-SIM
Data: WAVELENGTHS, MAX TRIALS, Retinal process T, Cortical decoder d
Function CMF-STM ():
base_errors +- Computeraseerrross ()
color_ colo-ddimensionality + PindMinnmmurrriariie (base_errors)
return color color_dimensionalliy
Function CompureeaseE:rors ():
Initialize an array errors
foreach wavelength À in WAVELENGTHS
(I, 1) RandomL.ocation(
(I, (),3) Randoml.ocation))
colorPatchA + MonechromatiiiCoooPPPcch.. (z. 4)
colorPatchB + MonochromaiicCCloPPtttth.. (æ, y)
perceptA T ( (T(coooPPatchA)
perceptB - (T(oooor()t)c)))
errors[A] PerceppualEEmor(perrpppA,, (z, A. perceptB, (I, y)
return errors
Function Fi FindMininnii wwwpaaaiiiimmmmmimmmmmmmmmm (base_errors)
num Batnn_rrimiiess... +- 0
repeat
num_primaries <- num num_primaries +
all all_tests_passed <- true
trial x-
repeat
trial <- trial +
Randomly initialize primaries = {pI.... Paum Paum_primaries primuries
foreach wavelength À in WAVELENGTHS do
Zero initialize coefficients a = { Opom [nim__rimaries] } for primary
repeat
(z, v)A, (z, v) RandomLocations() ansitive_alphe negative aapha < for all i, (-a;)+ for all i
lestColorPatch f MonccromatiiCCooruccccc,,, (x, y) +
WeiihtedCColoPPacchhenn
alpha, (z, ()
matchColorPatch <- WaighhccCCoooloratttttsssss alpha, p. v))
testPercept t- ([[(tsstColooPatch)))
matchPercept + + (matchColorPacch)
errors[A] PercepuallEimorttssPeeeeppp (rr, y) matchPercept, (xc y)
coefficients a < GradientDescentt(y a, target, match, errors)
until convergence
if error 2 baseaa_e. cesse__rrors_ then
all tests boats_passed +- false
break
until trial MAX TRIALS
until all il_eests_pussed return num_primaries
spectral functions identical. This "base perceptual error" is larger than the energy of each color percept itself, meaning that the emergent vision fails to recognize a patch as the same color across different parts of the retina.
The second baseline model a variant of the first, where we change the optic nerve stream to directly transfer cone activation values, omitting the encoding complications of foveated warping and lateral
26

---

Published as a conference paper at ICLR 2025
Cone Activations as Incernal Percept
Abblaiiigg Demosaicing Network
1. I.ONS as Internal Percept
Base Error
ID
ID
50%
CMF-SIM
100%
output
2 3
Parceptual
Error
550%
5.00%
100%
100%
Base Error
0.00
400
Frequancy
760
Energy Primary)
Energy (0 Primary)
Perceptual Error
543mm
554mm
Primary
Frequency (pm)
nm
Figure 8: Comparison CMF- results of our baseline cortical models with limited or no inferential processing. In all of these models, the input optic nerve stream comes from retina with cone cells as in regular human trichromacy. 1. The first baseline model represents no cortical learning involvement in human visual perception, by setting the internal pereept directly equal the input optic nerve signal. The resulting base error, the perceptual error between the same color patch at two different retinal locations, results in higher errors, compared to energy the internal percept. This means that this cortical model fails to recognize the same color consistently across different patches of the retina, so it is procedurally impossible to even try color matching function tests against reference match colors. Formally then, this model fails resolve color vision. 2. In the second baseline model, we treat cone cell activations as internal percepts, omitting lateral inhibition from the eye simulation; CMF-SIM measures the resulting internal percept 1D color vision, failing to resolve trichromacy. 3. The third baseline model is an ablation model in which we omit the demosaicking network D from our proposed model of Section B CMF-SIM also measures the resulting internal percept as ID color. Red base errors are superimposed on energy / primary error plots for visual comparison, and 150% of these base errors are applied as threshold determination.
inhibition. This is intended as a far simpler encoding, to pressure test whether color vision can be detected in the spectrally encoded formally measures that this model results in ID color (Figure g 2), instead of the expected 3D color from such a retina. This result shows that cortical processing required even on cone activations to produce color vision of the correct color dimensionality.
values without further processing. However, CMF-SIM
The third baseline model adds another perspective by taking the full-featured model detailed in Section B but omitting the demosaicking network &D (as defined in Section B.2.2). As shown in Figure 8 3, CMF-SIM measures the resulting percepts of this model also as ID, providing evidence of the importance of demosaicking process in the emergence of color vision.
E VARIATIONS IN CELL EXPRESSION AFTER GENE THERAPY
As described in Section 6 we simulate experiments aimed at boosting color dimensionality Mancuso al. (2009). our simulation, approximately 60% of M cones are affected by gene therapy, with three possible scenarios for their modification. First, affected M cones are completely transformed into pure cones (Figure 1 l). Second, affected M cones equally express M and L opsins (F Third, affected M cones express M and L opsins in random ratios, that is aL + 3MM where and / spatially vary across the retina (Figure 9 3). In all cases, we confirm that our cortical model acquires 3D color vision after adaptation (i.e. learningg formally measured by CMF-SIN This makes intuitive sense if we consider the emergent vision from linear systems theory. Each different opsin can be interpreted as basis vector for the resulting linear color space. In this view, cells containing
27

---

Published as a conference paper at ICLR 2025
Complete Replacement
2. Eo-Expressoon
Varying Ratios
MS
LMS
MS
L
MS
Mixed expression of L&M
Sanatiiity
Somitneit
Sensitrity
Senutivity
hm
nmM
Fm
Tm
FmM
at L+pM
Pre-Therapy
Post-Therapy
Pre-Therapp
Post-Therapy
Pre-Therapy
Post- Therapy
All converges to color vision after post-therapy re-learning.
Figure 9: Three scenarios of cell expression after gene therapy adding a third cone type: 1. affected
M cones transform into L cones, 2. affected M cones express 50:50 M and L opsins, and 3. affected M cones express M and L. opsins spatially-varying, random ratios (i.e. a, B E [0, 1], + B 1). CMF-SIM shows that all cases converge to 3D color vision after post-therapy re-learning.
mixture of two cones (M and L in this case) can be interpreted as linear combinations of two basis vectors, which cannot increase the dimensionality of the linear space.
F PROBING EFFECT OF SPECTRAL RESPONSE AND ENVIRONMENTAL COLORS IN ACQUIRING TETRACHROMATIC VISION
In Section tests of tetrachromatic eyes assumed the fourth cone type was a pigeon Q cone, with peak sensitivity at 506 nm, between the S and M cone peak sensitivities (419 nm and 530 nm, respectively). In this section, we report additional experiments using instead a human Q cone, modeled with peak sensitivity at 545 nm between M and L peak sensitivities (530 nm and 560 nm, respectively). As shown in the spectral graphs of Figure [0] the pigeon Q cone is more decorrelated from L., M, S, which in principle may more easily support emergence of 4D color vision.
Further, we probe the effect of visual environment on the emergence of tetrachromacy, simulating the learning of color vision with three different input scene image datasets. The first environment is the set of real hyperspectral photographs of everyday scenes (Arad et al.) (022) as described in Section A. However, natural hyperspectral images are thought to be lacking in human tetrachromatic colors (Lee al. 2024). in the second and third datasets augment the 900 hyperspectral images with 100 tetrachromatic color patches. In the second dataset, the colors are sampled from the tetrachromatic hue sphere computed using recently developed N-dimensional color theory for tetrachromacy (Lee et al. 2024), customized to each of the observers here (i.e. with human Q or pigeon Q, respectively). In the third dataset, the colors are sampled from an idealized tetrachromatic color space in which each of the four cone channels is allowed to take any value, allowing theoretically maximum levels of chromatic contrast. Since natural hyperspectral images have been found lacking in human tetrachromatic colors (Lee et al.) 2024). in principle we may expect the likelihood that 4D color vision emerges to increase across these three simulated environments.
Indeed, in line with theoretical intuition, Figure 10 shows that 4D) color vision emerges, as measured formally by CMF-STM, only for the third visual environment the case of the human Q cone, but 4D color vision emerges with any of the visual environments for the pigeon Q cone. Our simulations suggest strong genetic and environmental effects on emergence of tetrachromatic color vision.
G FURTHER EVALUATION OF COLOR EMERGENCE CORTICAL MODEL
G.1 VALIDATION OF SIMULATED COLOR MATCHING FUNCTIONS AGAINST HUMAN PSYCHOPHYSICAL DATA
Our simulation produces results that are highly consistent with the psychophysical measurements of actual human subjects Specifically, we compared the output of CMF-SIM with the empirical color matching function data reported in Stiles & Burch (1955). To match the experimental setup in Stiles & Burch (1955). we employed the same monochromatic color primaries at 444nm, 526nm, and 645nm. As shown in Figure [1] the CMFs resulting from color matching simulation in CMF-STM
28

---

Published as a conference paper at ICLR 2025
Human Q cone
Pigeon Q cone
Gone Speceral Sensitivity
Cone Speceral
Human
Sensitivity
Pigeon
Hyperspectral
+ Hue sphere
LQMS full colors
Hyperspecoral
Hue sphere
LMQS full colors
3D
3D
4D
4D
4D
4D
CME-SIM outpuc
Base Error
Energy (0 Primary)
597mm
522mm
573mm
58 mm
607mm
585mm
Primary
502 m00m
532,620mm
49R.61 4nm
576mm
49 (51,5mm
460 581 mm
Primaries
4225118,584m
449.509 598mm
440,52 521 664mm
433,500
439,529,574mm
110,488.3 573mm
Primaries
442.433541.5999 597mmm
420.992.6446222m
454. $54.499,552600 20126601N0
54.52853357mmm
4+1. 191.997728.22mmm
437. 37.1188,333,57m
Primaries
Figure 10: Results of expanded experiments of boosting color dimensionality from 3D 4D, by addition of a Q cone to trichromatic L. M, S cones. Panel I models addition of Q cone from natural human tetrachromacy, and panel 2 is for a Q cone from pigeon vision; spectral responses are shown. Each panel compares emergence of color vision with three different visual environments: real hyperspectal images (Arad et al.) 2022); synthetic images containing tetrachromatic colors sampled from the hue sphere customized to that color observer (Lee et al., 2024); synthetic images containing idealized tetrachromatic colors with chromatic contrast between cone channels at the theoretical maximum. The simulation results show that 4D color vision only emerges for the human Q cone with the third environment, while it emerges for the pigeon Q cone under all three environments.
closely align with the empirical color matching function curves, providing further validation of our simulated cortical model an accurate representation of human trichromatic color vision.
G.2 ROBUST EMERGENCE OF CORRECT COLOR DIMENSIONALITY IN CORTICAL MODEL
The emergence of correct color dimensionality in our simulations remained consistent across varia- tions in initialization, designed mimic the natural biological variability between humans (Carroll et al., 2002 Hofer et al. 2005). We tested this rigorously by varying both cone type ratios and initial noise parameters. First, we altered the L:M cone ratios in the retinal patch, testing 2:1 (original), I:1, equal L:M:S ratio of 1:1:1. In all cases, the framework consistently converged and 1:2, as well demonstrating adaptability to different cone distributions. Second, we sampled to 3D color vision, the random seed for noise parameters, including: photon-shot noise during photoreceptor activation. lateral inhibition noise, Perlin noise for cell position randomization, and cortical model parameter initialization. Despite these changes, the model always converged to 3D color vision when trained
29

---

Published as a conference paper at ICLR 2025
Generated Color Matching Functions from our Simulated Cortical Model
CMFs from Stiles & Burch (1955)
Coefficients 2
2
0
400
500
600
700
400
500
600
700
Wavelength (nm)
Wavelength (nm)
Same primaries used: 444nm, 526nm, 645nm
Figure 11: Side-by-side comparisons of two color-matching functions (CMFs) are presented. Left: The CMFs generated from simulated cortical model, trained using trichromatic retina. Right: The CMFs obtained from Stiles & Burch comparison, our simulation used the set of spectral primaries (i.e., 444nm, 526nm, and 645nm). This comparison highlights the validity our trained cortical model as an accurate representation of color observer.
Burch] (1955), based on real human subjects. To ensure direct
with trichromat retina, underscoring the robustness of the simulated cortical learning to biological and physical variability.
To further illustrate the robustness of our cortical model, we present its performance across diverse range of input stimuli during testing, as shown in Figure [2] The figure demonstrates the Neural- Scoped internal percept of our cortical model trained with trichromatic retina. Specifically:
1.
For hyperspectral images derived from RGB inputs, converted using the method described in Section A.I]] the model successfully reconstructs accurate color percepts (3rd column) from optic nerve signals (2nd column), which have been time-averaged for improved visualization. Additionally, the model demonstrates complete robustness to various hue variations, accurately reconstructing color percepts even under significant changes in the chromatic properties of the input stimuli.
2. For hyperspectral images from a standard dataset (Arad et al. 2022). the model accurately reconstructs color percepts (6th column) from optic nerve signals (5th column). To aid clarity, we include RGB-projjcted versions of the hyperspectral images in the 4th column.
This highlights the model's ability to generalize across varying input types and maintain robust performance under diverse conditions.
H TABLE OF IMAGES WITH VARYING NUMBER OF CONES FOR PHOTORECEPTOR ACTIVATIONS AND RGC SPIKES
To enhance intuition and visual clarity. we present a table in Figure 13 where the number of cone types in the simulation varies across columns. The rows display images of photoreceptor activations. bipolar signals, and optic nerve signals.
The second row Figure 13 reveals that photoreceptor activations become increasingly noisy as additional cone types are added to the retinal mosaic. The third row shows bipolar signals, computed by applying center-surround lateral inhibition kernel greater complexity compared to the photoreceptor layer. By the time optic nerve signals (spikes) are generated, differences between cone mosaics become almost indistinguishable, underscoring the cortical challenge of extracting color vision with the correct dimensionality.
photoreceptor activations, introducing
also present a variant of Figure 13 that features the balloon image in Figure
30

---

Published as a conference paper at ICLR 2025
Learned internal percept of our cortical model trained with trichromatic retina
1. Hyperpppccrlllccoeerrde RGB image input stimulus
Hyperspectral image input stimulus
Scene Stimulus (RGB-projected)
Time-averaged optic
Neural-Scoped internal percept
Scone Stimulus (RGB-projected)
Time-averaged optic
Neural-Scoped internal percept
Our Icarned cortical model demonstrates robust performance actoss wide variety of inputs during testing. Here, we present the neural-scoped internal percept of our cortical model trained with trichromatic retina: 1. For hyperspectral images derived from RGB inputs (Section A. the model accurately reconstructs the color percept (3rd column) from optic nerve signals (2nd column, time-averaged for clearer visualization). 2. For hyperspectral images (Arad al. (022), the model similarly produces accurate color percepts (6th column) from optic nerve signals (5th column). For visual clarity, we show -rojjereet hyperspectral images in the 4th column.
Figure 12:
31

---

Published as a conference paper at ICLR 2025
Monochromat (M cone only)
Dichromat (M S cones)
Trichromat
(LMS+Pgeon Q)
Cone Mosaic
GGG
Photoreceptor
Activations
GGGG
Bipolar Signals
GAGG
Optic Nerve Signals
Figure 13: Expanded version of Figure 4, showing the full field of view with simulated retinal
ganglion cell outputs (i.e., spiking optic nerve signals in the last row). Red boxes highlight close-up details of the full-sized data near the left eye of the Mona Lisa. L, M, S, and cones visualized as red, green, blue, and black, respectively, in the first row.
32

---

Published as a conference paper at ICLR 2025
Monochromat (M cone only)
Dichromat (M $ cones)
Trichromat
Tetrachromat (LMS+Pigeon Q)
 GOPO
Cone Mosaic
Photorsceptor
Activations
GAGO
Bipolar Signals
GAGG
Optic Nerve Signals
Figure 14: Expanded version of Figure 44 variant of Figure 13
featuring the balloon image.
33


Reviews:
[{"review_id": "05dBJwRgBs", "reviewer": "Reviewer_1FHw", "summary": "The authors claimed that they reverse engineered how color vision may emerge in a biologically inspired cortical learning model. They use simulated optic nerve signals (ONS) as input and trained neural networks to individually approximate three invariant properties of the retina: cell positions, cone spectral type, and magnitudes of lateral inhibition. Their model optimizes the error between the predicted ONS signal at the next sampling step and the real simulated ONS signal. They showed that during the minimization of error between inferred ONS and real ONS, the internal perception of their model gradually adds individual primary colors. To show the aggregation of primary colors, they probe the learned embedding with another separately trained model; they call this separate color matching model CMK-SIM. This CMK-SIM is based on text book colorimetry. They use the observation that CMK-SIM found three primary colors in their trained model (instead of the raw optic nerve signal as the model input) as the success of recovering color vision.", "questions": "A few additional questions: \n \n1. What is $N$ in the cortical learning model? What would happen if the model directly set $N=3$? \n \n2. Does the $NX3$ matrix (neural scope) hold constant in all the shown examples (like Figure 3.3) or is this matrix refitted every single time given the different internal percepts? \n \n3. Would the authors comment on why it takes additional 99K steps to learn the 3rd color? Would this (#steps) be different if there were only 2 colors? \n \n4. If I read the manuscript correctly, the visual percepts are the output of U-net? \n \n5. U-net (the backbone for demosaicking here) contains substantial inductive bias in its architecture. Is the separate translation operator a model merely a biologically inspired model choice? It seems that the model is trained on pairs of data (the input ONS is from time t and the output ONS is from $t+\\Delta t$) instead of sequences. If U-net absorbs the learning of $\\Omega$, would it change the performance? I understand eye movements shift ONS, but is it necessary to map it into a separate subfunction?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors claim that the main novelty of the paper is the framing of the emergence of color vision as a learning problem. I agree the machine learning framework includes a couple biologically inspired constraints. I would appreciate some clarifications to better understand the significance of the cortical learning model.", "weaknesses": "The technical details of this paper are a bit hard to follow (those missing ?? reference in supplementary do not help). For example, I would appreciate further clarification on how the authors choose their baseline models. The authors merely took the entire cortical learning model out (baseline I) or simplified their ONS signal (Baseline II). I guess color/cone spectral type information is present in the ONS signal? Can the author verify this (although CMK-SIM choose to only use 1 color)? \n \nNow another possibility is that the U-net architecture the authors use for the C and D gets way overfit (for example, [1] showed that U-net can generate natural image from white noise inputs and its overfitting occurs between 2400 and 50K steps. The authors here use 100K steps) to recover 3 colors. Considering how much inductive bias is there in the deep architecture, this large number of iterations makes it unclear *where* exactly the color dimensionality emerges. Although the authors postulated the decoder and encoder into factorizable components. There are no ablation studies showing how different learnable components contribute to the emergence of color vision (or if this is not possible given how intricate these components are, would the authors please clarify?)\n\n[1] Ulyanov et al., Deep Image Prior https://arxiv.org/abs/1711.10925\n\nIn fact, recent applications of U-net [2] [3] rarely exceeds 5000 steps and they take advantage of its inductive bias on natural images. My main hesitation to fully agree with the results is that the model seems to operate in the \"over-fitting\" regime to recover the 3rd color. \n\n[2] Falck et al., A Multi-Resolution Framework for U-Nets with Applications to Hierarchical VAEs NeurIPS 2022\n\n[3] Wang, et al., Learning low-dimensional generalizable natural features from retina using a U-net NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qb3OEMA9eP", "reviewer": "Reviewer_L1yp", "summary": "The authors propose that realistic (because of being 3-dimensional) encoding of color percepts emerge assuming a certain number of spectral sensors and temporal predictive coding. They explore this idea with different number of spectral sensors (or kinds of cones).\nIn their proposal, first, they learn the different elements of the encoding function by maximizing the consistency (minimizing the error) between the signal captured at time t+dt and the prediction computed from the signal at time t and the displacement information between saccades. Then, the dimensionality of the color percept is defined as in classical colorimetry by looking for the minimal number of independent sources that need to be changed to make a color match (to make the encoded responses the same).", "questions": "* Authors claim the emergence of color vision. Please tone down / reformulate this contribution. What you get is the dimensionality of the color space. \n\n* The limitation of the 3-d result should be acknowledged and discussed in the work. (My guess) is that richer training and more general testing (e.g. asymmetric matching) would lead to larger dimensionality.\n\n* Cite alternative principles that can be used to derive the number of necessary sensors to encode spectral (or chromatic) information.\n\n* How do you reconcile these two apparently contradictory sentences? (line 23-25): \"When the retina contains N types of color photoreceptors, our simulation shows that N-dimensional color vision naturally emerges\". (line 352): \"CMF-SIM reveals the color dimensionality is 1-D for observers with 3 cone types.\" \n\n* The Color Matching Function Simulation in the main paper is only explained in FIg.4 and this may not be enough for the average ICLR reader.\n\n* I missed the explanation of the Neural Scope. Note that the generation of images that simulate the perception of dichromats is not trivial (Brettel et al. JOSA 97, Capilla et al. JOSA 04).\n\n* Please do not refer to supplementary videos to explain concepts (as in line 64 of fig.1 or as in line 201 of the supplementary material). If extra explanations are required, please do it through additional sections in the supplementary material. \n\n* Please define the acronyms the first time they appear. Pay special attention to CMF-SIM and NS. I think CVD is not even defined.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is original in proposing predictive coding as a way to fit the elements of color coding. \nAnother originality is bringing to the machine learning community a concept from vision science: using a response matching experiment to define the dimensionality of the encoding space for certain concept (in this case, the color percept).\nHowever some of the results obtained from this original view seem obvious (see below).", "weaknesses": "* Authors claim the emergence of color vision (e.g. titles of the section 4 and 6), however what they reproduce is just the dimensionality of the color space, or the number of illuminants required for response matching. This dimensionality (even if correct) is not a full description of color vision, and should not be reported as that. For example, the dimension does not explain the positive and negative lobes of spectral sensitivity of spectral sensors [Wiesel&Hubel J.Neurophysiol.66], it does not explain their adaptivity under change of illuminant [Von Kries 1902], nor the nonlinearity of their response [Krauskopf & Gegenfurtner JOSA 92], nor the non-Euclidean nature of subjective distances between colors [MacAdam JOSA 42], to cite a few facts. \n\n* Moreover authors refer that the dimension of color space is 3 when assuming 3 different spectral sensors. That is consistent with the result of classical *symmetric* color matching experiments with colors seen in *isolation*, the paradigm of classical tristimulus colorimetry [Wyszeki & Stiles Wiley 82]. However, when seing colors *in context*, the independent dimensions are not 3 but at least 5: not only brightness, hue and colorfulness, but also lightness (not the same as brightness), and chroma (not the same as colorfulness) [see Fairchild, Color Appearanace Models, Wiley 2013]. Therefore, the chromatic content of the spatial context (not only the number of sensors) is critical for the emergence of this 5-dimensional nature, and this is not mentioned in the work. This 3d limitation may come from training limited to regular (white / daylight) illumination, and testing using symmetric color matching with simple (isolated flat) samples, as in the CIE1931 colorimetry. This limitation should be acknowledged and discussed in the work.\n\n* The authors say that previous literature always assumed the dimensionality of the color space. However, there is literature, based on different principles, that can be used to derive the minimal number of sensors and their spectral sensitivity. For instance, [Singh et al. Proc. 3rd Int’l Workshop on Statis. Comp. Theories of Vision 03, Jimenez & Malo IEEE Trans.Geosci.Rem.Sens.14] develop expressions for the accuracy of the recovery of the reflectance of the surfaces and spectrum of the illuminant depending on the use of spatial information and on the number and spectral sensitivity of the sensors. These represent a different principle (inference of object reflectance with color constancy) that can be used to set the dimensionality of the color encoding in a different way as that proposed by the authors.\n\n* The proposed method gives obvious results at some points and these are contradicted at some other points. An example of obvious result is what is said at the abstract (line 23-25): \"When the retina contains N types of color photoreceptors, our simulation shows that N-dimensional color vision naturally emerges\". However, a later statement seems to contradict the above: in line 352: \"CMF-SIM reveals the color dimensionality is 1-D for observers with 3 cone types.\" \n\n* Methods as the Color Matching Function Simulation or the Neural Scope are not clearly explained in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7wuoU2q6Ta", "reviewer": "Reviewer_Xmgd", "summary": "This paper presents a novel computational framework for understanding the emergence of color vision in the brain, centered on a self-supervised learning model that interprets optic nerve signals without predetermined color dimensions. The framework includes a simulation engine that processes spectral images through realistic eye mechanisms, including eye motion and cone cell sampling, to generate optic nerve signals. The model demonstrates how different types of color vision (monochromatic, dichromatic, trichromatic, or tetrachromatic) can naturally emerge based on the number of cone types in the retina without requiring predefined color encoding. Notably, the framework successfully models enhancement scenarios, such as the conversion from dichromatic to trichromatic vision demonstrated in squirrel monkey gene therapy experiments. It suggests the possibility of enhancing human color vision from trichromatic to tetrachromatic.", "questions": "1. In part 1 of Figure 2, fixational eye movements are modeled as small drifts controlled by a uniform probability distribution (*dX*, *dY* ) ∼ *U*{(−15, 15) × (−15, 15)}. I watched the supplementary vedio about the model, it appears (if I'm correct) that the drift pattern displays frequent small adjustments with occasional larger movements. Could you clarify how the uniform probability distribution achieves this effect, given that it should theoretically produce equally likely shifts across the specified range? And why do you choose 15-pixel as the range of movements?\n2. I am also curious about the drift pattern moving from the eyes to the mouth. How do the fixational eye movements capture both the eyes and the mouth?\n3. I suggest authors review the use of LaTeX \\ref{} commands, as there are missing figure numbers in several instances (e.g., lines 71, 141, and 180) in the Supplementary Materials.\n4. In your model, could you explain the process used to convert the \"scene stimulus\" into \"incoming spectra\"? Specifically, I'm interested in whether you apply any spectral sampling or decomposition steps and if there are transformations based on the eye’s optical properties to simulate the spectra reaching the retina.\n5. Could you clarify how cone cell activation is calculated based on the received light? I am wondering whether you apply spectral sensitivity functions for each cone type (L, M, S) or any other specific computational approach to simulate how cones respond to incoming light.\n6. ON/OFF pathways in bipolar cells and the center-surround organization in RGCs work together to enhance contrast and edge/contour detection, but in your Figure 2 part 3, I don't see the edge or contour section of the scene has been enhanced?\n7. Could you clarify how lateral inhibition works among cone cells in your model? If I understand correctly, this mechanism balances the activation of different cone cells and adjusts their response to light, and then enhances edge and contour. Additionally, could you provide more details on how cone cells, bipolar cells, and RGCs are connected, especially in terms of how these connections contribute to spike generation? I believe these details would be helpful to discuss.\n8. In Part 1 of the simulation engine of biological eyes, are there any specific learning rules or algorithms used to adjust parameters or process visual information? \n9. Why do you choose the standard deviations of 0.15 and 0.9 in DoG kernel?\n10. Could you elaborate on the mechanisms that contribute to the emergence of color representation in your model? From an intuitive perspective, I’m curious how the self-supervised learning and prediction error minimization guide the model toward developing a stable and structured color space, especially as it expands beyond initial color dimensions.\n11. Concerning Questions point 10, the model predicts a 4D color space (LMQS) from 3D RGB color images. Since we don’t have four types of cones, we can’t truly envision or confirm what 4D images would look like. Have you tried using 2D colorful images to predict a 3D color space? This approach could help test and validate the model’s predictions.\n12. According to Weaknesses point 3. I'm interested in understanding whether the mechanisms in your model might have implications for curing color blindness, particularly in primates such as squirrel monkeys. If the color representation depends on ventral pathway of V1-V2-V4, do you think the your model could potentially inform approaches to treat or \"cure\" color blindness?  I’d appreciate any insights you could share more information with me.\n13. A typo \"investigated\" on line 156 should be corrected.\n14. I don't see the code releases.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposes a new model that can simulate the emergence of color vision through a cortical learning mechanism. This model uses self-supervised learning to develop color representation from optic nerve input, demonstrating how color perception emerges naturally from visual data without labeled learning.\n2. By simulating color vision without explicit biological pathways, the authors propose a new theory of how color vision could emerge in the brain, suggesting that specific neural pathways may not be strictly necessary for color representation.\n3. The model’s insights could inform research into treating or understanding color blindness, possibly providing theoretical guidance for therapeutic approaches.", "weaknesses": "1. The paper proposes a cortical learning mechanism for color representation without explicitly mimicking the V1-V2-V4 pathway commonly associated with color processing in biological systems. Does this imply that, in your view, the V1-V2-V4 pathway may not be strictly necessary for achieving color representation? I’m curious if your findings suggest that color perception could emerge through other mechanisms or pathways outside of this traditional hierarchy.\n2. Does the model in your paper learn directly from RGB format images? If so, how does it expand from the RGB color space to a 4D color space within the learning process? I’m curious about the mechanisms that allow the model to infer or represent a 4D color space based on RGB input.\n3. While your model provides an explanation for the emergence of color representation, I find it challenging to accept that the V1-V2-V4 pathway might not play an essential role in color perception. For instance, the study by Li, Liu, Juusola, and Tang (2014) (see below) on perceptual color mapping in macaque visual area V4 underscores the significance of this pathway in color processing. Could you clarify how your model accounts for or relates to this traditional neural pathway?\n\n> Li, M., Liu, F., Juusola, M., & Tang, S. (2014). Perceptual color map in macaque visual area V4. _Journal of Neuroscience, 34_(1), 202–217. https://doi.org/10.1523/JNEUROSCI.4549-12.2014", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rA1aIDXMCH", "reviewer": "Reviewer_bhn4", "summary": "The manuscript introduces a new approach to computational modelling of colour vision, combining established knowledge of human visual systems with modern artificial neural networks. The proposed framework suggests that the cortex (the agent’s processing unit) lacks direct information about the retina (the agent’s sensory hardware) and investigates the types of colour perception that could arise in such a setup. The model is trained using a biologically realistic principle: predicting future optic nerve signals, aligning with predictive coding theory, a successful approach to unsupervised learning in vision research. To examine the internal representations created by this model, the study employs psychophysical tests commonly used in human colour-matching experiments. The results provide interesting insights into the dimensionality of colour perception at the single-pixel level, offering promising avenues for future exploration into the possibility of tetrachromacy in humans.", "questions": "Below is a list of not only questions but also general comments:\n\n### Optic Nerve Signals\n- I think the depiction of optic nerve signals in Figure 1 and the supplementary video should include the idea of distinct photoreceptor cells, which create separate pathways, like parvocellular and magnocellular pathways.\n- Following the previous point, in the simulation, if I understand correctly, the optic nerve signal (ONS) is modelled as a 2D matrix, which closely resembles raw images from cameras without further processing. However, since photoreceptor positions are known, a more accurate simulation might use a 3D matrix with the third dimension representing photoreceptor channels. This would mean at each spatial location only one channel contains information (light absorption values). I’d be interested in your thoughts on this.\n- In the statement, \"[...] colors of the scene are spectrally sampled by different types of color sensitive cells (cone cells) in the retina, appearing as a layer of spatial noise in the ONS [...]\", I would argue that raw photoreceptor signals, while noisy, are not spatial noise but spatially meaningful. This is reflected in Figure 3, \"True ONS at time t\". I think Figure 1 should display a similar visualisation rather than the highly noisy image currently shown.\n\n### Colour Decoding or Encoding?\nThroughout the manuscript and in the supplementary video, the authors describe human colour vision as decoding colour. For instance, the abstract begins: \"It is a mystery how the brain decodes color vision purely from the optic nerve signals it receives, with a core inferential challenge being how it disentangles internal perception with the correct colour dimensionality from the unknown encoding properties of the eye.\"\n\nI’d suggest that we do not decode colour vision but rather construct it (in a sense, we encode colour to interpret the world more easily). It is not only from optic nerve signals; our colour perception is heavily influenced by semantic and cognitive processes. I don't mean to play with linguistic terms but raising a conceptual difference. For instance, in my view, the function Phi in Figure 3 \"encodes\" the internal percept and the CMF-SIM block shows different potential decoding of the internal percept. Currently, Phi is based solely on ONS, but a more complete model would incorporate input from cognitive processes that impact how we encode colour information of a scene.\n\n### Measurement of Colour Dimensionality\nI agree that colour is represented in the cortex as a high-dimensional space, $R^N$, but I think $N$ includes more than mono-, di-, tri-, and tetrachromacy. It also encompasses other aspects of colour appearance that are independent of photoreceptor dimensionality, such as transparency, glossiness, etc. In essence, properties of colour perception often relate to objects rather than individual pixels (e.g., see \"Colour appearance and the end of Hering’s Opponent-Colors Theory\", *Trends in Cognitive Sciences*, 2023, and \"Colour perception: objects, constancy, and categories\", *Annual Review of Vision Science*, 2018). I’m curious if you’ve analysed different dimensions of the internal percept in the cortical model, as hypothesised, and whether similar concepts emerged there.\n\n### Why Did Human Colour Vision Emerge?\nOne of the manuscript’s interesting findings is that monochromacy, dichromacy, trichromacy, and tetrachromacy emerge in sequence during the learning steps. This suggests that colour space dimensionality increases evolutionarily without altering the ecological conditions of the system (i.e., the system’s interaction with its input remains the same through the learning process). In contrast, the colour vision dimensionality in biological organisms varies based on their environmental needs. For example, human trichromacy is thought to have evolved for tasks like distinguishing ripe fruit and foliage or recognising emotional states in faces. In the manuscript’s unsupervised learning model, the system’s needs remain constant, yet the colour vision dimensionality expands as learning progresses. I have a few questions about this:\n   - What drives this change in dimensionality within the network’s colour vision?\n   - How consistent is this emergence? If you repeated the process 100 times, how much variation would appear in the resulting colour vision systems?\n   - Why does this evolution stop at four dimensions rather than expanding to five or more?\n\n### Minor Points\n- What does a \"private-line\" visual pathway refer to?\n- In Figure 2, for consistency, I suggest using the same balloon image as Figure 1 to clarify the model’s detailed processes.\n- In the context of this statement, \"[...] random distribution of L, M and S cone cells on the retina with a relative probability of 0.63, 0.32 and 0.05 [...]\", it might be helpful to mention the large individual variation in these probabilities, which significantly impacts individual colour perception. Have you experimented with different distributions, and could this framework contribute to understanding this variation?\n- In the context of this statement \"Specifically our DoG kernel has standard deviations of 0.15 and 0.9 cone diameters, respectively, for the positive center and negative surround Gaussians [...]\", the authors might consider modelling dynamic, contrast-dependent centre-surround modulation of lateral inhibition (e.g., see physiological studies by Alessandra Angelucci). This could enhance the efficiency of DoG models (e.g., \"Colour constancy beyond the classical receptive field,\" *PAMI*, 2017, and \"Feedback and surround modulated boundary detection,\" *IJCV*, 2018).", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Overall, I found the article very interesting with several fascinating ideas. Its greatest strength lies in its innovative thinking, presenting a fresh computational approach—sensory-agnostic perception—that stands apart from prior models.", "weaknesses": "I would summarise the current weaknesses of the manuscript into three areas:\n\n1. **Presentation**: While generally well-written and easy to follow, some parts of the manuscript require considerable effort from readers to fully understand. For example, abbreviations like \"CMF-SIM\" and \"NS\" are not immediately clear. Later, NS is explained as \"Neural Scope,\" but CMF-SIM remains unclear. In the supplementary materials, the \"colour matching function\" is mentioned, but what about SIM? This ambiguity demands extra effort from readers. Also, there are multiple unresolved cross-references (\"??\") in the supplementary materials.\n\n2. **Results**: The results section could benefit from being more thorough. Specifically, a broader set of analyses across multiple training instances would strengthen the rigour of the findings and clarify them for readers.\n\n3. **Related Work**: Although the manuscript introduces a unique approach that differs from previous models, a brief section in Related Work could discuss other methods using artificial neural networks to model aspects of human colour perception, such as colour categories, contrast sensitivity in chromatic channels, and colour opponency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7qwrilNMaj", "reviewer": "Reviewer_c2wB", "summary": "This paper presents a computational framework to explore how color perception might emerge in the cortex through a self-supervised learning model inspired by biological vision. The authors aim to simulate the early visual processing stages by building a model that integrates a retinal layer with different types of cone cells (sensitive to various wavelengths) and a cortical layer using predictive coding. The model learns to extract color and shape information from visual inputs without explicit color labels, reflecting a form of emergent perception.\n\nA significant contribution is the model's ability to simulate the transition from dichromatic to trichromatic vision, akin to genetic interventions shown in experimental biology. The framework demonstrates that under different cone configurations, the model can adapt to various perceptual dimensions, which provides insights into color processing mechanisms in the brain. This interdisciplinary approach contributes to our understanding of self-learning in color perception and offers a basis for future studies on adaptive vision systems in neuroscience and AI.", "questions": "1 There are many parameters in the model, especially in the cortical learning layers. How were these parameters selected, and are they based on biological references or empirical tuning?\n2 The experiments are mostly theoretical, focusing on emergent color dimensions in simplified scenarios. Have the authors considered testing their model on more complex datasets or tasks, such as object recognition with varying lighting conditions?\n3  The model relies on self-supervised learning for color perception emergence. Biological systems often use feedback and interaction between cortical areas, which may influence color perception development. How do the authors justify the exclusion of such feedback mechanisms?\n4 The paper mentions using divisive normalization for edge detection in the model but does not detail why this specific normalization was chosen over others. Could the authors elaborate on why divisive normalization aligns best with their framework, or if they explored alternatives?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The originality stems from the integration of self-supervised learning with a computational framework inspired by biological color perception. By combining a retinal model with a cortical learning mechanism, the authors attempt to create a biologically plausible framework for emergent color and shape perception. This interdisciplinary approach bridges machine learning with neurobiological modeling, which is a valuable exploration in color perception studies.\n2 The paper’s methodology reflects a reasonable approach to simulating early visual processing stages. The authors carefully design the retinal model to simulate color cone responses and introduce predictive coding for the cortical learning mechanism. Though some limitations exist, the study provides valuable insights into the use of predictive coding in visual processing models, highlighting how cortical networks could feasibly learn color information.\n3 The paper is generally well-written, providing a clear overview of the methods, experiment design, and results. Figures and visualizations are well-crafted, aiding in understanding the experimental findings. The authors successfully explain complex processes, such as predictive coding and color perception, in an accessible way.\n4 This study is significant within the niche area of computational neuroscience focusing on color perception. It addresses the unsolved question of how color perception might emerge in a self-learning cortical network, which is a meaningful contribution to understanding visual processing mechanisms. The attempt to simulate gene therapy’s effect on color perception (such as enhancing dichromatic to trichromatic vision) offers a unique perspective that could inspire further studies in perceptual enhancement.", "weaknesses": "Biological Plausibility of Input Data:\nThe model uses natural RGB images as input, which already contain color information. This approach diverges from biological realism, as the retina does not process pre-encoded color data but rather interprets light of various wavelengths. To improve biological plausibility, the model could use spectrally-encoded images or raw spectral data as inputs to better reflect how retinal cones process different wavelengths. This adjustment would align the model closer to real visual processing and allow more meaningful exploration of emergent color dimensions.\n\nValidation against Biological Data:\nWhile the paper presents a computational framework, it lacks direct validation with biological data, which is essential to support its claims of modeling emergent color perception. Adding quantitative comparisons with known biological data on cortical color responses, such as V1 or V4 neuron responses in primates, would strengthen the study’s credibility. Incorporating benchmarks like psychophysical data on color matching could also make the findings more robust and biologically grounded.\n\nAmbiguity in Learning Mechanisms:\nThe model’s reliance on predictive coding for learning color dimensions in the cortex is interesting but raises questions about biological accuracy, as it assumes self-supervised learning can fully capture color processing. However, in biological systems, factors like feedback from other cortical regions and adaptive changes play significant roles. Expanding the model to incorporate feedback loops or other cortical interactions could offer a more comprehensive view and make the learning process more aligned with biological color perception theories.\n\nOverstated Novelty in Emergent Color Dimensions:\nWhile the paper claims novelty in emergent color dimensions, related work on unsupervised learning of color and perceptual features is well-established. For example, studies on hierarchical models of color vision (e.g., Komatsu, 2006; Conway et al., 2023) and other unsupervised learning models in color processing (Zhou & Mel, 2008) address similar concepts. The authors could enhance clarity by explicitly discussing these foundational works, explaining how their approach extends or diverges from these models, and adjusting novelty claims accordingly.\n\nKomatsu H. The neural mechanisms of perceptual filling-in. Nature reviews neuroscience. 2006 Mar 1;7(3):220-31.\nZhou C, Mel BW. Cue combination and color edge detection in natural scenes. Journal of vision. 2008 Apr 1;8(4):4-.\nConway BR, Malik-Moraleda S, Gibson E. Color appearance and the end of Hering’s Opponent-Colors Theory. Trends in Cognitive Sciences. 2023 Sep 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)