Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary
-------------
 
The authors analyze the quality of known color representations (non-opponent and opponent) in terms of the quality of the reconstructed images when spatio-chromatic information is constrained by a bottleneck of a discrete (quantized) representation of reduced dimensionality. 
 
The quantized representation and encoding-decoding transforms are defined by a loss function optimized through an autoencoding tool (in particular a Vector-Quantized Variational Autoencoder). The quality of the reconstructed images is measured in terms of (1) low-level similarity metrics (some of them with perceptual meaning), and (2) performance in higher-level tasks such as classification and segmentation. 
 
The conclusion is that perceptually meaningful opponent representations such as DKL and CIELab are better suited to the imposed bottleneck as opposed to color representations related to retinal sensors such as RGB or LMS. 

General opinion and recommendation
-----------------------------------------------------

I think the metodology and findings are really interesting to understand why the brain may have developed the opponent 
representations that have better performance in the presented experiments. The use of an autoencoding tool to enforce the minimization of the loss suggests that comparison between the color representations is fair.  

However, this message (pointing out the advantages of opponent representations wrt more trivial non-opponent representations) is not clearly stated. The biological meaningfulness of the selected constraints is not discussed either. Presentation is confusing at many points (see specific list below), but this can be fixed. Therefore,  I think the work should be accepted after proper clarifications and removing some misconceptions.

Major Points
----------------

The goal of the paper (in my view, a fair comparison of different color spaces in a bottleneck context using an appropriate optimization tool) is not clearly stated.

Instead some sentences in the abtract and introduction of the paper suggests that color representation is learnt. For instance, in the abstract and intro it is said "We propose a novel unsupervised task —colour conversion— to explicitly examine the colour representation learnt by deep networks (referred to as ColourConvNets)." and "the structure of internal representation provides insights on how this [color] transformation is performed within a neural network". At this point the reader may think that the proposed autoencoder will learn a specific color representation well suited for certain goal(s). However, the autoencoder is not learning color representations (they are imposed at input and output), the autoencoder is only imposing certain bottleneck and hence, it is a controlled way to assess the suitability of the considered color representations in constrained settings. In fact, spatial and chromatic parts of visual information are mixed in the vectors of the inner representation of the considered autoencoders. This (hard to interpret) mixture necessarily comes from the reduced dimension of the vectors. Therefore, strictly speaking, there is no "pure color representation learnt" in the inner representation of the autoencoder. This misunderstanding about "learning a color representation" when it is actually a spatio-chromatic representation imposed by the constraints and the selected input-output color spaces happened to me, and only disappeared at the end of page 3 and 4. This misunderstanding should be avoided in abstract and intro.

Another confusing description is talking about the "correlation" and "decorrelation" properties when presenting the considered spaces (in section 2.2) just after talking about learning efficient representations through the autoencoder. Mentions to the "efficiency" of color space through citations to [Buchsbaum83,Ruderman98,Lee01] should appear later in the discussion (not as early as in section 1.1).
Readers aware of Barlow's efficient coding hypothesis that leads to transforms that favour decorrelation and equalization
(which in color lead to PCA-like transforms [Buchsbaum83], and nonlinear equalizations that explain chromatic adaptation [Laparra12]) may wonder why the cost function did not include decorrelation or independence measures. I would suggest talking about the decorrelation properties only in the discusion (do not mention in section 1.1 and remove the left part of fig 2 -devoted to highlight correlation and decorrelation-, and make these points in an expanded "performance advantage" section). Actually, decorrelation and equalization properties of these color spaces has been measured in information theoretic units by Foster et al. 2008 and by Malo 2020. I think these references [Buchsbaum83,Ruderman98,Lee01,Foster08,Laparra12,Malo20] should be included in this decorrelation-equalization discussion.

Another confusing statement is the apparent contradiction between this statement "the conversion of RGB images
into other colour spaces yield to no performance improvement in ImageNet (Mishkin et al., 2017)" and the interesting 
findings done in this work. It is important to stress that maybe results in (Mishkin et al., 2017) were not subject to big enough dimensionality constraints and then proper representation of color was not that relevant.

It is unclear how Fig 8 and Fig. C.1 were computed. To me this is really important to clarify the non-trivial mixture of spatial and chromatic information in the vectors. From the text, I guess, a single codevector was used to decode the image.
But, given the dimension of the codevectors (bigger than 3), they encode not only chromatic information but also spatial information. Then, how is it possible to obtain uniform color (as in figs 8 and C.1) with no spatial variation from a single codevector? 

Minor Points
------------

* it is important to stress that the advantage of the proposed metodology to compare color spaces is that additional constraints can be included (such as entropy, energy, wiring, etc...). This framework able to encompass additional constraints is relevant to understand why the considered representations could have emerged in the brain.
 
* Correlation = 1 between L and M seems like too much. Is this correct?

* The value of the loss function is comparable in the different cases after training? This would be necessary for a fair comparison, isnt it?

References:
----------------

[Foster08] D.H. Foster, I. Marin-Franch, and S.M.C. Nascimento. 2008. Coding efficiency of CIE color spaces. In Proc. 16th Color Imag. Conf. Soc. Imag. Sci. Tech., 285–288

[Laparra12] V. Laparra, S. Jiménez, G. Camps and J. Malo. 2012. Nonlinearities and adaptation of color vision from sequential principal curves analysis. Neural Computation 24, 10 (2012), 2751–2788

[Malo20] J. Malo. Information Flow in Color Appearance Neural Networks. Accepted in Entropy Conference 2020 https://arxiv.org/abs/1912.12093
review_title: Interesting assesment of color representations using autoencoders

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to study an interesting problem of how color informaiton is structured in the variational autoencoders (VAEs). Several instances of VAEs are trained in an unsupervised manner to perform color space conversion. Both low-level and high-level evaluations are performed to study the local statistics and global content of converted images. Several interesting conclusions are drawn from the experiments that help interpret the encoding process of autoencoders.

Overall, this paper studies an interesting problem and presents several insightful conclusions. My only concern is how significant the proposed method/task is, and how significant insights these conclusions could provide. To address this, it may show some applications based on the proposed task/conclusions.


I only have some minor issues.

1. It could be better to include results of state of the art methods on, e.g., object classification and scene segmentation. This could further show the potential application of proposed method.

2. encode -> encodes, Line 7, Page 2

3. Figure 1 is not referred to in the main text. Besides, it could be better to prodive more details in the caption.
review_title: review 3765

final_score: rating: 4: Ok but not good enough - rejection
review_content: The motivation for this paper is quite hard to understand. A VQ-VAE is directly applied to convert an image from one colour space to another one. However, the colour space transform is human-defined, usually involving linear and a few non-linear (like selecting the maximum value is HSV) procedures. In this case, the latent space of VQ-VAE should be collapsed into this simple equation easily. The analysis of this paper does not teach us any additional knowledge.
The motivation of finding a better embedding space of colour is admirable, unfortunately, the analysis and methodology does not support the motivation. 
review_title: The motivation is unclear and non  additional knowledge is given

Meta-review: This paper proposes a novel unsupervised task of colour conversion. In this respect, the task
becomes more like a regression problem -- rather than autoencoding the decoder needs to reconstruct the pixels in a different color system. 
While the idea is potentially interesting, there are fundamental problems with the paper:

* The motivations of the paper are obscure, (understanding colour representation in complex visual systems? Learning better representations? Disentangling color related information from the rest)
* No analysis is provided to highlight what the novel objective is achieving

The answers of the authors to AnonReviewer1 are not very convincing. As AnonReviewer1 has pointed out, the mapping between the color spaces is typically a simple invertible map so any conclusion that the authors arrive about ‘substantial impact’ could be simply the artefact of the particular architecture choice. The other claim, that ‘the proposed framework is able to encompass additional constraints relevant in understanding why the considered representations could have emerged in the brain’ quite far fetched and speculative at best.

The authors have a point in their reply ii) to AnonReviewer1 but if the claim is about the particular color coding schemata, it would be natural to include simple experiments where some arbitrary 3x3 invertible mapping (e.g. rgb in spherical or cylindrical coordinates) next to other color schemata to make a stronger point.

In point iii) the authors refer to tasks without being very explicit about what the tasks are. Colorization is a known proxy pretext task for learning representations when the downstream classification task is not known a-priori. The paper would have been much more easy to motivate if the authors could demonstrate the merit of the proposed objective using a more extensive and careful representation learning evaluation methodology.

In light of the above points, I feel that the paper needs further iterations to be presented at ICLR.

Decision: Reject