Name Your Colour For the Task: Artificially Discover Colour Naming via Colour Quantisation Transformer
Authors: Shenghan Su
Lin Gu
Ziteng Cui
Yue Yang
Jingjing Shen
Hiroaki Yamane
Zenghui Zhang
Tatsuya Harada
Venue: Submitted to ICLR 2023 (2023)
Keywords: colour quantisation
image compression
artificial colour naming system

Abstract: The long-standing theory that a colour-naming system evolves under the dual pressure of efficient communication and perceptual mechanism is supported by more and more linguistic studies including the analysis of  four decades’ diachronic data from the Nafaanra language.  This inspires us to explore whether artificial intelligence could evolve and discover a similar colour-naming system via optimising the communication efficiency represented by high-level recognition performance. Here, we propose a novel colour quantisation transformer, CQFormer, that quantises colour space while maintaining the accuracy of machine recognition on the quantised image. Given an RGB image, the annotation branch maps it into an index map before generating the quantised image with a colour palette, meanwhile the palette branch utilises a key-point detection way to find proper colours in palette among whole colour space. By interacting with colour annotation,  CQFormer is able to balance both the machine vision accuracy and colour perceptual structure such as distinct and stable colour distribution for discovered colour system. Very interestingly, we even observe the consistent evolution pattern between our artificial colour  system and basic colour terms across human languages. Besides, our colour quantisation method also offers an efficient quantisation method that effectively compresses the image storage while maintaining a high performance in high-level recognition tasks such as classification and detection. Extensive experiments demonstrate the superior performance of our method with extremely low bit-rate colours. We will release the source code upon acceptance. 

Under review as a conference paper at ICLR 2023
NAME YOUR COLOUR FOR THE TASK: ARTIFICIALLY DISCOVER COLOUR NAMING VIA COLOUR QUANTI- SATION TRANSFORMER
Anonymous authors Paper under double-blind review
ABSTRACT
The long-standing theory that a colour- nombiig system evolves under the dual pres- sure of efficient communication and perceptual mechanism supported by more and more linguistic studies including the analysis of four decades' diachronic data from the Nafaanra language. This inspires to explore whether artificial intel- ligence could evolve and discover similar colour-naming system via optimising the communication efficiency represented by high-level recognition performance. Here, we propose novel colour quantisation transformer, CQFormer, that quan- tises colour space while maintaining the accuracy of machine recognition on the quantised images. Given an RGB image, Annotation Branch maps it into an index map before generating the quantised image with colour palette, meanwhile the Palette Branch utilises key-poin detection way to find proper colours in palette among whole colour space. By interacting with colour annotation, CQFormer is able to balance both the machine vision accuracy and colour perceptual structure such as distinct and stable colour distribution for discovered colour system, Very interestingly, even observe the consistent evolution pattern between artifi- cial colour system and basic colour terms across human languages. Besides, our colour quantisation method also offers efficient quantisation method that ef- fectively compresses the image storage while maintaining a high performance in high-level recognition tasks such classification and detection. Extensive exper- iments demonstrate the superior performance of our method with extremely low bit-rate colours. We will release the souree code upon acceptance.
INTRODUCTION
Hath not Jew eyes?
Hath not Jew hands, organs, senses, affections, passions?
William Shakespeare "The Merchant of Venice"
Does artificial intelligence share the same perceptual mechanism for colours as human beings? We aim explore this intriguing problem through AI simulation in this paper.
Colour involves the visual reception and neural registering of light stimulants when the spectrum of light interacts with cone cells in the eyes. Physical specifications of colour also include the reflective properties of the physical objects, geometry incident illumination, etc. By defining colour space (Forsyth Ponce, 2002), people could identify colours directly according these quantifiable coordinates.
Compared to the pure physiological nature hue categorisation, the complex phenomenon of colour naming or colour categorisation spans multiple disciplines from cognitive science to anthropology. Solid diachronic research (Berlin Kay 1969) also suggests that human languages are constantly evolving to acquire new colour names, resulting in an increasingly fine-grained colour naming sys- tem. This evolutionary process is hypothesised to under the pressure of both communication efficiency and perceptual structure. Communication efficiency requires shared colour partitioning to communicated accurately with a lexicon as simple and economical as possible. Colour per- ceptual structure is relevant to human perception in nature. For example, the colour space distance

---

Under review as a conference paper at ICLR 2023
(
Accurecy
lnnt len
Complexity (bits)
Colour Size (bits)
(b)
Figure 1: (a) the theoretical limit of efficiency for colour naming (black curve) and cases of the
WCS probability map of human colour language copied from Zaslavsky et al. (2022), (b) the colour size (from I-bit to 6-bit)-accuracy curve on the Lin--imaennet200 (Le & Yang 2015) dataset. The WCS probability maps generated by our CQFormer are also shown along the curve. (c) the colour naming stimulus grid used in the WCS (Kay et al. 2009). (d) the three-term WCS probability map of CQFormer after embedding 1978 Nafaanra three-colour system (light ('fiNge"). ) dark ('wOO'), ), and warm or red-like ("nyiE")) into the latent representation.
(e) the four-term WCS probability map of CQFormer evolved from (d). The evolved fourth colour, yellow-green, consistent with the prediction of basic colour term theory (Berlin & Kay 1969)
between nearby colours should correspond to their perceptual dissimilarity. This structure of percep- tual colour space has long been used to explain colour naming patterns across languages. A recent analysis of human colour naming systems, especially in Nafaara, contributes the very direct evidence to support this hypothesis through the employment of Shannon's 's communication model (Shannon 1948). Very interestingly. this echos the research on colour quantisation, which quantises colour space to reduce the number of distinct colours in image.
Traditional colour quantisation methods (Heckbert) 1982 Gervautz & Purgathoter 1988; Floyd & Steinberg 1976) are perception-centted and generate new image that is visual perceptually sim- iar as possible to the original image. These methods group similar colours in the colour space and represent each group with a new colour, thus naturally preserving the perceptual structure. Instead of prioritising the perceptual quality, Hou et al. (Hou et al. (020) proposed a musk--cettrddmmcciin centred colour quantisation method, ColorCNN, focusing on maintaining network classification ac- curacy in the restricted colour spaces. While achieving an impressive classification accuracy on even a few-bit image, ColorCNN only identifies and preserves machine-centred structure, without directly clustering similar colours in the colour space. Therefore, this pure muchine-centres strategy sacrifices perceptual structure and often associates nearby colours with different quantised indices.
Zaslavsky et al. (Zaslavsky et al., 2022p measure the communication efficiency in colour naming by analysing the informational complexity based on the information bottleneck (IB) principle. Here, argue that the network recognition accuracy also reflects the communication efficiency when the number of colours is restricted. Since the human colour naming is shaped by both perception structure and communication efficiency (Zaslavsky et al. 2019a), we integrate the need for both perception and machine to propose a novel end-to-end colour quantisation transformer, CQFormer, to discover the artificial colour naming systems.
As illustrated in Fig. (b), the recognition accuracy increases with the number of colours in our discovered colour naming system. Surprisingly, the complexity-acuuray trade-offss are similar the numerical results (Fig. [ (a) independently derived from linguistic research (Zaslavsky et al. 2022).
What is more, after embedding 1978 Nafaanra three-colour system Vafaaanra-9978 Fig. (d) into the latent representation of CQFormer, our method automatically evolves the fourth colour closed to yellow-green matching the basic colour terms theory (Berlin & Kay 1969) summarised in different languages. Berlin and Kay found universal restrictions on colour naming across cultures and claimed languages acquire new basic colour category in a striet chronological sequence. For example, if a culture has three colours (light (" ('fiNge"), ) dark ("wOO"), and warm red-like ("nyiE") in Nafaanra), the fourth colour it evolves should be yellow or green, exactly the one (Fig. (c)) discovered by our CQFormer.
The pipeline of CQFormer, shown in Fig. 2 comprises two branches: Annotation Branch and Palette Branch. Annotation Branch annotates each pixel of the input RGB image with the proper quantised colour index before painting the index map with the corresponding colour in the colour palette. We localise the colour palette in the whole colour space with novel Palette Branch which detects the
2

---

Under review as a conference paper at ICLR 2023
key-poin with explicit attention queries of transformer. During the training stage, illustrated in the red line and black line of Fig. 2 Palette Branch interacts with an imput image and reference palette queries maintain the perceptual structure by reducing the perceptual structure loss. This perception-centred design groups similar colours and ensures the colour palette sufficiently repre- sents the colour naming system defined by the World Color Survey (WCS) colour naming stimulus grids. As shown in Fig. F (b), each item in colour palette (noted by an asteroid) lies in the middle of corresponding colour distribution in the WCS colour naming probability map. Finally, the quantised image is passed to a high-level recognition module for machine accuracy tasks such as classification and detection. Through the joint optimisation of CQFormer and consequent high level module, we are able to balance both perception and machine. Apart from automatically discovering the colour naming system, our CQFormer also offers an effective solution to extremely compress the image storage while maintaining high performance in high-level recognition tasks.
Our contributions could summarised as following:
We propose a novel end-to-end colour quantisation transformer, CQFormer, to artificially discover colour naming system by considering the needs of both perception and machine. The discovered colour naming system shows similar pattern as human language on colour.
We propose a novel colour palette generation method that takes the colour quantisation as an attention-based key-point detection task, inputs independent attention queries to gener- ate 3D coordinates, and then automatically finds the selected colour in the whole colour space.
Our colour quantisation achieves superior performance on both classification and object detection with extremely low bit-rate colours. The source code will be released to facilitate the community.
2 RELATED WORKS
Colour Quantisation Methods: Traditional colour quantisation (Orchard & Bouman
1991 Deng
al. 1999 Liang et al., 20033 Wu 1992) reduces the colour space while maintaining visual fi- delity. These perception-centre methods usually cluster colours to create a new image as visually perceptually similar as possible to the original image. For example, MedianCut (Heckbert) 1982) and OCTree, (Gervautz & Purgathofer 1988) are two representative clustering -ased algorttmms. Dithering (Floyd & Steinberg 1975) eliminates visual artefacts by including noise pattern. The colour-quantised images can be expressed as indexed colour (Poynton) 2012), and encoded with PNG (Boutell 1997). Recently, Hou el al. propose pure muchine-centred CNN-based colour quantisation method, ColorCNN (Hou al., 2020), which effectively maintains the informative structures under an extremely constrained colour space. In addition to colour quantisation, Cam- poseco et al. also design task-centred image compression method for localisation in 3D map data (Camposeco et al.] 2019). Since human colour naming reflects both perceptual structure and communicative need, our CQFormer also considers both perception and machine artificially dis- cover the colour naming system.
The World Color Survey (WCS) (Kay al. 2009) comprises colour name data from 110 languages of non-industrialised societies (Zaslavsky et [..99b with respect to the stimulus grid shown in Fig. [T]. There are 320 colour chips in colour naming stimulus grids, and each chip is at its maximum saturation for that hue/lightness combination, while columns correspond to equally spaced Munsell hues and rows to equally spaced Munsell values. Participants were asked to name the colour of each chip to record the colour naming system.
World Color Survey:
3 METHODOLOGY
3.1 PROBLEM FORMULATION
For an image-label pair (z, y) in dataset D, the recognition network fo(-) encodes input image ø to predict its label ý (i.e. class probability in classification task). fo(-) can be optimised by minimising the loss between prediction ğ = fo(z) and ground truth y. defined as machine-centee centred loss L.. to find the appropriate parameter 0*:
3

---

Under review as a conference paper at ICLR 2023
Annotation Branch
Intra cluntedutterrre Colour Similrity Regulartatiinn
UNe
Traning Stage
with Inrperatue
High-lesel
Resogniion Modile
Teeiing Stage
Oneict
Palette Branch
Trainin Stage
Textigg flage
((
Barn enge
(B0)))
Reference Paletto Oueries
Colour Polere PUAY)
(bb) WCS Colour Probability Map
[a] PAM:
Fi
DWWCone
Blaton
Cross Attention
Facto
FFN
Spmold
(1982
Relerunce Paletta Quarias
Structure of CQFormer (3-colour quantisation). Red lines are only for training stage; Purple lines are only for testing stage; black lines are both used in both stage. (a) is the detailed structure of PAM. (b) is WCS colour probability map counted across all pixels of the input image.
Figure 2:
0 arg min (æ.y)ED
Σ
LM fo(x)).
We aim discover the artificial colour naming system by meeting the need for both machine accu- racy and human perception. Therefore, CQFormer not only strives for recognition accuracy but also maintains the perceptual structure. The function in Eq. [ can be rewritten as follow:
3 0* = arg min
Σ
LM (v, fo (x)())) + Lr.
(2)
$10
(a.
where vt, 0 respectively denotes the parameter of CQFormer g and the following high level recog- nition network f. We jointly optimise g and f to find optimal parameters vi* and 0". Lp is the perceptual structure loss that is perception-centred and will be introduced in detail in Sec. 3.3
3.2 CQFORMER ARCHITECTURE
Overall Architecture. An overview of the CQFormer is depicted in Fig. 2 which mainly consists of two branches: (1) Annotation Branch annotate each pixel of an input RGB image with proper quantised colour index, and (2) Palette Branch to acquire a suitable colour palette.
Given input image x R Annotation Branch generates a probability map m. (x) XCCCCC during training stage or a one-hot colour index map One-hot(MM(x) E Rfx KC dur- ing testing stage. H and W denotes height and width of the input image C represents the number of quantised colours, and > denotes the temperature parameter of the Softmax function (He) al. 2018). In Palette Branch, we first define reference palette queries Q Cxd consisting of C' learnable vectors of dimension d. Each vector represents an automatically mined colour. The Q interacts with the keys K E R HW
# (X)x and values V E RC TS jxd generated from the input image æ to produce the colour palette P(x) E Cx3 that consists of C ttiples of (R, G, B). Each triple represents one of the prechine-discoveeed C' colours. Finally, CQFormer outputs the quantised im- age by calculating a matrix multiplication between m, (x) and P(x) during the training stage. For testing stage, we get it from One-hot(M (M()) and P(æ). Then, feed the colour-quantised image into the high-level recognition module for recognition tasks, iee, classification and object detection.
Annotation Branch. The first component of Annotation Branch is UNeXt (Valanarasu & Patel 2022) encoder that outputs per-pixel categories. Given the input image æ, the encoder generates a T H W'xC with crucial and semantically rich features.
class activation map Factivation
(1) Testing stage: We then use the class activation map Factivation as the input to a moft function over C channels followed by a arg max function obtain a colour index map AM (æ) E Hx:
(x) = arg max Sosoffaa(aaettiiita
(3)
4

---

Under review as a conference paper at ICLR 2023
After that, one-hot colour index map One-hot(M(zz) B is generated from the M(x) using a one-hot function One-hot(-) and finally we utilise the One-hot(M((x) and the colour palette P(x) to generate the test-time colour-quantised image E via a matrix multiplication:
a (M @ P(x),
where @ represents matrix multiplication.
(2) Training stage: Since the arg max function is not differentiable, utilise the Soflmax function instead of the arg max. To avoid over-fitting, we add a temperature parameter v (He et al., (0018) to the max function, foreing the probability map's distribution closer one-hot vector, and obtain the probability map m, (x) with the temperature parameter т. The my (æ) is computed as:
my (a) Softmax Pactivation
(5)
As extensively discussed in He et al. (2018), when in a low temperature (0 < < 1), the output similar to one-hot vector with large diversity; while in high temperature > 1). the output
distribution is similar to uniform distribution with small diversity. Therefore, we fix the 0 < << < 1 to approximate the One-hot (M (z)) using ma(a+) during the training procedure. The train-time colour-quantised image is generated as:
I = (m) @ P(x).
(6)
Palette Branch. We locate the representative colours by an attention-based key-poin detection strategy, which is originally designed to utilise transformer queries automatically find key-point location (i.e. bounding-box (Carion et al. 2020), human pose (Li et al. (02T)) by attention mecha- nism. In other words, reformulate the problem of colour quantisation localisation task among the whole RGB colour space. Given the input image æ, we first extract a high-ding dich-imensional and lower resolution feature Fo E R # % xd by two stacked convolutional layers. After that, the Fo is fed into a palette acquisition module (PAM) to acquire the colour palette P(x)). As shown in Fig. a),, we use a depth-wise convolution (DWConv) (Chu et al., (021) with resid- ual connection to complete position encoding, which is suitable for different input resolutions. After that, since the cross attention block expects sequence input, we collapse the spatial dimensions into one dimension and obtain the flattened feature map Fi ((4%)) )xd;
3D spatial key-point
F Platten(DWCoo
(Fo) Fo).
Different from DETR (Carion et al. 2020). where Q. K and V are generated from the same input, our Q are explicit learnable embeddings, called reference palette queries, representing each auto- matically mined colour. K and V are the features extracted from F1 via Layernorm (LN), which is followed by two fully connected layers (FCs). Generally, we have Q E Rxdd K, V E R ( xd The attention matrix Fattn& E RCxd is computed by the cross-attention (CA) mechanism as
(8) Next, a multi-layer perceptron (MLP) that has two FCs with GELU non-linearity between them is utilised for further feature transformations. The LN is added before MLP, and the residual connec- tion is employed only for MLP. The whole process is formulated as FMLP = MLP(LN((attt) + Fattn: where the FMLP E RCxd is the MLP features. Finally, the FALLP would pass through a feed forward network (FFN), which has the same architecture as MLP, followed by Sigmoid function to get the final C sets of 3D coordinates Fooord E [0, C3 which corresponds to values from 0 to 255 in the whole RGB colour space. Hence, take 255 of the &coord the colour palette P(x):
Farm Allention (Q. K, V) Softmax (QK / //d)v.
P(x)
255 x Sigmoid(FFN(FALL))
(10)
High-Level Recognition Module. High-level recognition module takes the generated or as inputs, and we adopt the popular Resnet (He et al. 2016) as the classifier and the Faster-RCNN (Ren et al. 2015) as the detector.
3.3 PERCEPTUAL STRUCTURE LOss
The CQFormer is trained in end-to-end fashion using machine-centred loss Lv. To balance the machine accuracy and perceptual structure, we also add a perceptual structure loss Lp includ- ing perceptual similarity loss Peeeceptual- diversity regularisation PDiversity and intra-cluster colour similarity regularisation RColour. The perceptual structure loss can be summarised as follows: Lp = acCooou + R.wersitt + Perceptuul:
(11)
5

---

Under review as a conference paper at ICLR 2023
where a B and y are the weights that control the contributions of Reolour, RDiversity and Peerceptual, respectively. We combine the Lys and the Lp as the total loss L. and then minimise L for the clssification and detection task:
(12) Additionally, we define WCS colour probability map mwwes E R on the colour naming stimulus grids in the WCS and use it calculate RColour- As shown in Fig. (b)) the horizontal axis represents the Munsell hue ranging from 0 to 360", and the vertical axis denotes the Munsell value ranging from 0 to Each grid associated with the equally spaced (hue, value) coordinate and a grid's colour index with the C' corresponding occurrence frequencies. The map for human language is collected by the participants colour perception in WCS (see in Fig. cl)), and the ma- chine one is counted across the pixels' colour index in the datasets (see in Fig. [d) ,Fig. (b)). Precisely, we first map the colour index of each pixel to the grid with the same (hue, value) coor- dinate as the pixel and then count the frequency of occurrence of each colour index in each grid. Finally, we take the colour index with the most considerable frequency as the grid's 's colour index.
C=LM Lp.
The perceptual similarity loss Perceptuul is mcan squared error (MSE) loss between the quantised image and the input image æ, which makes the T or æ per- ceptually similar the æ. In particular, the Lerrcppuul ensures each item of the P(x) (noted by asteroids in Fig. (2)) lies in the centre of corresponding colour distribution in the WCS colour naming probability map. The Peercoptual is formulated as:
Perceptual Similarity Loss:
(13) Diversity Regularisation: To encourage the CQFormer to select at least one pixel of all C' colours, adopt the diversity regularisation term RDiversity proposed by Hou et al. Diversity is simple yet efficient metric that serves as an unsupervised signal to maintain colour diversity in the quantised image by maximising the maximum probability of each channel. The RDiversity is caleulated as:
Peercpptul = LMSE m)),
1 x Σ 1
RDiversity = log2 C' x
1-
max(m-
(14)
(u,F)
Intra-cluster Colour Similarity Regularisation: The CQFormer associates each pixel of the x with a colour index, and the pixels with the index e form a cluster Cluster, covering a part of the mwcs (see the different colour grids in Fig. (b)). To make sure the pixels in the same cluster are perceptually similar in colour as possible, we propose the intra-cluster colour similarity regu- larisation Rçolour and calculate in the WCS's Munsell HSV colour space. At first, we calculate the centroid colour value fee of Cluster Then, we compute the squared colour distance Dist between all pixels in Cluster, and Pe in the conical representation of HSV. Finally, we calculate the mean value of Distfisv for each cluster and obtain the Roolour
Dist [1u $1, v1], [h2, $2, U2]] (U2 01)
+su + +200 2s 221 cos (h2 - -h1). (15)
No
RColour
C Σ Dist (
(16)
where No is the number of all pixels in Cluster and [ represents the Munsell HSV value of the i-th pixel in Clustere.
3.4 COLOUR EVOLUTION
To discover the colour naming system, we explore the colour evolution based on the classification task using the CQFormer cascaded with the Resnet-18 (He et al. (0166) without any pre-train models. It consists of two successive stages. Since there are various colour naming systems associated with corresponding languages, the first embedding stage aims to embed the colour perceptual knowledge of a certain language into the latent representation of the CQFormer. For example, CQFormer first learns and matches the Nafaanra with three colours. Specifically, we design two embedding solutions to either distil full colour probability map emhedding or only representative colours to CQFormer. The second evolution stage then lets CQFormer evolve more colours, i.e. splitting the fourth colour from the learned three colour system under the pressure of accuracy without restriction.
Embedding Stage: (1) Colour Probability Map Embedding: This embedding forces our CQFormer to match the identical WCS colour probability map of a certain language. At first, we train CQ- Former to have the same colour number C of the designed human colour system. As illustrated in Fig. 3 for each pixel in the input image, we collect the pixel's spatial position coordinate (i.j) in the input image, and locate its Munsell H and V value the mwcs of certain language to find its

---

Under review as a conference paper at ICLR 2023
corresponding C probability values PHumau (i,j) E RC of human colour categories (e.g. 9% for 'fiNge', 4% for 'wOO", and 87% for "yyiE' in the little red square Fig. C)). After perform- ing the above operations (PHuman ji. E [0, H) E [0, w) We arrange each PHuman (f in this set according to its spatial position (1.j) and obtain new H W x C matrix regarded as human language probabil- ity map Human (z). Then we force the m, (x) to match the Human (x) with cross-entropy loss Leg. Here utilise the arg max on Mumman (x) for efficient embedding and replace the loss func- tion in Eq. [2] with Eq. [7]. In this knowledge distillation way, our CQFormer successfully inherits the colour naming system of human language, which used in the subsequent colour evolution
all pixels of the imput image, set of probability values is generated
stage.
C=Ly + (c arg max max(mHaa Hummn (z))).
(17)
(2) Central Colour Embedding: Alternatively, we could distil less information of human colour naming system. Here, we only em- bed representative colours and their C cen- tral colour H and V coordinates MHumannc E R² C [0, C') in human's 's mwws (noted by as-
24
teroids in Fig. Rc).. Specifically, we utilise to draw the three mwcss in Fig. lbb.. By op- timising the loss function in Eq. 12 where we replace the Me in RColou with aforementioned
Figure 3: The procedure of colour probability map embedding. (a) the origin RGB image. (b) the WCS colour naming stimulus grids. (c) the mwcs of Nafaanra-1978. (d) the human colour index map arg (ax(()) for Nafaanra-1978.
AHuman.c and ignore the S value of HSV, we obtain the embedded CQFormer with the central colour.
Colour Evolution Stage: After inheriting the existing colour naming system of human language, the CQFormer could evolve to acquire fine-grained new colour on top of the embedded model. In this stage, we remove the restriction of the number of colours C and encourage the CQFormer to evolve more colours under the pressure of accuracy.
4 EXPERIMENTS
We evaluate our CQFormer on mainstream benchmark datasets both image classification (Sec. (4.2) task and object detection task (Sec. 4.3) Additionally, specifically design colour evolution experiment (Sec. 4.44 to demonstrate how our CQFormer automatically evolves to in- crease fine-grained colours. For ablation study, visualisation and detailed results, please refer to Appendix Sec. 6
4.1 DATASETS AND EXPERIMENT SETUP
Datasets: For classification, we utilise CIFARIO (Krihhvsky et 2009). CIFARI00 KKriheessky
et al. 2009), STLIO (Coates et al., (01T) and uny-imagenet-2000 (Tiny200) Le & Yang 2015) dataset. Both CIFARIO and CIFARIO0 contain 60000 images, covering 10 and 100 classes of general objects, respectively. The STL10 consists of 13000 images (5000 for training and 8000 for testing) with 10 classes. The Tiny200 is a subset of ImageNet (Deng et al.) 2009) and contains 200 categories with 1 11kk images. All train images are random cropped, random horizontal flipped and resized into their respective original resolutions. For object detection, we utilise MS COCO (Lin ct al. 2014) dataset, which contains ~ 118k images with bounding box annotation in 80 categories. Here, we use COCO t train2011 set as the train set and use COCO val201 set as the test set.
Evaluation Metrics:
For classification, report top-1 classification accuracy the evaluation metric. For object detection, we report average precision (AP) value in COCO evaluation setting.
Implement Details:
(i) Upper bound:
We utilise the performance of classifier/detector without additional colour quanti- sation methods in full colour space (24 bit) the upper bound. For classification upper bound, adopt Resnet-18 18 (He et al. (0166) network. For detection upper bound, we adopt Faster-RCW (Ren)
et al. 2015) network with Re (He et al. 2016) backbone and FPN (Lin et al. 2017) neck.
(it)Trainng Settings: All colour quantisation experiments are finished at quantisation levels from 1-bit to 6-bit, i.e. C E {2, 4,,,,8, 8, 12, 32, 64). For classification, we cascade CQFormer with the
7

---

Under review as a conference paper at ICLR 2023
CHRAR10
OFARL00
STL10
Tiry200
100
30
100
33
AccuntE 990
43
23
Coocles
Uper beard
Colour Siee (b10
Colour
Colour Size
Colour
Figure 4: Top-1 classification accuracy of colour-quantised images on four datasets with Resnet- 18 (He et al. 2016 networks.
classifier and jointly train them without any pre-trained models on single GeForce RTX A6000 GPU. We set the temperature parameter т = 0.01 and the a = 1, 3 = 0.3 7 1. We employ an SGD optimiser for 60 epochs using the Cosine-Warre Warm-Restart (Loshchilov & Hutter 2016) the learning rate scheduler. A batch size of 128 (STL10 is set to 32), an initial learning rate of 0.05, momentum is 0.5, a weight decay is 0.001 are used. For detection, only the Resnet-50 backbone of the detector is initialised with Imagenet (Deng et al.) 2009) pre-train weights. We also cascade and jointly train the combination for 12 epochs on Tesla V100 GPUs, with the a = 0, 8 1. = 0.1, and batch size is set to 8. We also adopt the SGD optimiser with an initial learning rate 0.01, momentum 0.9 and weight decay 0.0001, learning rate decay to one-tenth at and 11 epochs.
Comparison Methods: As illustrated in Fig. 4 we compare with three traditional perception- centred methods: MedianCut (solid green line) (Heckbert) 1982). MedianCut-Diither (solid orange line) (Floyd & Steinberg, 1976) and OCTree (solid grey line) (Gervautz & Purgathofer 1988), and a task-centred CNN- method ColorCNN (solid yellow line) (Hou et al.] 2020). Specifically, for task-centred ColorCNN, we adopt the same training strategy as we did for CQFormer. For the traditional colour quantisation methods, we conduct comparative experiments as described in the ColorCNN (Hou et al., 2020).
Table 1: Object detection results on MS COCO dataset (Lin et al. 2014) with Faster-RCNN Ren et al. (2015) detector, here we report the average precision (AP) value.
Method
1-bit
bi
3-bit
4-bit
5-bit
6-bit
Full Colour (24-
Upper bound
37.2
Median Cut (w/o D) (Heckbert. 1982
6.5
9.7
11.4
14.0
16,8
18.1
Median Cut (w D) FFloyd & Steinberg 1976
7.3
8.8
11.2
13.4
14.8
15.6
OCTree (Gervautz & Purgathofer
1988)
8.7
9.0
9.8
10.9
13.4
14.5
CQFormer
8.9
11.2
12.8
14.5
16.6
19.4
4.2
CLASSIFICATION TASK EVALUATION
Fig. presents comparisons to the state-of-the-an methods the four datasets. Our proposed CQFormer (solid blue line) has a consistent and obvious improvement over all other methods in extremely low-bit colour space (less than 3-bit). Moreover, our CQFormer archives superior per- formance than the task-centred method ColorCNN (Hou et al. 2020) under all colour quantisation levels from 1-bit to 6-bit. As the number of the quantised colours increases, the accuracy of the CQFormer continues to improve, which is similar to the complexity-ccurrac trade-offs (Fig. Iaa independently derived from linguistic research (Zslavsky et al. 2022).
Similar to the task-centred ColorCNN, we are also inferior to the traditional method under a large colour space (greater than 4-bit), which is an inherent limitation of the CQFormer. As extensively discussed in Hou et al. (202)). this is very normal since traditional methods clusters all pixels to enforce intra-cluster similarity to make a collective decision. Actually, this also explains why our method consistently outperforms pure machine- centred ColorCNN as our method considers the per- ceptual structure. In addition, we are quite satisfied with the current superior performance on limited colour numbers, most human languages only use fewer than 3-bit colour terms. This implies that discovering more colours not only compromises the principle efficiency but also goes contrary to the expectation of better perceptual effects. In other words, the performance of the CQFormer on limited colour categories may hint at the optimal outcome restricted by the unique neurological structure of human vision and cognition, which are, in turn reflected in wide array of languages.
8

---

Under review as a conference paper at ICLR 2023
4.3 DETECTION TASK EVALUATION
Table. T shows the object detection results on MS COCO dataset Lin et al. (2014) with Faster- RCNN Ren et al. (2015) detector, we report the average precision value (AP) of CQFormer and the comparison methods. We could see that our CQFormer also gains superior performance in object detection task in most colour sizes. We also evaluate CQFormer with more recent SOTA object detectors. For more details, please refer to the Appendix part.
4.4 COLOUR EVOLUTION EVALUATION
Settings:
This experiment is based on the classification task on CIFARIO (Kriheessky et al. 2009) dataset using CQFormer and Resnet-18 (He et al. 2016) without any pre-train models. As intro- duced in Sec. 3.4] in the first embedding stage, we embed the Nafaanra-1978 (Fig. B c)) using the colour probability map embedding. Here we set C = 4, T = 0.1 a = S = Y U. and force the fourth colour be split from light ("fiNge'), dark ('wOO'), and warm or red-like ("nyiE"), respec- tively. Therefore, the number of quantised colours output by CQFormer is still 3, and we optimise the loss function in Eq. [7 for the initial 40 epochs. In the second colour evolution stage, we make the CQFormer inherit the parameters from the first stage and remove the restriction on the last two colour indexes. The changed as 1, and the loss function in Eq. 12 is minimised for the subse- quent 20 epochs.
During the first stage, the WCS colour probability map generated by the embedded CQFormer is shown in Fig(T])), and similar to Nafaanra- aefaanna-1978 in Fig During the second stage, the CQFormer automatically evolves the fourth colour that is split from dark ( ('wOO') and close to yellow-green (see in Fig. Ie)) matching the basic colour terms theory (Kay & McDaniel 1978). However, we are not able to see the fourth colour split from either light ( ('fiNge') or warm/red-like ('nyiE') in the mwcs. since only 3.7% (if split from light (' ("fiNge')) and 5.9% (if split from warm/red-like ('nyiE)) of all pixels are assigned to the fourth colour, compared with 23.5% (if split from dark ('wOO') )) Very interestingly, this phenomenon echoes the evolution of the information bottleneck (IB) colour naming systems (Zaslavsky et al.) 2018), where the fourth colour should be spilt from dark in the "dark-light-red"" colour scheme. In other words, similar to human language, the colour naming of AT also evolves under the pressure of accuracy and the discovered colour naming system shows similar pattern as human language on colour. (Zaslavsky et al. 2022
5 LIMITATION AND DISCUSSION
While the complexity-accuree tradeoff of maachine-disoveere colour terms, as shown in Fig. [(b,, is quite similar to the numerical limit of categorical counterparts for human languages, the current work is still preliminary. As shown in Fig. [ the newly discovered WCS colour probability map is still quite different from the human one. A more accurate language evolution replication needs to consider more complex variables such as environmental idiosyncrasies, cultural peculiarities, functional necessities, technological maturity, learning experience, and intercultural communication.
Another promising direction would be associating the discovered colours with human colour terms. This would involve much research Nature Language Processing, and hope to discuss it with experts from different disciplines in future works. Last but not least, the AI simulation outcome con- tributes to the long-standing universalist-relatiist debate of the linguistic community on colour cog- nition. Though not entirely excluding the cultural specificities of the colour schemes, the machine finding strongly supports the universalist view that an innate, physiological principle constraints, if not determine, the evolutionary sequence and distributional possibilities of basic colour terms in communities of different cultural traditions. The complexity-efficiene the finding that the numerical limitation of colour categories could lead to superior performance on colour-specific tasks, contrary to the intuitive expectation that complexity breeds perfection. The independent AI discovery of the "green--ellow" category on the basis of the fundamental tripartite "dark- -igrk-light-red" colour scheme points to the congruence of neural algorithms and human cognition and opens new frontier to test contested hypothesis in the social sciences through machine simu- lation. We would be more than delighted if this tentative attempt would prove to be a bridge to link scholars of different disciplines for more collaboration and generate more fruitful results.
principle is confirmed by

---

Under review as a conference paper at ICLR 2023
REFERENCES
B. Berlin and P. Kay. Basic Color Terms: Their Universality and Evolution. University of California Press, 1969. ISBN 9780520076358.
Thomas Boutell. Png (portable network graphics) specification version 1.0. Technical report, 1997.
Federico Camposeco, Andrea Cohen, Marc Pollefeys, and Torsten Sattler. Hybrid scene compres- sion for visual localization. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 7653-7662, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. Arxiv preprint 2102. 10882, 2021. URL
https://arxi:: org/ wtwp/////.p.oooooo//// /2002 10882.pd.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
J. Deng, W. Dong, R. Socher, L.JJ. Li, K. Li, and L.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
Yining Deng, Charles Kenney, Michael S Moore, and BS Manjunath. Peer group filtering and perceptual color image quantization. 1999 IEEE International Symposium on Circuits and Systems (ISCAS), volume 4, pp. 21-24, IEEE, 1999.
R. W. Floyd and L. Steinberg. An adaptive algorithm for spatial gray scale. Proceedings of the Society for Information Display, 17, 1975.
R. W. Floyd and L.. Steinberg. An adaptive algorithm for spatial grayscale. Proceedings of the Society for Information Display, 17, 1976.
David A Forsyth and Jean Ponce. Computer vision: modern approach. prentice hall professional technical reference, 2002.
Michael Gervautz and Werner Purgathofer. A simple method for color quantization: Octree quanti- zation. In New trends in computer graphics, pp. 219-231. Springer, 1988.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yu-Lin He, Xiao-Liang Zhang, Wei Ao, and Joshua Zhexue Huang. Determining the optimal tem- perature parameter for softmax function in reinforcement learning. Applied Soft Computing, 70: 80-85, 2018. ISSN 4946. doi: www. /tw.://cccccciicc....ooocaaaaata
2018.05.013 URL https:
Paul Heckbert. Color image quantization for frame buffer display. ACM Siggraph Computer Graph- ics, 16(3):297-307, 1982.
Yunzhong Hou, Liang Zheng, and Stephen Gould. Learning structure an image with few colors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10116-10125, 2020.
Paul Kay and Chad K McDaniel. The linguistic significance the meanings of basic color terms. Language, 54(3):60--64,, 1978.
Paul Kay, Brent Berlin, Luisa Maffi, William R Merrifield, and Richard Cook. The world color survey. CSLI Publications Stanford, CA, 2009.
10

---

Under review as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3 2015.
Ke Li, Shijie Wang, Xiang Zhang, Yifan Xu, Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1944-1955 2021.
DianLong Liang, Lizhi Cheng, and Zenghui Zhang. General construction of wavelet filters via lifting scheme and its application in image coding. Optical Engineering, 42(7):1949- 2003.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp- 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint wrXiv:1608.03983 2016.
M.T. Orchard and C.A. Bouman. Color quantization of images. IEEE Transactions on Signal
Processing, 39(12):6677-269,, 1991. doi: 10.1109/78. 107417.
Charles Poynton. Digital video and HD: Algorithms and Interfaces. Elsevier, 2012.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-422 1948.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 14454-14463, 2021.
Jeya Maria Jose Valanarasu and Vishal M Patel. Unext: Mlp-based rapid medical image segmenta- tion network. arXiv preprint arXiv:2203 200.02200.4477 2022.
Xiaolin Wu. Color quantization by dynamic programming and principal analysis. ACM Transactions on Graphics (TOG), 1144)348-377) 1992.
Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby, Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 1 113(31))7937-7744 2018. ISSN 0027-8424. doi: 10. 10.10073/nas.18000 content/115/31/395
URL http::/www: pnas org/
Noga Zaslavsky, Charles Kemp, Naftali Tishby, and Terry Regier. Color naming reflects both per- ceptual structure and communicative need. Topics in Cognitive Science, 11(1):27-221,, 2019a.
Noga Zaslavsky, Charles Kemp, Naftali Tishby, and Terry Regier. Communicative need in color naming. Cognitive Neuropsychology, 2019b. doi: 10.1000/0264999400999..6664
Noga Zaslavsky, Karee Garvin, Charles Kemp, Naftali Tishby, and Terry Regier. The evolution of color naming reflects pressure for efficiency: Evidence from the recent past. Journal of Language Evolution, 2022. ISSN 2058-4588
doi: 1001000/ooc//11c000..
11

---

Under review as a conference paper at ICLR 2023
APPENDIX
6.1 ABLATION STUDY
Table 2: Ablation study results under 1-bit colour quantisation, Tiny200 Le & Yang 2015 dataset, Renset18 He et al. (2016) classifier.
RColour
RDiversity
EPercentual
Palette Branch
Accuracy
45.1
6.9
X
43.9
43.7
44.6
X
X
X
37.3
V
X
30.4
Table 3: Ablation study results of robustness under under different colour quantisation levels from
1-bit to 4-bit, CIFARIO Krizhevsky et al. (2009) dataset, Renset 8 He et al. (2016) classifier.
bit
2 bit
3 bit
4 bit
CQFormer
80.7
83.1
83.8
85.2
CQFormer (with colour jitter)
79.3
81.6
83.4
84.6
CQFormer(with Gaussian blur)
80.6
81.7
81.9
83.6
As shown in Table we ablate the important elements in our CQFormer, using Tiny200 (Le Yang 2015) classi- fication dataset under 1-bit colour quantisation. We inves- tigate the effectiveness of the temperature parameter by setting r 1.0, perceptual structure loss by setting any terms of a, 3 and y as 0 and Palette Branch by replacing it with set of centroids in colour space. We also inves- tigate the robustness of CQFormer by adding colour jit- ter and Gaussian blur using CIFARIO (Krizhevsky et al. 2009) classification dataset under different colour quan- tisation levels from 1-bit 4-bit. Results shown in Table. and Table. B
Influence of temperature parameter: Without the tem-
perature parameter, a severe accuracy drop (-38.8%) has occurred, which shows that the temperature pa- rameter in our CQFormer can further approximate the One-hot(MM (M (z)) using (x) during training stage boost classification accuracy.
Influence of perceptual structure loss: With the RColour, Rpiversity and Perrceptull our CQFormer im- prove the top-1 accuracy by 1.2%. 1.4% and 0.5%, re- spectively. When we remove all of them, considerable accuracy drop of -7.8% has occurred. It demonstrates that the perceptual structure loss contributes to better machine accuracy besides maintaining perceptual similarity.
(a)
(b)
(c)
Figure 5: (a) is the original image. (b) is the quantised image with Palette Branch. (c) is the quantised image us- ing a set of centroids instead of Palette Branch.
Influence of Palette Branch: If we remove Palette Branch and make the CQFormer just learn
set centroids in RGB colour space, the accuracy is 30.4%, resulting in a severe drop of 14.7%. As shown in Fig. 5] it would make all the images have the same colour palette rather than the same amount of colours, resulting in a loss of perceptual similarity, egg. the blue sky in CoL(C) is represented light yellow. Therefore, our Palette Branch ensures that reference palette queries
12

---

Under review as a conference paper at ICLR 2023
sent to interact with the image and creates the colour palette using both machine preference and image perception features, which maintains the colour specificity each image.
Robustness of CQFormer: We add a colour jitter and Gaussian blur to the colour-quantised image. Results are shown in Table. S For example, on the CIFARIO classification dataset, we achieve 79.3%, 81.6%, 83.4%, and 84.6% top-1 accuracy with colour jitter from 1-bit to 4-bit colour quantisation. The colour jitter causes a little drop of 1,4%, 1.5%, 0.4%, and 0.6%, respectively. Therefore, our CQFormer robust enough to overcome different noises.
6.2 OBJECT DETECTION WITH OTHER DETECTOR
Table 4: Object detection results MS COCO dataset (Lin al. 2014) with Sparse-RCNN Sun et al. (2021) detector, here we report the average precision (AP) value.
Method
1-bit
2-bit
3-bit
4-bit
5-bit
6-bit
Full Colour (24-bit)
Upper hound
45.0
Median Cut (wo D) (Heckbert. 1982
11.5
12.7
15.4
17.0
20.4
23.2
Median Cut (w D) (Floyd & Steinberg, 1976
12.3
13.8
15.2
19.6
21.8
25.6
OCTree (Gervautz & Purgathofer
1988]
10.7
13.4
13.2
16.7
18.9
22.8
CQFormer
13,9
16.5
18.8
21.5
27.5
29.8
To evaluate our CQFormer's generalization on another object detector, we use the recent SOTA object detector Sparse-RCNN [Sun al. (202T) for evaluation. We jointly train our CQFormer with Sparse-RCNN for 36 epochs on 4 Tesla VI00 GPUs and adopt Adam optimizer. The initial learning rate is set to 1.25e -6, and the learning rate would decay to one-tenth at 24 epochs. The dataset and other training settings are the same Faster-RCNN experiments in Sec. 4.3
6.3 THE RELATIONSHIP BETWEEN THE IB COLOR NAMING MODEL AND CQFORMER
accuracy
accuracy
g(w)m)
complexity
CQFemeer
Colour Size
Classifier
(a)
(b)
Figure 6: (a) is the IB color naming model proposed by Zaslavsky (b) is our colour quantisation model.
Fig. 6 aaa is the information bottleneck (IB) color naming model proposed by Zaslavsky et al, A straightforward communication scenario, which can be derived from Shannon's communication model (Shannon) 1948), serves as the foundation for this theoretical framework.
Fig. 6(b) is colour quantisation model. The motivation of our colour quantisation model archi- tecture is driven by the IB color naming model (Zaslavsky et al.) 2018). As illustrated (b), the speaker represents the CQFormer, and the listener represents the classifier. We focus on the case where a colour quantiser (CQFormer) and a classifier communicate about colours. The CQFormer has "mental" representation, i.e. full-colour image æ associated with a prior label y drawn from prior distribution, and communicates this representation by encoding it into a colour- quantised image ø according to the perceptual similarity, The classifier receives and attempts to infer from it the full-colour image's label y by predicting the label and constructing another distribution that approximates y.
There exists a problem that both the speaker and listener in Fig. 6 (a) have a knowledge of colour naming/recognition the same time. In contrast, the untrained CQFormer and classifier lack knowl- edge of colour quantisation and image classification. Therefore, we jointly train both the CQFormer and classifier simultaneously to add prior knowledge colour quantisation and image classifica- tion under specific bit of colour. Finally, similar to the theoretical limit of semantic efficiency in Zaslavsky et al.] 2018), we obtain optimal image classification accuracy under the specific bit of colour.
13

---

Under review as a conference paper at ICLR 2023
6.4 DETAILED CLASSIFICATION RESULT
Table. 5 shows the detailed classification result, where the results are the same to Fig. 4
Table 5: Top-1 classification accuracy colour-quantsed images on four datasets with three net- works. We observe that CQFormer is significantly superior MedianCut (Heckbert 1982).
OCTree [Gervautz & Purgathofer, 1988), MedienCut+Dithhe (Floyd & Steinberg 1976) and Color-
CNN (Hou al. 2020) under low-bit bit (less than 3-bit) quantisation levels.
Colour size
Evaltaring
1-bit
Colour
baseline
94.6
MedianCut (Heckbert.
1982
28.1
39.5
77.8
86.7
90.9
92.9
MedianCu-DDibbc FFlydd Steinberp
1976
CIFARIO Krishevsky et al 2009
OCTTcee Gervantz Purgathofes 1988
22.7
40.0
60.6
77.9
85.7
90.8
ColorCNN Hou 2026
53.5
70.5
78.1
80.1
79.2
80.5
CQFormer
baseline
76.3
MedianCut (Heckberr.
1982
11.0
26.0
57.9
66.5
71.3
MedianCut+Diber FFlyyd Steinberp
CIFAR100 Krihhvsky et al. 2009
OCTTcee Gervantz Purgathofes 1988
6.2
13.1
26.8
45.5
57.5
66.4
ColorCNN Hoo al. 2026
17.6
32.9
41.5
46.1
49.3
50.7
CQFormer
baseline
84.3
MedianCut (Heckbert.
1982
19.9
54.8
74.8
81.3
83.0
84.0
MedianCut+ Floydd Steinberp
1976
STL10 Coates et al. 2011
OCTTce Gievvntz Purgathofes 1988
164
31.0
50.2
66.5
78.5
82.5
ColorCNN Hou al. 2026
41.1
65.1
66.9
60.45
68.0
66.9
CQFormer
baseline
69.1
MedianCut (Heckbert. 1982
7.0
24.4
46.5
56.2
60.4
62.5
MedianCut- FFoydd Steinherp
1976
T00
Le & Yang 2015
OCTice Giervantz Purgathffes 1988
3.5
10.3
26.4
45.1
56.8
61.5
ColorCNN Heu al. 2020
9.3
33.9
43.4
46.7
47.8
48.7
CQFormer
6.5 VISUALISATION
As shown Fig. 7 our CQFormer effectively preserves more perceptual structure and similarity. For instance, aeroplane wings, textures of architecture and vehicle windows.
14

---

Under review as a conference paper at ICLR 2023
(a) Original
(b) ColorCNN (1-bit)
(c) ColorCNN (2-bit)
(d) CQFormer (1-bit)
CQFormer (2-bit)
Figure 7: Visualisation of 1-bit and 2-bit colour quantisation.
15


Reviews:
[{"review_id": "y5-Ez9hYRz", "reviewer": "Reviewer_hYs4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents CQFormer: a color quantization transformer. The goal is to artificially discover the color naming system so that even with less bit representation acceptable performance on core vision tasks such as classification and detection can be achieved. Experiments show that the proposed method significantly outperforms related approaches for detection and classification tasks on public benchmarks. \n\n", "summary_of_the_review": "Though the method has some originality and achieves superior performance to related work, there are several weaknesses such as limited ablations, experimental choices, missing technical details, and poor presentation of the paper.\n ", "strength_and_weaknesses": "Strength:\n+ The ideas presented in this paper (i.e. CQFormer, loss formulation, and regularizers) are original to my knowledge. I found Perceptually Similarity Regularisation intuitive and useful. \n\n+ The experimental results show the effectiveness of the proposed approach for both detection as well as classification task. It is impressive that the proposed quantization with just a few bits is able to achieve a decent AP on MS-COCO object detection. The performance is clearly better than the related publications.\n\nWeakness: \n- Upper bounds are misleading. For example, why MS-COCO detection upper bound is this low? Is it the SOTA object detection? I do not think so. If it is the upper bound for the method then it has to be stated and compared against SOTA.\n\n- Are the weights of regularization terms (R) empirically decided? How sensitive is the model for a different alpha, beta, and gamma? Why alpha is chosen as 0 for the detection task? \n\n- Terms in equation 7 are not clearly defined. Such as what is c, (u,v)? Without these definitions, it is difficult to assess the technical correctness of this equation. \n\n- How sensitive is the model to different noises such as color jitter or Gaussian blur? Such studies are important for assessing the robustness of the proposed approach. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "taSAOyyjwD", "reviewer": "Reviewer_SHap", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers learning a quantisation of colour space, and the parallels to how humans learn colour names. \nA novel deep network architecture for colour quantisation is proposed, which is learned by optimising accuracy on a task of interest (image classification or object detection) when using images with quantised colours as input, but also takes into account several regularisation terms. \nExperiments are conducted on 4 image classification datasets (CIFAR-10/100, STL10, and TinyImages-200) and one object detection dataset (MS-COCO). Comparisons are made to an upper bound of non-quantised colours, and 4 existing methods for colour quantisation.\nGenerally, the proposed method performs well compared to others on image classification when few colour quantisation levels are used (4-16) and worse otherwise. For object detection the proposed method is compared to only three alternative methods, and compares consistently better or comparable to those, yet substantially worse than the upper bound obtained by not applying any color quantisation. \n", "summary_of_the_review": "The paper presents a novel color quantisation technique, and experiments that compare it to a number of baselines. \nThe main concerns that I have with the paper is a lack of clarity in some parts describing the methods and experiments, and the positioning wrt related prior work. Overall I'm not convinced that the current paper would make a significant impact in the ICLR community.\n", "strength_and_weaknesses": "Strengths\n\n+ The paper considers a relatively little studied problem, which might interest a (limited) subset of the ICLR audience. \n\n+ Experiments are conducted on two different tasks (image classification and object detection), and using 4 different datasets for the classification task. \n\n+ The proposed colour quantisation method seems more effective that existing alternative techniques. \n\n\nWeaknesses\n\n- It is not clear how the results for the existing methods were obtained. Were experiments conducted by the authors of the current paper? Was code provided by the authors of the existing methods? Please clarify the process. \n\n- It was not clear why an input-dependent approach is used to determine the color palette, while an input-independent network is used to map the input image to quantisation indices in the Annotation branch. What happens if we remove the \"Palette branch\"  and just learn a set of centroids in color space?\n\n- Parts of the paper are hard to follow, in particular the sections 3.4 and 4.4 and 4.5. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VXy6WE2Bia", "reviewer": "Reviewer_bh8b", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a color quantization transformer with the goal of reduding the number of bits needed to allocate the color information, but, at the same time, trying to look at the question: Do machine learning methods make evolve the different color terms as human civilizations?", "summary_of_the_review": "The paper has interest and that idea might be promising; however, I believe the results of this approach are still not yet comprisng with pure perception approaches, and also, the analysis of the appearance of the color names is perfectly guided by one of the losses applied in the experiment, and therefore cannot be really explained out of the quantization effect. ", "strength_and_weaknesses": "The paper has a positive side, on the study of how can we quantize an image without losing a large accuracy in downstream tasks. This said, this is not a novel research line, as it was already presented in ColorCNN. The main novelty of this work is therefore, trying to include perception into this task.\n\nIn terms of results (Figure 5), the proposed method seems to beat ColorCNN is all the cases, and other perception based approaches in low-bin level (1-4). This is positive, but I believe the for number of bins in which there exist an advantage, the accuracy is still not acceptable for the downstream tasks.\n\nRegarding the analysis on the order of appearances of the color names, I believe that the analysis is not of real help; as I am quite convinced the authors are forcing that order of appearance due to the enforcement of the perceptually similarity regularisation. That loss guides the net to actually predict exactly the order of colours in thw WCS, as the probabilities in WCS exactly represent that order of appearance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OcoKnAnNL_9", "reviewer": "Reviewer_4LP2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an end-to-end color quantization transformer that is able to discover the color naming system under the need of perception and machine. The framework contains two branches, respectively for structure mining and color localization. The proposed CQFormer quantizes the color space of the input image and the generated quantized image can maintain recognition accuracy, which is of practical application such as images compression. Besides, the proposed method can provide consistent evolution pattern between the color system and the basic color terms across human languages. The experiments also demonstrate the effectiveness of the method.", "summary_of_the_review": "This paper is generally of good quality and the novel method is supported by the sufficient experiments. I point out some weaknesses of the paper and ask for paper revision. In conclusion, this paper meets the requirement of the conference and I vote for Accept.", "strength_and_weaknesses": "Strength:\n(1)\tThe paper is well-written and description of the proposed method is clear and easy to follow.\n(2)\tThe core problem is novel and has potentially practical usage in vision field.\n(3)\tThe proposed modules are reasonable and the whole experiments indeed support the effectiveness of the method, which shows strong performance improvements compared with previous methods.\n\nWeaknesses:\n(1)\tFrom Table 3, we can see that the CQFormer indeed shows superiority under extremely low bits, such as 1-bit and 2-bit, and the performance maintains relatively stable when the color bit decreases. But it fails to compete with traditional methods like MedianCut and OCTree when the color bit is large, such as 5-bit and 6-bit. Could you provide explanation for this phenomenon, and if there are any improvements that can help CQFormer keep consistent performance superiority across all color bits?\n(2)\tFor the experiments part, the value of C is missing in the Training Settings part.\n(3)\tThe description for the Palette Branch is not enough. For example, the detailed structure of Cross Attention and FFN is not specified or formulated. And I do not know if there is a shortcut alongside or a normalization along with Cross Attention and FFN.\n(4)\tSome typo such as “to to” in the 3rd paragraph of the Introduction, “a explicit” in the last but one paragraph of Page 4. And the author should leave some blank between Figure 5 and the following half paragraph.\n(5)\tI am doubt with the name “transformer” because it seems that the author only implement one “Cross Attention + FFN” layer, which is only part of Decoder of standard Transformer. In my opinion, only the stacks of multiple Encoder layers or Decoder layers could be named as “Transformer”. The implementation in this paper seems like only adopting Cross Attention Mechanism. So I suggest change the name to the other one without “Transformer”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject