Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents CQFormer: a color quantization transformer. The goal is to artificially discover the color naming system so that even with less bit representation acceptable performance on core vision tasks such as classification and detection can be achieved. Experiments show that the proposed method significantly outperforms related approaches for detection and classification tasks on public benchmarks. 


strength_and_weaknesses: Strength:
+ The ideas presented in this paper (i.e. CQFormer, loss formulation, and regularizers) are original to my knowledge. I found Perceptually Similarity Regularisation intuitive and useful. 

+ The experimental results show the effectiveness of the proposed approach for both detection as well as classification task. It is impressive that the proposed quantization with just a few bits is able to achieve a decent AP on MS-COCO object detection. The performance is clearly better than the related publications.

Weakness: 
- Upper bounds are misleading. For example, why MS-COCO detection upper bound is this low? Is it the SOTA object detection? I do not think so. If it is the upper bound for the method then it has to be stated and compared against SOTA.

- Are the weights of regularization terms (R) empirically decided? How sensitive is the model for a different alpha, beta, and gamma? Why alpha is chosen as 0 for the detection task? 

- Terms in equation 7 are not clearly defined. Such as what is c, (u,v)? Without these definitions, it is difficult to assess the technical correctness of this equation. 

- How sensitive is the model to different noises such as color jitter or Gaussian blur? Such studies are important for assessing the robustness of the proposed approach. 


clarity,_quality,_novelty_and_reproducibility: Novelty: The ideas presented in this paper (i.e. CQFormer, loss formulation, and regularizers) are original to my knowledge. I found Perceptually Similarity Regularisation intuitive and useful. 

Reproducibility: I did not check for reproducibility very carefully. There are some technical details that need clarity. The datasets used are publicly available. The paper promises to release the code.

Clarity: The paper is not well-motivated. What is the impact of solving this task? Why someone would work on this problem? The motivation should be such that a reader from an outside area is able to appreciate the work. In the current write-up, it is highly unclear. Even the abstract does not make clear sense. The same applies to the title. It is much later possibly after going through the other related works such as colorCNN and the experiment section the motivation became somewhat clear. 

summary_of_the_review: Though the method has some originality and achieves superior performance to related work, there are several weaknesses such as limited ablations, experimental choices, missing technical details, and poor presentation of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an end-to-end color quantization transformer that is able to discover the color naming system under the need of perception and machine. The framework contains two branches, respectively for structure mining and color localization. The proposed CQFormer quantizes the color space of the input image and the generated quantized image can maintain recognition accuracy, which is of practical application such as images compression. Besides, the proposed method can provide consistent evolution pattern between the color system and the basic color terms across human languages. The experiments also demonstrate the effectiveness of the method.
strength_and_weaknesses: Strength:
(1)	The paper is well-written and description of the proposed method is clear and easy to follow.
(2)	The core problem is novel and has potentially practical usage in vision field.
(3)	The proposed modules are reasonable and the whole experiments indeed support the effectiveness of the method, which shows strong performance improvements compared with previous methods.

Weaknesses:
(1)	From Table 3, we can see that the CQFormer indeed shows superiority under extremely low bits, such as 1-bit and 2-bit, and the performance maintains relatively stable when the color bit decreases. But it fails to compete with traditional methods like MedianCut and OCTree when the color bit is large, such as 5-bit and 6-bit. Could you provide explanation for this phenomenon, and if there are any improvements that can help CQFormer keep consistent performance superiority across all color bits?
(2)	For the experiments part, the value of C is missing in the Training Settings part.
(3)	The description for the Palette Branch is not enough. For example, the detailed structure of Cross Attention and FFN is not specified or formulated. And I do not know if there is a shortcut alongside or a normalization along with Cross Attention and FFN.
(4)	Some typo such as “to to” in the 3rd paragraph of the Introduction, “a explicit” in the last but one paragraph of Page 4. And the author should leave some blank between Figure 5 and the following half paragraph.
(5)	I am doubt with the name “transformer” because it seems that the author only implement one “Cross Attention + FFN” layer, which is only part of Decoder of standard Transformer. In my opinion, only the stacks of multiple Encoder layers or Decoder layers could be named as “Transformer”. The implementation in this paper seems like only adopting Cross Attention Mechanism. So I suggest change the name to the other one without “Transformer”.

clarity,_quality,_novelty_and_reproducibility: This paper is well-written generally but with some minor typo and less clear in some parts. The quality and novelty meet the requirement of the conference. And the method is able to be reproduced.
summary_of_the_review: This paper is generally of good quality and the novel method is supported by the sufficient experiments. I point out some weaknesses of the paper and ask for paper revision. In conclusion, this paper meets the requirement of the conference and I vote for Accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a color quantization transformer with the goal of reduding the number of bits needed to allocate the color information, but, at the same time, trying to look at the question: Do machine learning methods make evolve the different color terms as human civilizations?
strength_and_weaknesses: The paper has a positive side, on the study of how can we quantize an image without losing a large accuracy in downstream tasks. This said, this is not a novel research line, as it was already presented in ColorCNN. The main novelty of this work is therefore, trying to include perception into this task.

In terms of results (Figure 5), the proposed method seems to beat ColorCNN is all the cases, and other perception based approaches in low-bin level (1-4). This is positive, but I believe the for number of bins in which there exist an advantage, the accuracy is still not acceptable for the downstream tasks.

Regarding the analysis on the order of appearances of the color names, I believe that the analysis is not of real help; as I am quite convinced the authors are forcing that order of appearance due to the enforcement of the perceptually similarity regularisation. That loss guides the net to actually predict exactly the order of colours in thw WCS, as the probabilities in WCS exactly represent that order of appearance.
clarity,_quality,_novelty_and_reproducibility: - The paper is mostly clear, although some parts can be improved, for example, the paragraph after equation 2.

- As explained in the previous comment, there is some novelty in combining both perception and machine learning techniques for color quantization, but the results are not enough good in my opinion.

- There are some typos here and there, for example:
               º Missing citation to the papers in section 2.1.
               º would generates (page 5)
               º we locates (page 5)

- Authors comment on the realising of code after acceptance, so it should be reproducible.
summary_of_the_review: The paper has interest and that idea might be promising; however, I believe the results of this approach are still not yet comprisng with pure perception approaches, and also, the analysis of the appearance of the color names is perfectly guided by one of the losses applied in the experiment, and therefore cannot be really explained out of the quantization effect.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers learning a quantisation of colour space, and the parallels to how humans learn colour names. 
A novel deep network architecture for colour quantisation is proposed, which is learned by optimising accuracy on a task of interest (image classification or object detection) when using images with quantised colours as input, but also takes into account several regularisation terms. 
Experiments are conducted on 4 image classification datasets (CIFAR-10/100, STL10, and TinyImages-200) and one object detection dataset (MS-COCO). Comparisons are made to an upper bound of non-quantised colours, and 4 existing methods for colour quantisation.
Generally, the proposed method performs well compared to others on image classification when few colour quantisation levels are used (4-16) and worse otherwise. For object detection the proposed method is compared to only three alternative methods, and compares consistently better or comparable to those, yet substantially worse than the upper bound obtained by not applying any color quantisation. 

strength_and_weaknesses: Strengths

+ The paper considers a relatively little studied problem, which might interest a (limited) subset of the ICLR audience. 

+ Experiments are conducted on two different tasks (image classification and object detection), and using 4 different datasets for the classification task. 

+ The proposed colour quantisation method seems more effective that existing alternative techniques. 


Weaknesses

- It is not clear how the results for the existing methods were obtained. Were experiments conducted by the authors of the current paper? Was code provided by the authors of the existing methods? Please clarify the process. 

- It was not clear why an input-dependent approach is used to determine the color palette, while an input-independent network is used to map the input image to quantisation indices in the Annotation branch. What happens if we remove the "Palette branch"  and just learn a set of centroids in color space?

- Parts of the paper are hard to follow, in particular the sections 3.4 and 4.4 and 4.5. 

clarity,_quality,_novelty_and_reproducibility: # Clarity
 I found the clarity to vary across sections of the paper. Most of the paper is is clear. but some technical presentations and experiments were hard to follow.
For example:
- Section 3.4 the paragraph "colour probability map embedding" was not clear. Is there any fine-tuning of the model involved here, or just an adaptation of an already trained model? I would suggest a more formal description of the method to clarify the exposition.
- Similarly the paragraph "Colour evolution" in the same section was unclear to me. Please add a more formal description of the approach here. 
- The same goes for the last paragraph in section 3.4 "Central colour embedding".  It was not clear to me whether this is an alternative regularisation method used during training, or a post-hoc adaptation of the trained model. 

The captions of Figure 4 and Figure 5 are not separated from the main text, probably by applying negative vspace in the latex source, which is hurting readability of the text, and might be in conflict with the author guidelines.

It was not clear to me whether the image classification and detection models are trained jointly with the color quantisation model, or whether the classification/detection models are pre-trained without color quantisation, and the used as is without further training them when learning the colour quantisation model. 

The purpose of Section 4.4. is not clear. 
For example, the first sentence was unclear to me: "Similar to task centered colorCNN [ref], we are also inferior to the traditional method under a large space." 
The section does not describe any experiment, but rather seems to be an interpretation of the previous experiments. 
The text itself should be clarified by clearly pointing to the experimental results on which it is based, and integrating it in the corresponding section. In its current state it did not bring me any additional insights. 

The colour evolution experiment in section 4.5 it not clearly described. It seem that in the first stage the proposed colour quantisation model is trained to align with the Nafaanra color naming in three colours, and to perform well on the CIFAR-10 classification task. The second stage learns a fourth colour bin, starting from the three colour quantisation. But it seems the design of the experiment already dictates which of the three initial colour bins will be split in the second stage, which seems to strongly direct the outcome of the second training stage. 
Moreover, it is not clear if the alternative color quantisation methods would exhibit similar behaviour, or whether this is unique to the proposed method. 

# Quality

- The paper refers to studies of color naming in the Nafaanra language, but does not explain why these studies are of particular interest as compared to color naming in other languages, nor how the findings for Nafaanra differ from those for other languages. When referring to analysis of Nafaara in the first paragraph of page 2, or elsewhere, there is no reference to the corresponding study.

- The discussion of related work in section 2 is rather limited, and does not succeed in clearly placing the current work in the context of related work, ie which problems of existing methods are solved? What additional insight does the current work provide over others?

- The technical approach lacks a clear motivation. Why is it insufficient to learn several centroids in color space, and assign pixels to the nearest color centroid? What is the motivation for the Palette and Annotation branches of the proposed approach over simpler alternatives? 

- Purity regularisation: it was not clear to me why a max over pixel positions is taken, this seems to ensure that at least one pixel is strongly assigned to each one of the quantisation bins, rather than that *all* pixels are assigned to just one color. Did the authors consider, for example, summing over the pixels the entropy of the assignment distribution?

- When considering image classification, and increasing the number of colour quantisation levels, the proposed method saturates to an accuracy level  that is lower than that for some of the other methods to which is compared (OCTree, MedianCut, MedianCut+dither). A discussion to interpret this result is missing. Is this an inherent limitation of the proposed method? It is due to what? Can it be overcome? 


#  Novelty
 The proposed technique to address color quantisation seems novel to me. The topic of study is relatively little studied, and the findings seem to make a contribution to this literature. The technical tools / architectures used in the work are not novel as such: combining existing architectures for the classification and detection tasks, and presenting a new color quantisation network built using attention blocks and a ResNext component. The paper does not go into detail on how the proposed technique differs from the ColorCNN baseline model to which comparison is made. What shortcomings does the prior work have, and how does the current work address these?


# Reproducibility
the authors say code will be released upon publication. Standard datasets are used. 
Classification experiments are run on a single GPU, for detection models a setup with 4 GPUs is used. 
The replication of the results for the existing methods is less clear: the paper does not discuss how these were obtained.  
It is not clear how the color index map for Nafaanra in section 4.5 is obtained, and whether it is accessible to other researchers. 


# typos 
The manuscript contains quite a few typos, here a are a few:
- Page 3 and 7: "asteroid" --> "asterisk"
- Page 6: in Eq 9 L_High is used, which in the line below L_H is used, these should be the same as I understand it. 
- Page 7: in paragraph "evaluation metrics" repetition in "...top-1 classification accuracy and top-1 classification accuracy..."

summary_of_the_review: The paper presents a novel color quantisation technique, and experiments that compare it to a number of baselines. 
The main concerns that I have with the paper is a lack of clarity in some parts describing the methods and experiments, and the positioning wrt related prior work. Overall I'm not convinced that the current paper would make a significant impact in the ICLR community.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary**: The authors propose a color quantization transformer with the goal of reducing the number of bits needed to allocate the color information, but, at the same time, trying to look at the question: Do machine learning methods make evolve the different color terms as human civilizations?
The paper initially received mixed reviews 3566. 

**Strengths**: The paper considers a relatively little studied problem, which might interest a (limited) subset of the ICLR audience. Experiments are conducted on two different tasks (image classification and object detection), and using 4 different datasets for the classification task. The proposed colour quantisation method seems more effective that existing alternative techniques in some scenarios.

**Weaknesses**:  The major concerns were:

1. upper bounds for MS-COCO detection are misleading. Is the SOTA detector really used? [hYs4]
2. how are the weights for the regularization (R) set? How sensitive is the model to the hyperparams (alpha, beta, gamma)? Why is alpha=0 for detection? [hYs4]
3. How sensitive is the model to color jitter or Gaussian blur? [hYs4]
4. Presentation issues: Terms in equations are not defined well. [hYs4]
5. the paper is not well motivated. [hYs4]
6. Why is the proposed method better than other methods in the low-bit regime, but worse in the high-bit regime? The proposed method saturates to an accuracy level below other methods. [4LP2, SHap]
7. The description of the palette branch needs improvement. [4LP2]
8. Other presentation issues. [4LP2]
9. Although the performance is better than other methods on low-bit regime, this is not a useful regime due to the poor accuracy in the downstream task. [bh8b]
10. the analysis on the order of color names is not helpful, because it is caused by the perceptually similarity regularization. [bh8b]
11. how were the results of existing methods obtained? [SHap]
12. why use an input dependent approach to determine the color palette? [SHap]
13. What if we remove the "palette branch" and just learn the centroids in the color space? [SHap]
14. descriptions are unclear, Sec 3.4, 4.4, 4.5. [SHap]
15. are the image classification and detection models jointly trained with the color quantization model? [SHap]
16. color evolution experiment in Sec 4.5 is not clear. The design of the experiment (3 clusters, then adding a 4th) dictates the outcome. Could other methods exhibit the same behavior? [SHap]
17. no explanation about why the study of color naming in Nafaarnra is of interest. [SHap]
18. limited discussion of related work. [SHap]
19. technical presentation lacks motivation. [SHap]
20. why use max instead of sum for purity regularization? [SHap]
21. the technical methodology is combination of existing works. [SHap]

**Discussion:** The authors wrote a response to address these concerns. During the discussion among reviewers, Reviewers SHap and bh8b were not convinced. Specifically, Reviewer SHap mentioned:
  - One point that stands out for me is that the approach leads to a per-image colour quantisation: that is the same number of colours is used to quantise each image, but the palette can differ per image (as per the palette branch). I asked this in my review, and authors confirmed in the rebuttal. This means that the quantisation does not correspond to a colour naming scheme, as there is not fixed set of colour bins to which is quantised. This puts the complete approach in question for me. (Points 12, 13)
  - On the experimental comparison: the authors say they used code from previous work te compare to those methods, but do not provide links to repositories where such code can be found. (Point 11)
  - I'm not convinced why only a single human color naming system is compared to with only 3 or 4 colour names. A stronger case of similarity to human colour naming could be made by comparing other human colour systems. (Point 17)
  - Also after the rebuttal, it was not clear to me why the colour naming evolution experiment cannot be conducted with baseline methods, so that comparison to those can be made.  (Point 16)
  - Overall, I feel the paper has a number of significant shortcomings, and needs a complete overhaul to address these before being ready for publication. 

Reviewer bh8b mentioned:
  - I am not convinced at all for the answer given to me about the perceptual loss not being important in the selection of colours. Basically, by enforcing closer colors in the color space, you are somehow already marking the size and distance among clusters you can have in that space. This is specially worrying as distances are also computed in Munsell space (and not in a perceptual uniform one as CIELAB or CIECAM). (Point 10)
  - Also, I still believe the numerical results are not good enough. (Points 6, 9)



justification_for_why_not_higher_score: The AC agrees with the two reviewers on their remaining concerns. Specifically, the color naming/evolution experiment could have been more rigorous: comparisons without there human coloring systems could be conducted, baseline methods should be compared. The perceptual loss uses a particular color space, which determines the color clusters. Would using a different color space lead to different clusters, that are similar to other human coloring systems?  Finally, the numerical results were also not impressive; while the proposed method performs better on low-bit regime (1-3 bits), it performs worse on 4-6 bits, and its performance appears to saturate. Thus using more bits (7-16) may not lead to improved performance, which is worrying. Perhaps some hyperparameters could be tuned to better trade off the color-based and task-based losses. 

Overall, the paper needs a major revision to fix these issues.

justification_for_why_not_lower_score: n/a
Decision: Reject