Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper works on Unrestricted color attacks, which manipulate semantically meaningful color of an image.
Current works usually sacrifice the flexibility of the uncontrolled setting to ensure the naturalness of adversarial examples. As a result, the black-box attack performance of these methods is limited. To boost transferability of adversarial examples without damaging image quality, they propose a Natural Color Fool (NCF) which is guided by realistic color distributions sampled from a publicly available dataset and optimized by our neighborhood search and initialization reset. Extensive experiments and visualizations demonstrate the effectiveness of their proposed method.
strengths_and_weaknesses: The writing and exposition are clear and of high quality. The authors generally address this problem in the style of CV instead of ML.

1. By exploiting the color distribution of semantic classes, the proposed Natural Color Fool (NCF) improves the flexibility of the current unrestricted color attack.
2. Some experimental improvement is obtained.

Problems:
1. The proposed method is mostly empirical without theoretical proof. For example, why the proposed method can improve flexibility?

questions: 1. The proposed method is largely based on a segmentation network and a clustering strategy. What is the impact of different segmentation and clustering algorithms? Also, the clustering is based on the ADE20k dataset. Can this guarantee performance?
2. In the main method, it is unclear to why a random color can successfully attack the classification model with such a high success rate?
3. The main idea seems to be similar to [42]. [42] is a different application, but technically what are their differences?
limitations: Authors have addressed some limitations and potential negative social impact of their work, what about the feasibility of the attack in the physical world?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a color based unrestricted adversarial black box attack on image classification deep neural networks by transferring the adversarial examples using a substitute network. Authors propose to generate a natural color distribution library based on the publicly available ADE20K dataset. They create a library of distinct color distributions for 150 semantic classes. They generate the adversarial examples by randomly picking several color distributions for each semantic class from the library and find the image that fools the substitute network. In addition, they perform neighborhood search on a 3x3 Transfer Matrix T that performs the color mapping to further boost the attack success rate. Furthermore, they reset the Transfer Matrix T like random restarts in PGD attack. Extensive results show that the proposed method maintains the image quality and boosts the attack transferability significantly compared to the existing methods. Major boost of the transferability comes from optimizing the matrix T to generate adversarial examples.
strengths_and_weaknesses: Strengths:
1)	Well written paper. Most of the parts are easy to understand.
2)	Proposes a novel method to generate transferrable adversarial attack.
3)	Method explanation is easy to follow.
4)	Conducted extensive experiments on wide variety of network architectures.
5)	Shown a significant improvement of the attack success rate with the proposed method on both undefended and defended models (L_p based defenses and input processing defenses).

Weakness:
1)	In the beginning of the paper, authors often mention that previous works lack the flexibility compared to their work. It is not clear what does it mean and thus makes it harder to understand their explanation. 
2)	It is not clear regarding the choice of 20 distribution sets. Can we control the number of distribution sets for each class? What if you select only few number of distribution set? 
3)	The role of Tranfer Matrix T is not discussed or elaborated.
4)	It is not clear how to form the target distribution H. How do you formulate H? 
5)	There is no discussion on how to generate x_H from H and what does x_H constitute of? 
6)	Despite the significant improvement, it is not clear how this proposed method boost the transferability of the adversarial examples.

questions: In addition to the points mentioned in weakness, I have two additional questions:

At line 129, it is mentioned that “Since the color distributions in each distribution set are similar and averaging all color distributions of a specific distribution set may not yield a natural representation, we randomly select a color distribution to represent the overall color characteristics for simplicity”. If their distributions are similar, why not select a single color distribution as a template from each set? What does the natural representation mean here?

In line 141, authors mention that “we craft adversarial perturbations in CIELab color space where the perception is more uniform than RGB color space”. What does the perception is more uniform mean here? How does that help to create adv examples?

limitations: As per my understanding, authors briefly addressed the limitations and negative impact in their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new unrestricted attack (NCF) that boosts the black box performance. It is done by color converting, neighborhood search, and initialization reset. The experiment includes both CNN and Vit architectures and compares a host of baseline methods. Overall, NCF achieves state-of-the-art performance. 
strengths_and_weaknesses: 
Strengths:
1. The paper specifically targeted on black-box setting and achieved much higher adversarial transferability than baseline methods. 
2. The authors consider a lot of recent ViT models and lots of defense methods to evaluate. 

Weaknesses: 
1. The authors state that "current unrestricted color attacks lack flexibility, which results in limited transferability of the adversarial examples". First, I am not exactly sure what flexibility means here. Then, I observe (from table 1) that other attacks usually outperform NCF in the white-box setting. If NCF is more flexible, shouldn't it also have a higher white-box attack success rate? Authors should explain why NCF is better on transferability and worse on white-box attacks. 

2.  I noticed that all black-box attacks are generated from one substitute model, whereas in some real-world settings, attackers may have multiple substitute models. Could the NCF and baseline methods improve if more substitute models are used?  This paper [1] is also worth to be discussed. 

[1] Baia et al. One for Many: an Instagram-inspired black-box adversarial attack


========================================================================
After rebuttal: 
I think the authors carefully address my concerns, and I am willing to increase my score. 


questions: See previous section
limitations: NA
review_title: null

Meta-review: metareview: The proposed approach exploits the color distribution of semantic classes, thus improving the flexibility of the current unrestricted color attack. This method generates novel transferrable adversarial attacks. The authors conducted extensive experiments on wide variety of network architectures. A significant improvement of the attack success rate is achieved with the proposed method on both undefended and defended models. 
confidence: Certain
Decision: Accept