Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: The paper proposes a method to generate CUD (color universal design) images with a deep learning based image enhancement method. The output CUD images are more discernible to those challenged by red/green color blindness (protanopia/deuteranopia). The network architecture proposed to generate CUD images consists of three steps -- i) feature layer wherein 512 dimensional features are extracted from concatenated input images, ii) fusion layer to combine the 3 features, iii) Finally an attention layer to generate an enhanced feature. Subsequently, 64 values are regressed from the enhanced feature to estimate a piecewise linear function for S/V channels of the output image. As stated in Sec 2.4, different loss functions are used -- stencil masking (only change areas that challenge deuteranopia vision), CIELab loss to enhance contrast, histogram loss to preserve original colors etc. Quantitative evaluation is performed in Tab. 1 which measures color preservation and contrast enhancement by comparing SSIM/PSNR metrics between predicted image (I^\hat) and original image (I), target image (T). In Fig. 5, protanopia/deuteranopia rank different methods based on distinguishability of output CUD image.
main_review: Strengths:
+ The paper addresses an important problem of using deep-learning based image enhancement techniques to generate CUD images which are distinguishable by those with red/green color blindness
+ Test subjects in user study of Fig. 5 prefer the CUD images generated by the proposed CUD-NET method over other baselines and original input

Weakness:
- The details in the paper are difficult to follow and the paper writing can be improved. For example, 
i) L118-119 "The training data is grouped about 1,600 color combinations into the same V and then simulates them with the deuteranopia vision, converting to the adjacent color family to comply color preservation." What is the process of simulating the images with deuteranopia vision? How does the conversion to adjacent color family work? The authors should explain these details with equations if necessary.
ii) L139-140 "After applying color inversion from the original RGB input image In, we generate the image Id with
an aspect of deuteranopia vision in equation 1." It is not clear how I^d is generated, can the authors explain? 
iii) L240-242 "Color experts has refined about 1,500 vectorized image for the training data and 300 publication images for the validation data. All the comparative experimental models used the same train, test, validation data in this paper." The authors should describe how they sourced the images, what annotation/enhancement process was used to generate supervision (i.e. enhanced/CUD images), and what are the train/val/test splits?

- Sec 2.4 describes various loss functions, however it is unclear how each of them helps improve the performance. An ablation study should be performed by training the network with different loss terms and measuring the effect on quantitative/qualitative metrics in Sec 3.

- Sec 2.3 describes the model architecture, however some design choices are not explained clearly
i) For shared convolutional layers, why is a standard architecture (possibly with pre-training) e.g. residual network not used? 
ii)  Does pooling and split attention mechanism help improve the performance? Can the authors demonstrate that using pooling + attention is better than simply concatenating the 3 512-dimensional features and regressing the 64 values of piecewise linear function?

The above weaknesses need to be address for the reader to get insights into the proposed method and its results.
limitations_and_societal_impact: The authors propose a method to generate images which are distinguishable to those with red/green color blindness. They perform a user study with 6 subjects to demonstrate increased distinguishability of the images generated by the proposed method. Larger study needs to be performed to ascertain the performance and failure modes of the proposed method before it can be deployed to assist those with color blindness.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper focuses on color universal design to satisfy both color preservation and distinguishment of color for the color weakness. To achieve that, the paper refines the color universal design dataset on specific criteria by color experts, expands the input image information via pre-processing, proposes a multi-modality feature fusion architecture in CNN, and suggests a deformable loss function. The experiments are conducted on the provided dataset. Besides, a user study is carried out. 
main_review: Pros:

+This topic is interesting and meaningful. Although there are numerous image enhancement methods, they are not suitable for color universal design. The paper raises this topic and provides some insights.

+The provided dataset and pre-processing are valuable for this task.  

+The paper provides sufficient experimental results in the supplementary material. 

Cons: 

-Although a new research topic is raised in this paper, the novelty of the paper is limited to me. Besides the dataset, the multi-modal feature fusion, and loss function are intuitive and straightforward. What the results would be like if just using a simple network such as a Unet-like network? With the same training data, a simple network may achieve similar performance under the constraint of the l1 or l2 norm.  

-The experiment is not much convincing to me. The proposed method is compared with several image enhancement methods. It is not clear if the compared methods are training using the same training data as the proposed method. Some compared methods are un-supervised or zero-reference-based. Thus, strong baselines should be included for comparison. For example, a supervised method is trained under the same training data as the proposed method. The subjects in the user study are only six. More subjects should be invited for a stable statistic. Moreover, the performance of the user study shows that the proposed method just achieves relatively good performance when compared with these baselines. However, the accuracy of the proposed method is far from the practical applications in real cases.

limitations_and_societal_impact: The author has clearly presented and solved the limitation and potential negative societal impact.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper proposes a deep-learning based method to create (vector-graphic) images that enables color deficient viewers to identify every object in the scene. The method boosts the contrast and/or modifies the hue of colors that are confusing to protanopes and deuteranopes. The method has a pre-processing step, a multi-model feature fusion architecture, and a variational loss function. Comparisons were carried out with GANs and image enhancement methods.
main_review: While the idea and project is certainly worthwhile pursuing as it benefits vision impaired people, this paper is not yet ready for publication due to several issues addressed below:

-	SOTA: There are a large number of algorithms out there that correct images for protanopes and deuteranopes, many of them listed in reference [RG19]. A discussion of some of them, conceptually closest to the method presented here, would improve the SOTA. But more importantly, a comparison to some of these methods in the Experimental Section would very much strengthen this paper.
-	Experiments: As already alluded above, I do not consider the comparison to GANs nor image enhancement methods sufficient to validate the proposed methodology. A comparison to at least some of the color-blind correction methods is needed. Also, PSNR and SSIM are not really relevant metrics here, as they do not measure perceptual color differences. Some Delta E variant would be more appropriate. I appreciated that a psychovisual study was done, but that one needs to be better described on what exactly the methodology was.
-	The proposed method so far only works for vector graphics. There are of course some applications for that, but it is limited. I encourage the authors to also extend the method to natural images. I do acknowledge that creating an appropriate ground truth database will not be trivial, thus I find a comparison to existing color deficiency correction algorithms even more important.
-	Lab Loss Function: I do not understand the motivation for the Lab loss function. Perceptual differences in Lab Color space are measured as an L2 distance which is equal to Delta E, an (almost) perceptual metric. Here L1 is used, as well as MS SSIM. Considering we are dealing with vector graphic images I am not sure SSIM is meaningful. I would like to see a better discussion of this Loss function.
-	English: The paper is hard to read as there are many grammatical mistakes. A good proof-reading by a native English speaker will certainly improve the quality of the manuscript.

limitations_and_societal_impact: The limitations and societal impact is mentioned
review_title: null

Meta-review: Reviewers agree there are fundamental deficiencies in this submission. One is the lack of baselines for the task, some of which pre-date deep learning methods. Their addition is key to improving the paper and the comparison of the results. 

All reviewers also mention the lack of clarity in the writing that makes it difficult to extract the details of the contribution. Unfortunately, I recommend rejection.
Decision: Reject