CUD-NET: Color Universal Design Neural Filter for the Color Weakness
Authors: Sunyong Seo
Hana Kim
Jinho Park
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Computer Vision
Image Enhancement
Color Correction
Color Universal Design
Color Weakness

Abstract: Information on images should be visually understood to anyone, including the color weakness. However, it is not recognizable if color that seems distorted to the color weakness meets an adjacent object. We suggest CUD-NET based on convolutional deep neural network to generate color universal design (CUD) images that satisfy both color preservation and distinguishment of color for input images. CUD-NET regresses the node point of the piecewise linear function based on information of input images and comprises a specific filter per image. We present the following methods to generate CUD images for the color weakness. First, we refine the CUD dataset on specific criteria by color experts. Second, the input image information is expanded through the pre-processing specialized on the color weakness vision. Third, we suggest a multi-modal feature fusion architecture that combines features to process expanded images. Finally, we suggest a deformable loss function by the composition of the predicted image through the model to avoid the one-to-many problems of the dataset.

CUD-NET: Color Universal Design Neural Filter for the Color Weakness
Anonymous Author(s)
Affiliation Address
email
Abstract
Information on images should be visually understood to anyone, including the color weakness. However, is not recognizable if color that seems distorted the color weakness meets an adjacent object. We suggest CUD-NET based on convolutional deep neural network generate color universal design (CUD) images that satisfy both color preservation and distinguishment of color for input images. CUD-NET regresses the node point the piecewise linear function based on information of input images and comprises a specific filter per image. We present the following methods generate CUD images for the color weakness. First, we refine the CUD dataset specific criteria by color experts. Second, the input image information is expanded through the pre-processing specialized on the color weakness vision. Third, suggest multi-modal feature fusion architecture that combines features to process expanded images. Finally, we suggest a deformable loss function by the composition of the predicted image through the model to avoid the one-to monnto problems of the dataset.
6
10
12
14
Introduction
15
16
1.1
Motivation
The green and red color blindness made up of 8% of males and 0.5% of females in Northern European descent[Won]]] blindness is the most common pattern, followed by blue, yellow, and total color blindness, In this paper, we generate Color Universal Design (CUD) images, which are color weakness friendly design forms, through deep learning around the aspect of the red color weakness (protanopia) and green
18
which is almost up to rate of one person in 20 people. Green and red
19
20
21
22 color weakness (deuteranopia) vision. Protanopia is insensitive to red color and deuteranopia is insensitive to green color, although it varies depending on individual color weakness extent.
23
24 There are studies that help color discrimination to the color weakness, including wearable devices and surgeries (V7CC22) [VZCR20] However, since these research require time and cost, we simply generate CUD images with an image enhancement method based on deep learning to make the corresponding color visible for the color weakness. For an example of the left-above image in Figure the people who are not color weakness can distinguish the letter '5' in the image. But as a deuteranopia vision in left-below image 1d the surrounding color and the letter '5' are very analogous, making it ambiguous
25
26
27
28
29
30 to distinguish the bound of adjacent object. The right-bottom target image 7 refined image by color
31 expert designers, shows that the letter '5' appeared well at the deuteranopia vision. Here, we define
32 the non non-CUl CUD objects as the letter '5' and surroundings invisible to deuteranopia vision in the image
33 I, and define the CUD objects as the letter '5' and surroundings visible to deuteranopia vision
image T. In other words, CUD object means that adjacent objects are distinguishable on both the
Cooee available http://///ghhhhh..///oooooooooss
anonymous
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

non-CUD Input
CUD-NET
CUD Target
Normal vision
5
5
5
(1)
(
(T)
Deuteranopia vision
5
5 (7
(IS)
(M
Figure 1: Comparisons of the non-CUD image, our CUD-NET's predicted image, and CUD image. The above row is represented in normal vision, and below row is represented in deuteranopia vision.
normal vision and the color weakness vision. The non-CUD object means that adjacent objects distinguishable normal vision but not the color weakness vision. Consequently, we generate î that satisfies CUD with a specific filter to the image T.
37
We want to apply weak filter possible to CUD objects to preserve color, which requires certain level of object comprehension mechanism to so. There are various studies from classic sraningg-bsedd object segmentation methods[TSSC2 ZGL"20 to define specific objects or areas in image. The research on semantic segmentation, which even provides labels between objects, seems that deep learning still does not have complete comprehension of all objects
39
PCA [WEG87] to machine
41
43 in the real-world. The visual question answering to arbitrary questions about object's 's interactions, the most general issuc comprehension of object, does not have high transmission power to be 18, KZG 17 LYL.20) Therefore, we expand feature of the input image around the information of color weakness vision and define the robust neural filter. In summary, suggest
45 practical uses AHBB [ses[AHB"
47 a CUD-NET that generates an image suitable for CUD, while complying with the color preservation
48
for the source image.
49 this paper, we suggest Color Universal Design Network (CUD-NET) to satisfy both color preservation and contrast of non-CUD objects (CUD suitability). We introduce core contributions of CUD-NET.
51
52
Dataset refinement criteria for CUD image We refine training data into two groups, the with simple color tone image based H and V in the HSV color space, the other with two or more non-CUD objects that must be distinguished in publications.
54
55
Image pre-processing for CUD-NET We carry out pre-processing to expand the infor- mation of the input image. Input image is reconstructed with three expanded feature information with noise removed.
57
58
Multi-modal feature fusion architecture We define a feature layer, the fusion layer, and regression layer to handle pre-processed images. The three features from the feature extracting layer are combined into the one fusion feature, and finally a filter is constructed by regressing the node point of the piecewise linear function, indicator of filter.
59
60
81
Variational loss function We suggest deformable loss function by the composition of the predicted image through the model. Our data have problem of one-to-mane where the specific color input image is mapped into multiple colors in target image T'.
83
64
85
1.2
Related Works
Image-to-Image translation based on GAN GAN is used in various image translation areas, including image generation, style transfer, and colorization(KWK22 [ZZE17]. In preliminary experiment, CycleGGANIPEZZ20 has reached the best performance in maximizing the contrast of
87

---

Contrast
Contrast
Filter adjustment
by
Color
Color
Color space of input image
Color space of predict image
Figure 2: The ideal color conversion of predicted image between contrast and color preservation. Non-CUD object should increase the gap compared to the input image and preserve its original color, while the CUD object maintain both contrast and color.
non-CUD objects. However, our goal is to keep the color preservation of the input image as well, 70 so the case of black color, which has lost all its color of the input image, it is considered the
71 worst case for color preservation. Enlighten Enlighten-GANNGLL GANGGL 21] complements those instability, enabling
72 them to generate more stable results on color preservation. But since most of the GAN-based image translation fixes the size of the predicted image. reshaping high-resolution image causes information
74 loss of source image. Also, it is difficult to reconstruct the complete geometry for the source image it generates images through the dilated convolution layer.
78 Image enhancement based on neural filter estimation Unlike GAN, there are researches that scale the pixel values of images based neural filter estimation (arraattioonnaaaaaaaaa [WZF 19 DLT18 BCPS19]. Zero- DCE|GLG 20 a low-light image enhancement research that provides a brighter visual display of input image. It estimates pixel-wise and high-order filter for dynamic range adjustment of input images with lightweight deep network, DCE-Net. DepLPF[MMM' 20) tried to solve the problem by using a graduated filter, elliptical filter, and polynomial filter. The authors not only tried to visually enhance the contrast of images but also to comprise stable filters that are easy to understand for the spectators while keeping the color preservation. In our problem, however, the contrast factor is almost same results as the input image in both visions, while complying the high color preservation, resulting over-stable filter. It assumed that the inability in comprehension of object's interaction leads to over-stable filter.
78
80
81
82
83
84
a5
86
2
Methodology
We define the ideal predicted image as an increase the contrast between non-CUD objects and the color preservation for the input image. The non-CUD object should be mapped into á and CUD object b should preserve its color and contrast like an ideal example of Figure 2. However, as our neural filter affects the whole pixels throughout the image, we have the constraint of applying the 92 same filter to objects a and b. It is very hard to make the contrast and color of object exactly the same as before the filter adjustment while maximizing the contrast of object a. Therefore, propose 94 a deep learning-baat based regression to comprise the specific filter per image that maximizes the contrast of object while minimizing the adjustment of features object b.
86
90
96 First, we propose a solution maximize the contrast of the channel values in CIELab color [RG9). We empirically confirmed that protanopia and deuteranopia, which account for the most proportion of color weakness, can distinguish the difference by L channel values in common when the non-CUD objects are adjacent to each other. To illustrate Figure I again, the L channel value of letter '5' in image is 61 and the surrounding color is 61. The distinguishment between the two objects is easy to normal vision, however the image 1d, the deuteranopia vision, is very ambiguous. On the contrary, the CUD target image T and 7 have a difference of channel value 75 for the letter '5' and 45 for the surroundings, making it easy to distinguish between the normal and the deuteranopia vision. Due to the characteristics of these data, we refine a data pair by defining a criterion that separates two invisible non-CUD objects by L. channel values.
97
98
100
102
104
105
106
Secondly, we propose variational loss function and multi-modal feature fusion network for color preservation. It can be said that the increase in the contrast of L channel values between non-CUD
3

---

108
objects is quantitatively superior, but not in the case of increasing the differences in color preservation of imput images. When non-CUD objects exist, as simple example, the most likely way to maximize contrast is to polarize the color of the object black and white. But it is the result of complete ignorance for color preservation, so just enabling to distinguish between non-CUD objects not always good answer. A strong filter must be applied to distinguish non-CUD object, but its impact should not be
109
110
111
112
113
too extensive to leading the loss of information in CUD objects. In this paper, we solve this problem by taking an appropriate trade-off of color preservation and contrast of L channel value.
115
2.1
Dataset refinement criteria for CUD image
The training data is refined by two groups. The one is vectorized image with two colors divided by value V and hue degree H in HSV color space space [MK019), and the other is image with two or more objects that must be distinguished while preserving the color of non-CUD objects. The training data is grouped about 1,600 color combinations into the same V and then simulates them with the deuteranopia vision, converting to the adjacent color family to comply color preservation. All conversions are scaled within only S and V in HSV color space to increase at least 15 difference the L channel value of selected non-CUD objects. The colors are combined with the 10 essential H 125 and tones, and the similar color simulated with the deuteranopia vision was converted. Consequently, the key part of refining training data is preservation of color, allowing the models to comply with the same approach on learning.
116
117
118
119
120
122
124
126
2.2
Image pre-processing
127 Our model regresses node points of piecewise linear function, which will be described in the model architecture section, and the final filter is a multiplication operation for the input image. Therefore, the multiplication operations of less than the number tend fade the color saturation. The image without color inversion converges the white color value to I, so if the multiplying value is in the [0,
128
129
130
131
I] range, the white color is shifted the black. By inverting the color of input image, ignores the multiplication operations for white value with O.
132
133
We generate the map image 1" based on original RGB input images calculating the difference value between the image with an aspect of normal vision and the image with an aspect of deuteranopia vision. Recent studies have been conducted to augment the information or expanded the models' perspective RFB15] In our experiment, however, the transformer model tends to generate the predicted image ignoring the source color, which result in the polarized color to black and white like Cycle-GAN's.
134
135
136
through transformer
137
138
m = invert(("") - invert(Td)
(1)
I = (catchannel (I", 1d, Im)
(2)
After applying color inversion from the original RGB input image I", we generate the image Id with an aspect of deuteranopia vision in equation 1. From these two generated images, we can get the absolute difference value to compose the map image I" In equation 2, the final input concatenated with 9 H W dimensions passes through the model. The 8(.) clips output to a range of [0, 1].
139
140
141
142
2.3
Model Architecture
144
CUD-NET regresses the node points of piccewise linear filter function from the input. The value of each node points computes the multiplication operation and generates the predicted image. The input I is compressed into 3 feature blocks matching each input through convolution layer, pooling layer, and global pooling layer. The input with 9 channels is separated into 3 x 3 channels before passing the model. The first channels are literally used as the main inputs, where the multiplication operation takes place, while the remaining 6 channels are used as features.
146
148
150
First of all, we use multi-modal fusion architecture for three separate inputs to extract expanded features. The three inputs converted to the HSV color space pass through a weights-sharing convolu- tion layer to extract a feature block corresponding to the inputs. Each convolution layer consists of kernel size=3, stride=1, and padding=l, reducing dimension through average pooling. We empirically
152

---

Conv2d
Ploiing
Kernel size 3x3 Padding =
MCB
MCB
Stride
MCB
Pooling Average Activation
Fusion Feature 512 x
Pooling block
Foature 612 x
OOO 000
byel Starion
52
curve
Spit
32
Fusion Feature 512 x x
Enhanced Feature
curve
Figure Overview structure of CUD image generation
154
noticed that the most values of output feature have distribution within the range of [-1, 1] with valid values for constructing the node points, so we use hyperbolic tan for activation function. Since we use inputs with unstructured image size, the last global pooling block holds the size of the feature instead of the average pooling block[LCY14].
155
156
157
158
The three feature blocks are combined through the multi-modal compact bilinear pooling gate (MCB)FPY*16) following the fusion process shown in Figure 3. The MCB gate allows both features to interact in multiplicative way with low memory consumption and computation times. The fusion features are complemented to enhanced feature through the split attention mechanism (WZ*20). At the beginning of the experiment, we have applied the convolutional block attention mechanism (WPLK18) of each MCB gate, but found that it does not make sense of understanding the feature itself, so we apply only one attention block to the last fusion feature.
180
162
163
164
The enhanced feature pass through the fully-connected regression layer. We picked the 64 points to be regressed to compose the piecewise linear function, which is empirically confirmed to the optimized number of points in this research. The first half of the values construct the node points of the S channel and the other half comprise the V channel in HSV color space. Finally, node points become the scaling factors generate predicted image in equation 3 MMS19
186
188
M-1
(( = Ko +
Σ 1==0
(km+1 km)) 8 (M m)
(3)
170
The total number of node point M, each pixel values of S, V channel in input image 1 are multiplicated with the slope actual regressed value km. the m th generated node point. The specific node points M is scaled through a multiplication operation to pixel value of the input image according to each node point.
171
173
174
2.4
Loss function
Our dataset has one-to-many problems between input and target data. In dataset pair (i). (42, T2), .... (In Tx), for example, the red color in Z can targeted to purple color in Tr. and the red color in I can be targeted to orange color in T2. With these one-to me-to-many dataset structures, we design the loss function L that expresses the potential and the diversity of predicted
176
178
image in equation 4.
Labioss (v(w (I:))
Hioss ( (1.1.)
C -
(4)
180 Stencil Masking As explained the dataset refining criteria, we not proceed with color conversion for all areas in the target images, but only for areas with color combinations that are
5

---

182
invisible to the deuteranopia (non-CUD object). For this reason, the imput image has color regions of converting color and unconverting color, which also can be referred to as non-CUD object and CUD object. To imply the color bound to model, the stencil masking method is introduced.
183
(1+) - l.s (1g Tsj)
(1s)
(5)
185
We consist a stencil maps through the logical and operations '.' of each pixel value Tij. Tij. Stencil map can specify the non-CUD area and be computed with predicted image ly of logical operation *I in equation 5. Consequently, CUD object of the image adjusted with stencil mask does not carry the neural filter computation, such the same way refine the target image. This refined image is calculated on the loss function.
186
187
188
189
190
We use the CIELab channel loss function maximize the contrast of color on deuteranopia vision. To stabilize the contrast and brightness of the predicted image, we calculate the MS- SSIM(multi-scale structural similarity (SBB3) of channel.
CIELab Loss
181
192
-1 Lab (1:90) - Lab (T736) ||, + MS-SSIM (Lab (1l)). Lab (T!))
Labioss =
(6)
193
The Lab (.) expression in equation 6 returns the CIELab channel corresponding to the RGB channel, and all calculations are made only on the I. channel.
194
195
Histogram Loss We use the histogram loss function to comply with the color preservation of the image. The RGB channel is used to preserve its color, contrary using only the L channel in other loss functions. Handling the RGB channel as loss function rather than using Lab's 's ab channels has shown better results on color preservation.
196
197
198
Hioss = Whisst / (pro0 -))
N NN (Tr
(7)
199
When simply designing a loss function with the LI distance of the RGB channel pixel values, it was very sensitive to certain values and the gradients diverged, resulting in untrainable experiment. Therefore, we used gaussian expansion method[SAC*II denoted by N.) to infer differentiable
200
201
202
histogram loss function equation 7. We compute the difference of the RGB channel of the differentiable histogram function, which can be altered to mean squared error or cosine similarity.
203
204
The scaler Whist is determined in inverse proportion to the size of the input image. By maintaining the RGB similarity between the predicted image and the target image, we can comply with the color preservation.
205
206
207
Variational Prediction There are various ways to maximize difference of the L channel in the image. And the target image is converted at least two colors compared to the input image. However, the predicted image of the model is generated by the neural filter, so it is unpredictable which area of color modified. Therefore, the color in predicted image is over-shifted in the color value of opposite shifts to the target, the loss will rather increase. In addition
208
209
210
one-to-many problem that the data pair itself does not matches one-to-one a particular color, it is necessary to generate alternative predicted image with the same aspect of the data pair. We calculate the loss function with
212
214 a variational prediction based on the predicted image for potential color shifts.
The first potential is the case of excessive shifts. Assume that 1.- = (74, 41, 79), Ff = {97, 10, 70). T (50, 41, 80} in L channel value. The first and third components of each image is non-CUD objects, and second component is CUD object. Therefore, we refined data paired with a difference of
215
216
217
15 on L channel. Here, we clip the excessive L channel value in 1.6 by equation 8. Up to this point, no calculation is made as no value is exceeded in this example. The second potential the case of opposite shifts. It can be said that complete neural filter has been proceeded for value 97, 10, 70 where L channel difference 27. However, if we actually calculate the mean square error between
218
219
220
221
it and T.L, it will be an large value over k Here we can generate alternative predicted image from
222
equation 9 and 10.

---

max
lij. Tip)
Iij > Tij
clip (Is)
(8)
min
Tij. Tig )
lij Tij
R1 211 Tej, R2
Pig (9)
v (Ig) argmin (Ietip (R1,) - Tilll2) (10)
As mentioned above, we define thresholds by the maximum and minimum value of each corresponding pixel position of 7/6 and T. By computing difference of residual map and the input image, induce the alternative two images R1, Rg. As result, I with smaller L2 distance selected to alternative predicted image in equation 10, and it is finally computed with loss function compared to
226
227
the T.
The above equation establishes V (..) (1.)) = (54, 41, 80) and the mean square error to the target image is approximately 5, which is agrecable loss value respect to 1. itself.
228
Idettity Loss ZPIE17 TPW16 TPWI6 We use Lidentity (Tt) to apprehend the CUD object to the model. In the case of target image that already satisfy the CUD, the filter should be relatively weakly applied than input image. The input of identity loss is target image Tij instead of input image Tiy. and the reference of the loss function also target image Tiy to maintain the value itself. In computing identity loss, we do not require variational prediction as we cannot judge the potential region by equation 10.
230
231
233
235
3
Experiments
237
The experiment was performed with Tesla V100 SXM2 and Intel Xeon Gold 5120 and the computation speed was about 40 images per minutes. We refined a dataset with Adobe Photoshop maximize contrast in the L channel by adjusting saturation and brightness for areas that require color conversion based on deuteranopia vision simulation. Color experts has refined about 1. 1,500 vectorized image for the training data and 300 publication images for the validation data. All the comparative experimental models used the same train, test, validation data in this paper. We used the inference data in publications, which is almost composed of vectorized images, colors often appear distorted in gradation -ich image. The Figure 4 is arranged in descending order of the number of combinations in colors from the top image.
238
239
240
242
244
245
246
Both structure similarity (SSIM) [BSS044 and peak signal to noise ratio (PSNR) in Table 1 can indicate whether the image is suitable for CUD or not. As a notable aspect, the result has shown that comparative models with lower metrics are sensitive
247
248
gigghgradttion input images, which generated color-heterogencou image. SSIM and PSNR itself can determine the increase in contrast compared to the target image but do not determine whether the color preservation complied. Therefore, we evaluated SSIM and PSNR with three references, inputs images I, predicted images f, and target
249
250
251
images T. The higher the estimation of the Î and J, the more color preservation factor worked.
252
The higher the estimation of the I and T, the more increase in contrast can be considered. We also define SSIM mean absolute error, PSNR mean absolute error to measure the extent of the conversion between the F I + T and F I - î in equation 11 and 12, respectively. The N is total number of inference data.
253
255
MAIM-MAE-
SSIM (i, T.) - -SSIM (I,
(11)
PSNR-MAE- 1 N
PSNR i, T.) (S, (12)
N
7

---

Input
Cycle-GAN
Zero-DCE
DeepLPF
Enlighten-GAN
CUD-NET
COLOR
COLOR
COLOR
COLOR
COLOR
COLOR
Color
Coler
Color
Color Universal
Color Universal
Color Universal
Universal
Universal
Design
Hksigg
Design
Design
Design
Design
Figure 4: Comparisons of predicted images in deuteranopia vision. The color experts selected the validation data that do not satisfy the CUD publications.
Architecture
SSIM [,T)
PSNR [,
PSNR (,TT)
SSIM-MAR MAR
PSNR MAE
Cycle-GAN
0.630
0.634
13.28
13.83
0.3191
8,4430
Zao-DDE
0.924
0.888
21.95
18.67
0.0661
3,7300
0.850
0.831
26.31
20.34
0.1
2,0566
EnlightenGGAN
0.820
0.808
21.85
19.58
0,1470
3.9983
Ealishees-GANicclleel)
0.966
0.921
24.86
21.36
0,0392
3,4937
CUD-NNTT(OO bortl-ncoo festure)
0.897
0.866
27.77
21.01
0.0901
2,0826
CUD-NET
0.962
0.924
29.54
21.19
0.0312
1.4760
Table Evaluation table of comparison experiment. The CUD-NET with low bottle-neck feature achieves better results in the experiment of the deuteranopia and the protanopia subjects(Figure 5), although the evaluation metrics lower than that of CUD-NET.
Cycle-GAN and Zero-DCE showed worse result than others. Cycle-GAN model had difficulty reconstructing a geometry of a particular object, and overall color had low saturation and brightness, resulting in color conversion into almost grey scale image. Zero-DCE is faded in color, and the contrast was not much different from the input image. The overall image lost its color preservation, which we focused solve in this paper.
258
280
282
PPepLPF meets both the color preservation and contrast that we deal with for. However, DeepLPF tends color be over-stably filtered for the images with fewer color combinations. Although the color preservation has complied better than other experiments, there were many failed results from the perspective of contrast, which the over-stable filter leads to by DeepLPF.
264
266
Remarkably, predicted images of Brilighte-GANN showed reasonable results. However, simple color combinations or the images with already satisfying the CUD often showed results degenerated with low CUD suitability. Enlighten-GAN was able to generate the results we targeted, but its deviation of filter is so high that it sometimes failed to satisfy the contrast even simple images or decreased contrast. As the problem GAN-based method including Enlighten- GAN, moreover, model
267
268
270
fixes the width and height of the predicted image. If width and height of T' and I down-scaled

---

to size of Enlighten-GAr (appproxmmately 25K pixels in this experiment), the SSIM (i, and 272
showed higher estimation some metrics than CUD-NET. In the opposite case of î u scaled to size of T and I, the significantly low estimation recorded due to the information loss of up up-scaling problem.
PSNR R [, T
273
275
278
CUD-NET showed stable and robust predicted images in both color preservation and increase in the contrast compared to other experiments. In comparing the values the same region of / and the model scaled two L channel values with opposite side in the most of case, the one goes up and the other goes down. When we reduced the number of bottle-neck feature of model, it tends to record relatively high deviation of filter scales according to the number of combinations of colors. In summary, the CUD-NET showed the highest estimations for 4 evaluation metrics. Moreover,
277
278
280
282
model adopted a neural filter unlike generation models, there is no loss of information regarding the scaling of predicted images.
Deuteranopia Subjects
Protanopia Subjects
GOY
DoUR ooer
Roppect sibe
Figure 5: The box bar is ordered to the left side, input image 1, Enlighten-GAN, DeepLPF, CUD- NET. The position box bar represents mean and length of the box bar represents a deviation of each experiment. The lower the graph is, the higher the rank is.
284
The figure 5 shows the evaluation of the deuteranopia and the protanopia. The evaluation metrics consist of object distinguishability and color harmony in order of input image I, predicted image of Enlighten GAN, DeepLPF, and CUD-NET. User study has tested upon the total of 6 subjects, deuteranomaly and 2 protanomaly. The subjects were asked to list the ranks of object distinguishability and color harmony of 4-paired-image for each model-blinded item. As the experimental results, the deuteranopia subject ranked the 1-st in the object distinguishability of CUD-NET an average
286
287
288
289
290
rank of 1,821, followed by Enlighten-GAN at an average rank of 2.512. Similarly, the protanopia subject also ranked the 1-st in CUD, followed by Enlighten-GAN GAN. DeepLPF, and input images. The
291
evaluation of color harmony showed that the subjects tend to assume that the image with a good
293
object distinguishability has good color harmony preferentially. For total of six subjects, the five subjects chose the CUD-NET, with the exception of one who ranked Enlighten-GAN GAN by a subtle gap
294
295
4
Conclusion
In this paper, we proposed deep network to generate CUD images from non-CUD input images. The pre-pros pre-processing and multi-modal fusion layer could comprehend the information for color weakness, and the variational loss function makes the model further adapt to CUD dataset. Compared to other research, we are able to maintain high-resolution images and both stable color preservation and contrast with neural filter per images.
297
299
301
Our current research shows a robust filter for single color, such as vectorized images, but it is difficult to expect stable results in the case of a real-world image with high gradation in hues. We consider the same limitation of our work when the certain pixel values react sensitively, making noise
302
303
appear more prominent in the predicted image. In the future, we plan to create additional datasets with gradation on the vectorized image and focus on the fusion layer improve performance of the model.
305

---

References
307
[AHB* 18]
ANDERSON P.. HE X., BUEHLER C., TENEY D., JOHNSON M., GOULD S., ZHANG L.: Bottom- up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018).
309
310
311
[BCPS19]
BIANCO S., CUSANO C., PICCOLI F., SCHETTINI R.: Contenng tone adjustment for image enhancement. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR) Workshops (June 2019).
312
313
314
[LIT18]
DENG Y., LOY C. C., TANG X.: Acsthetic-driven image enhancement by adversarial learning. In Proceedings of the 26th ACM International Conference on Multimedia (New York, NY, USA. 2018), MM 18, Association for Computing Machinery, p. 870-878. UtL:lhttps://do..coo org/ 10. 10.1145/3240508. 3240531
315
317
doi:10.. 1145/3240508. 3240531
[FPY 16]
FUKUI A., PARK D. H., YANG D., ROHRBACH A., DARRELL T., ROHRBACH M.: Multimodal compact bilinear pooling for visual question answering and visual grounding, 2016. arXiv
319
320
1606.01888
321
[LLG* 20]
Guo C., C., GUO J., LoY C. C., Hou J., KWONG S., CONG R.: Zero-reference deep curve estimation for low-light image enhancement, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020).
322
323
324
[HMK019]
HIRA S., MATSUMOTO A., KIHARA K., OHTSUKA S.: Hue rotation (hr) and hue blending (hb); Real-time image enhancement methods for digital component video signals to support red-green color-defcctive ohservess. Journal of the Society for Information Display 27, 7 (2019), 409-426.
325
327
URL: [toL::atttps:/ooonnneiirrrr wiley com doi htsii//bs/10... 1002/ jsid. 758 arXiv:https
328
//onlinelibrary www.:.wwww........... com cttp://cccmooo//pccc/c pdf 1100 1000/ doi doi:https: http://oi.oooog/////
329
1002/jsid.74 758
330
[1ZZE17]
ISOLA P., ZHU Y.,Y., ZHOU T., EFROS A. A.: Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR) (July 2017).
[JGL"2I
JIANG Y., GONG X., LIU D., CHENG Y., FANG C., SHEN X., YANG J., ZHOU P., WANG Z. Enlightengan: Deep light enhancement without paired supervision. JEEE Transactions on Image
335
Processing 30 (2021), 2340-2349, doi: 1109/TIP 2021 3051462
338
[JSZK66]
JADERBERG M., SIMONYAN K., ZISSERMAN A., KAVUKCUOGLU K.; Spatial transformer
337
networks, 2016. arriiv 1506. 02025
338
[KWK21]
KUMAR M.. WEISSENBORN D., KALCHBRENNER N.: Colorization transformer. International
339
Conference on Learning Representations (2021). URL: URL: https:// www.:neeeoo/mue 5HA1PimICFu
340
341
[7ZG*
KRISHNA R., ZHU Y.. GROTH O., JOHNSON J., HATA K., KRAVITZ J.. CHEN S., KALANTIDIS
342
Y., LI L.-J., SHAMMA D. A., BERNSTEIN M. S., FEI-FEI L.: Visual genome: Connecting
language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision 123, (May 32-73. htt.htttps///ooorrrttt
344
11000001011000 016 0981-7
doi: 1007/
345
s1 s11263- 016-0-1---998177 0981-7
346
[LCY14]
LIN M., CHEN Q., YAN S.: Network in network, 2014, arXiv:1312.44
[LYL'22]]
X., YIN X., LI C., ZHANG P., HU X., ZHANG L., WANG L., HU H., DONG L.,, WEI F., CHOi Y., GAO J.: Oscar: Object-semantics aligned pre-training for vision-language tasks. In
348
349
Computer Vision - ECCV 2020 (Cham, 2020), Vedaldi A., Bischof H., Brox T., Frahm J.-M., (Eds.), Springer International Publishing, pp. 121-137.
350
351
[MMM 20] MORAN S., MARZA P., MCDONAGH S., PARISOT S., SLABAUGH G.: Deeplpf: Deep local parametric filters for image enhancement. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patiern Recognition (CVPR) (June 2020).
[MMS19]
MORAN S., McDONAGH S., SLABAUGH G.: Curl: Neural curve layers for global image enhancement, 2019, 1911v:
355
[PEZZ20]
PARK T., EFROS A. A., ZHANG R., ZHU J.-Y.: Contrastive learning for unpaired image-to tnage-to-image translation. In Computer Vision ECCV 2020 (Cham, 2020). Vedaldi A., Bischof H., Brox T., Frahm J.-M., (Eds.), Springer International Publishing, pp. 319-345.
357
358
359
[RFB15]
RONNEBERGER O., FISCHER P., BROx T.: U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer Cosputer-Assisted Intervention MICCAI 2015 (Cham, 5). Navab N., Homngger J., Wells W. M., Frangi A. F., (Eds.), Springer International Publishing, pp. 241.
360
381
10

---

363
[RG19]
RIBEIRO M., GOMES A. J. P.: Recoloring algorithms for colorblind people: A survey. ACM Compul. Surv. 52, 4 (Aug. 2019). http /ttp://...rrr11.1111/33311111111 doi: :01:10.10 1145/
365
3329118
[SAC 17]
SCHÜTT K. T., ARBABZADAH F., CHMIELA S., MULLER K. R., TKATCHENKO A.: Quantum- chemical insights from deep tensor neural networks. Nature Communications (2017).
367
[TPW16]
TAIGMAN Y., POLYAK A., WOLF L.::
Unsupervised cross-domain image generation, 2016.
arXiv: 1611 1611.02200
370
[TSC20]
TAO SAPRA K., CATANZARO B.: Hierarchical multi-scale attention for semantic segmentation.
2020. arXiv:2005 10821
372
[VZCR20]
VARIKUTI V. N., ZHANG C., CLAIR B., REYNOLDS A. L.: Effect of enchroma glasses on color vision sereening using ishihara and famsworth d-15 color vision tests. Journal of American Associ-
ation for Pediatric Ophthalmology and Strabismuus 24, 3 (2020), 157.e1-137.e5. URL: https
375
//www.acientte wwwencedireect.................c com samneeciince//eeeeee/eee http:////eeeeeaaaaaaa/////////// attpac/ce//.11111111111100000000
doi triihhtps://
376
doi. org/ 1016// jaapos. 2020. 03.006
377
[WEG87]
WOLD S.. ESBENSEN K., GELADI P.: Principal component analysis. Chemo- Proceedings of
378
metrics and Intelligent Laboratory Systems 2. 1 (1987), 37-52.
379
the Multivariate Statistical Workshop for Geologists and Geochemists.
URL: https:
7/www www.saccceeeeeccccooooooococccce http:////ccccccee/trrree///////a doi. doi.org/10. 1ttp:/00.1101/011---11 74396///199--799 (87)
doi boihtttp::/
382
/ [Wonl
WONG B.: Points of view: Color blindnesss, 2011. doi:https://doi. org..org111 101111110000 1038/nmeth
1618
384
[WPLK18]
Woo S.. PARK J., LEE J.. KWEON I. S.: Cbam: Convolutional block attention module. In Proceedings of the European Conference Computer Vision (ECCV) (September 2018).
386
[WSB03]
WANG 7.,, SIMONCELLI E. P., BOVIK A. C.: Multiscale structural similarity for image quality assessment. In The Teerety-Seventh Asilomar Conference Signals, Systems Computers, 2003
388
(2003), vol. 2, pp. 1398-1402 Vol.2. doi: 10. 1109/ACSSC, 2003. 1292216
389
[WZF* 19]
WANG R., ZHANG Q.. FU C.-W., SHEN X., ZHENG W.-s.. JIA J.: Underexposed photo enhancement using deep illumination estimation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019).
391
[ZBSS04]
ZHOU WANG, BOVIK A. C., SHEIKH H. R., SIMONCELLI E. P.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600-612. :o1::001109/T11I 2003 819861
393
394
[ZGLL22]]
ZOPH B., GHIASI G., LIN T.-Y., Cui Y., LIU H., CUBUK D., LE Q. V.: Rethinking pre- pre-trainig and self teafnnraining. 2020. arXiv: 06882
396
[ZPIE17]
ZHU J..Y., PARK T., ISOLA P., EFROS A. A.: Unpaired mmage-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) (Oct 2017).
398
399
20]
ZHANG H., WU C., ZHANG Z., ZHU Y., LIN H., ZHANG Z., SUN Y., HE T.. MUELLER MANMATHA R., M., SMOLA A.: Resnest: Split-attention networks, 2020. arXiv
401
402
2004. 08955
Checklist
404
1. For all authors...
405
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
407
(b) Did you deseribe the limitations of your work? [Yes] See on conclusion section
408
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This work is for positive societal impacts
410
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL.)? [Yes] See on github: https: bettss//gitthen httt:////ghhabbb..coooooooooooo Atg.abbboo//aoyyyooasaea5////// NET anonymous
414

---

418
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] on github repository
417
418
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See section 2.3
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See on Experiment section
422
3. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
425
(a) your work uses existing assets, did you cite the creators? [Yes] Cooperated with Co-author
(b) Did you mention the license of the assets? [Yes] on github repository
(c) Did you include any assets either in the supplemental material URL? [Yes] on github repository
427
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] on github repository
429
430
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
12


Reviews:
[{"review_id": "PsFcIpCE7n", "reviewer": "Reviewer_Zn84", "summary": "The paper proposes a method to generate CUD (color universal design) images with a deep learning based image enhancement method. The output CUD images are more discernible to those challenged by red/green color blindness (protanopia/deuteranopia). The network architecture proposed to generate CUD images consists of three steps -- i) feature layer wherein 512 dimensional features are extracted from concatenated input images, ii) fusion layer to combine the 3 features, iii) Finally an attention layer to generate an enhanced feature. Subsequently, 64 values are regressed from the enhanced feature to estimate a piecewise linear function for S/V channels of the output image. As stated in Sec 2.4, different loss functions are used -- stencil masking (only change areas that challenge deuteranopia vision), CIELab loss to enhance contrast, histogram loss to preserve original colors etc. Quantitative evaluation is performed in Tab. 1 which measures color preservation and contrast enhancement by comparing SSIM/PSNR metrics between predicted image (I^\\hat) and original image (I), target image (T). In Fig. 5, protanopia/deuteranopia rank different methods based on distinguishability of output CUD image.", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The paper addresses an important problem of using deep-learning based image enhancement techniques to generate CUD images which are distinguishable by those with red/green color blindness\n+ Test subjects in user study of Fig. 5 prefer the CUD images generated by the proposed CUD-NET method over other baselines and original input\n\nWeakness:\n- The details in the paper are difficult to follow and the paper writing can be improved. For example, \ni) L118-119 \"The training data is grouped about 1,600 color combinations into the same V and then simulates them with the deuteranopia vision, converting to the adjacent color family to comply color preservation.\" What is the process of simulating the images with deuteranopia vision? How does the conversion to adjacent color family work? The authors should explain these details with equations if necessary.\nii) L139-140 \"After applying color inversion from the original RGB input image In, we generate the image Id with\nan aspect of deuteranopia vision in equation 1.\" It is not clear how I^d is generated, can the authors explain? \niii) L240-242 \"Color experts has refined about 1,500 vectorized image for the training data and 300 publication images for the validation data. All the comparative experimental models used the same train, test, validation data in this paper.\" The authors should describe how they sourced the images, what annotation/enhancement process was used to generate supervision (i.e. enhanced/CUD images), and what are the train/val/test splits?\n\n- Sec 2.4 describes various loss functions, however it is unclear how each of them helps improve the performance. An ablation study should be performed by training the network with different loss terms and measuring the effect on quantitative/qualitative metrics in Sec 3.\n\n- Sec 2.3 describes the model architecture, however some design choices are not explained clearly\ni) For shared convolutional layers, why is a standard architecture (possibly with pre-training) e.g. residual network not used? \nii)  Does pooling and split attention mechanism help improve the performance? Can the authors demonstrate that using pooling + attention is better than simply concatenating the 3 512-dimensional features and regressing the 64 values of piecewise linear function?\n\nThe above weaknesses need to be address for the reader to get insights into the proposed method and its results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors propose a method to generate images which are distinguishable to those with red/green color blindness. They perform a user study with 6 subjects to demonstrate increased distinguishability of the images generated by the proposed method. Larger study needs to be performed to ascertain the performance and failure modes of the proposed method before it can be deployed to assist those with color blindness.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9irpMc9j0Ea", "reviewer": "Reviewer_r6iV", "summary": "The paper proposes a deep-learning based method to create (vector-graphic) images that enables color deficient viewers to identify every object in the scene. The method boosts the contrast and/or modifies the hue of colors that are confusing to protanopes and deuteranopes. The method has a pre-processing step, a multi-model feature fusion architecture, and a variational loss function. Comparisons were carried out with GANs and image enhancement methods.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the idea and project is certainly worthwhile pursuing as it benefits vision impaired people, this paper is not yet ready for publication due to several issues addressed below:\n\n-\tSOTA: There are a large number of algorithms out there that correct images for protanopes and deuteranopes, many of them listed in reference [RG19]. A discussion of some of them, conceptually closest to the method presented here, would improve the SOTA. But more importantly, a comparison to some of these methods in the Experimental Section would very much strengthen this paper.\n-\tExperiments: As already alluded above, I do not consider the comparison to GANs nor image enhancement methods sufficient to validate the proposed methodology. A comparison to at least some of the color-blind correction methods is needed. Also, PSNR and SSIM are not really relevant metrics here, as they do not measure perceptual color differences. Some Delta E variant would be more appropriate. I appreciated that a psychovisual study was done, but that one needs to be better described on what exactly the methodology was.\n-\tThe proposed method so far only works for vector graphics. There are of course some applications for that, but it is limited. I encourage the authors to also extend the method to natural images. I do acknowledge that creating an appropriate ground truth database will not be trivial, thus I find a comparison to existing color deficiency correction algorithms even more important.\n-\tLab Loss Function: I do not understand the motivation for the Lab loss function. Perceptual differences in Lab Color space are measured as an L2 distance which is equal to Delta E, an (almost) perceptual metric. Here L1 is used, as well as MS SSIM. Considering we are dealing with vector graphic images I am not sure SSIM is meaningful. I would like to see a better discussion of this Loss function.\n-\tEnglish: The paper is hard to read as there are many grammatical mistakes. A good proof-reading by a native English speaker will certainly improve the quality of the manuscript.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact is mentioned", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "1LnvfdnUWeh", "reviewer": "Reviewer_Sm8T", "summary": "This paper focuses on color universal design to satisfy both color preservation and distinguishment of color for the color weakness. To achieve that, the paper refines the color universal design dataset on specific criteria by color experts, expands the input image information via pre-processing, proposes a multi-modality feature fusion architecture in CNN, and suggests a deformable loss function. The experiments are conducted on the provided dataset. Besides, a user study is carried out. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+This topic is interesting and meaningful. Although there are numerous image enhancement methods, they are not suitable for color universal design. The paper raises this topic and provides some insights.\n\n+The provided dataset and pre-processing are valuable for this task.  \n\n+The paper provides sufficient experimental results in the supplementary material. \n\nCons: \n\n-Although a new research topic is raised in this paper, the novelty of the paper is limited to me. Besides the dataset, the multi-modal feature fusion, and loss function are intuitive and straightforward. What the results would be like if just using a simple network such as a Unet-like network? With the same training data, a simple network may achieve similar performance under the constraint of the l1 or l2 norm.  \n\n-The experiment is not much convincing to me. The proposed method is compared with several image enhancement methods. It is not clear if the compared methods are training using the same training data as the proposed method. Some compared methods are un-supervised or zero-reference-based. Thus, strong baselines should be included for comparison. For example, a supervised method is trained under the same training data as the proposed method. The subjects in the user study are only six. More subjects should be invited for a stable statistic. Moreover, the performance of the user study shows that the proposed method just achieves relatively good performance when compared with these baselines. However, the accuracy of the proposed method is far from the practical applications in real cases.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author has clearly presented and solved the limitation and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree there are fundamental deficiencies in this submission. One is the lack of baselines for the task, some of which pre-date deep learning methods. Their addition is key to improving the paper and the comparison of the results. 

All reviewers also mention the lack of clarity in the writing that makes it difficult to extract the details of the contribution. Unfortunately, I recommend rejection.
Decision: Reject