CUD-NET: Color Universal Design Neural Filter for the Color Weakness

Information on images should be visually understood to anyone , including the color1 weakness . However , it is not recognizable if color that seems distorted to the color2 weakness meets an adjacent object . We suggest CUD-NET1 based on convolutional3 deep neural network to generate color universal design ( CUD ) images that satisfy4 both color preservation and distinguishment of color for input images . CUD-NET5 regresses the node point of the piecewise linear function based on information of6 input images and comprises a specific filter per image . We present the following7 methods to generate CUD images for the color weakness . First , we refine the CUD8 dataset on specific criteria by color experts . Second , the input image information9 is expanded through the pre-processing specialized on the color weakness vision.10 Third , we suggest a multi-modal feature fusion architecture that combines features11 to process expanded images . Finally , we suggest a deformable loss function by the12 composition of the predicted image through the model to avoid the one-to-many13 problems of the dataset.14 1 Introduction15 1.1 Motivation16 The green and red color blindness are made up of 8 % of males and 0.5 % of females in Northern17 European descent [ Won11 ] , which is almost up to rate of one person in 20 people . Green and red18 blindness is the most common pattern , followed by blue , yellow , and total color blindness . In this19 paper , we generate Color Universal Design ( CUD ) images , which are color weakness friendly design20 forms , through deep learning around the aspect of the red color weakness ( protanopia ) and green21 color weakness ( deuteranopia ) vision . Protanopia is insensitive to red color and deuteranopia is22 insensitive to green color , although it varies depending on individual color weakness extent.23 There are studies that help color discrimination to the color weakness , including wearable devices24 and surgeries [ VZCR20 ] . However , since these research require time and cost , we simply generate25 CUD images with an image enhancement method based on deep learning to make the corresponding26 color visible for the color weakness . For an example of the left-above image I in Figure 1 , the people27 who are not color weakness can distinguish the letter ‘ 5 ’ in the image . But as a deuteranopia vision in28 left-below image Id , the surrounding color and the letter ‘ 5 ’ are very analogous , making it ambiguous29 to distinguish the bound of adjacent object . The right-bottom target image T d , refined image by color30 expert designers , shows that the letter ‘ 5 ’ appeared well at the deuteranopia vision . Here , we define31 the non-CUD objects as the letter ‘ 5 ’ and surroundings invisible to deuteranopia vision in the image32 I , and define the CUD objects as the letter ‘ 5 ’ and surroundings visible to deuteranopia vision in the33 image T . In other words , CUD object means that adjacent objects are distinguishable on both the34 1Code available at https : //github.com/Anonymous68864576/CUD-NET-anonymous Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . normal vision and the color weakness vision . The non-CUD object means that adjacent objects are35 distinguishable on normal vision but not the color weakness vision . Consequently , we generate Î that36 satisfies CUD with a specific filter to the image I .37 We want to apply as weak filter as possible to CUD objects to preserve color , which requires a38 certain level of object comprehension mechanism to do so . There are various studies from classic39 PCA [ WEG87 ] to machine learning-based object segmentation methods [ TSC20 , ZGL∗20 ] to define40 specific objects or areas in image . The research on semantic segmentation , which even provides labels41 between objects , seems that deep learning still does not have a complete comprehension of all objects42 in the real-world . The visual question answering to arbitrary questions about object ’ s interactions,43 the most general issue on comprehension of object , does not have high transmission power to be44 practical uses [ AHB∗18 , KZG∗17 , LYL∗20 ] . Therefore , we expand feature of the input image around45 the information of color weakness vision and define the robust neural filter . In summary , we suggest46 a CUD-NET that generates an image suitable for CUD , while complying with the color preservation47 for the source image.48 In this paper , we suggest the Color Universal Design Network ( CUD-NET ) to satisfy both color49 preservation and contrast of non-CUD objects ( CUD suitability ) . We introduce 4 core contributions50 of CUD-NET.51 • Dataset refinement criteria for CUD image We refine training data into two groups , the52 one with a simple color tone image based on H and V in the HSV color space , the other with53 two or more non-CUD objects that must be distinguished in publications.54 • Image pre-processing for CUD-NET We carry out pre-processing to expand the infor-55 mation of the input image . Input image I is reconstructed with three expanded feature56 information with noise removed.57 • Multi-modal feature fusion architecture We define a feature layer , the fusion layer , and58 a regression layer to handle pre-processed images . The three features from the feature59 extracting layer are combined into the one fusion feature , and finally a filter is constructed60 by regressing the node point of the piecewise linear function , or indicator of filter.61 • Variational loss function We suggest a deformable loss function by the composition of the62 predicted image through the model . Our data have a problem of one-to-many , where the63 specific color in input image I is mapped into multiple colors in target image T .64 1.2 Related Works65 Image-to-Image translation based on GAN GAN is used in various image translation areas,66 including image generation , style transfer , and colorization [ KWK21 , IZZE17 ] . In a preliminary67 experiment , Cycle-GAN [ PEZZ20 ] has reached the best performance in maximizing the contrast of68 non-CUD objects . However , our goal is to keep the color preservation of the input image as well,69 so in the case of black color , which has lost all its color of the input image , it is considered the70 worst case for color preservation . Enlighten-GAN [ JGL∗21 ] complements those instability , enabling71 them to generate more stable results on color preservation . But since most of the GAN-based image72 translation fixes the size of the predicted image , reshaping a high-resolution image causes information73 loss of source image . Also , it is difficult to reconstruct the complete geometry for the source image74 as it generates images through the dilated convolution layer.75 Image enhancement based on neural filter estimation Unlike GAN , there are researches that76 scale the pixel values of images based on neural filter estimation [ WZF∗19 , DLT18 , BCPS19 ] . Zero-77 DCE [ GLG∗20 ] is a low-light image enhancement research that provides a brighter visual display78 of input image . It estimates pixel-wise and high-order filter for dynamic range adjustment of input79 images with lightweight deep network , DCE-Net . DeepLPF [ MMM∗20 ] tried to solve the problem by80 using a graduated filter , elliptical filter , and polynomial filter . The authors not only tried to visually81 enhance the contrast of images but also to comprise stable filters that are easy to understand for82 the spectators while keeping the color preservation . In our problem , however , the contrast factor is83 almost same results as the input image in both visions , while complying the high color preservation,84 resulting over-stable filter . It is assumed that the inability in comprehension of object ’ s interaction85 leads to over-stable filter.86 2 Methodology87 We define the ideal predicted image as an increase in the contrast between non-CUD objects and the88 color preservation for the input image . The non-CUD object a should be mapped into á and CUD89 object b should preserve its color and contrast like an ideal example of Figure 2 . However , as our90 neural filter affects the whole pixels throughout the image , we have the constraint of applying the91 same filter to objects a and b . It is very hard to make the contrast and color of object b exactly the92 same as before the filter adjustment while maximizing the contrast of object a . Therefore , we propose93 a deep learning-based regression to comprise the specific filter per image that maximizes the contrast94 of object a while minimizing the adjustment of features on object b.95 First , we propose a solution to maximize the contrast of the L channel values in CIELab color96 space [ RG19 ] . We empirically confirmed that protanopia and deuteranopia , which account for the97 most proportion of color weakness , can distinguish the difference by L channel values in common98 when the non-CUD objects are adjacent to each other . To illustrate Figure 1 again , the L channel99 value of letter ‘ 5 ’ in image I is 61 and the surrounding color is 61 . The distinguishment between100 the two objects is easy to normal vision , however the image Id , the deuteranopia vision , is very101 ambiguous . On the contrary , the CUD target image T and T d have a difference of L channel value 75102 for the letter ‘ 5 ’ and 45 for the surroundings , making it easy to distinguish between the normal and103 the deuteranopia vision . Due to the characteristics of these data , we refine a data pair by defining a104 criterion that separates two invisible non-CUD objects by L channel values.105 Secondly , we propose a variational loss function and multi-modal feature fusion network for color106 preservation . It can be said that the increase in the contrast of L channel values between non-CUD107 objects is quantitatively superior , but not in the case of increasing the differences in color preservation108 of input images . When non-CUD objects exist , as a simple example , the most likely way to maximize109 contrast is to polarize the color of the object black and white . But it is the result of complete ignorance110 for color preservation , so just enabling to distinguish between non-CUD objects is not always a good111 answer . A strong filter must be applied to distinguish non-CUD object , but its impact should not be112 too extensive to leading the loss of information in CUD objects . In this paper , we solve this problem113 by taking an appropriate trade-off of color preservation and contrast of L channel value.114 2.1 Dataset refinement criteria for CUD image115 The training data is refined by two groups . The one is vectorized image with two colors divided116 by value V and hue degree H in HSV color space [ HMKO19 ] , and the other is image with two117 or more objects that must be distinguished while preserving the color of non-CUD objects . The118 training data is grouped about 1,600 color combinations into the same V and then simulates them119 with the deuteranopia vision , converting to the adjacent color family to comply color preservation.120 All conversions are scaled within only S and V in HSV color space to increase at least 15 difference121 in the L channel value of selected non-CUD objects . The colors are combined with the 10 essential H122 and tones , and the similar color simulated with the deuteranopia vision was converted . Consequently,123 the key part of refining training data is preservation of color , allowing the models to comply with the124 same approach on learning.125 2.2 Image pre-processing126 Our model regresses node points of piecewise linear function , which will be described in the model127 architecture section , and the final filter is a multiplication operation for the input image . Therefore,128 the multiplication operations of less than the number 1 tend to fade the color saturation . The image129 without color inversion converges the white color value to 1 , so if the multiplying value is in the [ 0,130 1 ] range , the white color is shifted to the black . By inverting the color of input image , it ignores the131 multiplication operations for white value with 0.132 We generate the map image Im based on original RGB input images calculating the difference value133 between the image with an aspect of normal vision and the image with an aspect of deuteranopia vision.134 Recent studies have been conducted to augment the information or expanded the models ’ perspective135 through transformer models [ JSZK16 , RFB15 ] . In our experiment , however , the transformer model136 tends to generate the predicted image ignoring the source color , which result in the polarized color to137 black and white like Cycle-GAN ’ s.138 Im = ∣∣ invert ( In ) − invert ( Id ) ∣∣ ( 1 ) I = δ ( catchannel ( In , Id , Im ) ) ( 2 ) After applying color inversion from the original RGB input image In , we generate the image Id with139 an aspect of deuteranopia vision in equation 1 . From these two generated images , we can get the140 absolute difference value to compose the map image Im . In equation 2 , the final input I concatenated141 with 9×H ×W dimensions passes through the model . The δ ( . ) clips output to a range of [ 0 , 1 ] .142 2.3 Model Architecture143 CUD-NET regresses the node points of piecewise linear filter function from the input . The value144 of each node points computes the multiplication operation and generates the predicted image . The145 input I is compressed into 3 feature blocks matching each input through convolution layer , pooling146 layer , and global pooling layer . The input with 9 channels is separated into 3× 3 channels before147 passing the model . The first 3 channels are literally used as the main inputs , where the multiplication148 operation takes place , while the remaining 6 channels are used as features.149 First of all , we use multi-modal fusion architecture for three separate inputs to extract expanded150 features . The three inputs converted to the HSV color space pass through a weights-sharing convolu-151 tion layer to extract a feature block corresponding to the inputs . Each convolution layer consists of152 kernel size=3 , stride=1 , and padding=1 , reducing dimension through average pooling . We empirically153 noticed that the most values of output feature have distribution within the range of [ -1 , 1 ] with valid154 values for constructing the node points , so we use hyperbolic tan for activation function . Since we155 use inputs with unstructured image size , the last global pooling block holds the size of the feature156 instead of the average pooling block [ LCY14 ] .157 The three feature blocks are combined through the multi-modal compact bilinear pooling gate158 ( MCB ) [ FPY∗16 ] , following the fusion process shown in Figure 3 . The MCB gate allows both159 features to interact in a multiplicative way with low memory consumption and computation160 times . The fusion features are complemented to enhanced feature through the split attention161 mechanism [ ZWZ∗20 ] . At the beginning of the experiment , we have applied the convolutional162 block attention mechanism [ WPLK18 ] of each MCB gate , but we found that it does not make sense163 of understanding the feature itself , so we apply only one attention block to the last fusion feature.164 The enhanced feature pass through the fully-connected regression layer . We picked the 64 points165 to be regressed to compose the piecewise linear function , which is empirically confirmed to the166 optimized number of points in this research . The first half of the values construct the node points of167 the S channel and the other half comprise the V channel in HSV color space . Finally , node points168 become the scaling factors to generate predicted image in equation 3 [ MMS19 ] .169 S ( Is , vi ) = k0 + M−1∑ m=0 ( km+1 − km ) δ ( MIs , vi −m ) ( 3 ) The total number of node point M , each pixel values of S , V channel in input image Is , vi are170 multiplicated with the slope of actual regressed value km , the m − th generated node point . The171 specific node points M is scaled through a multiplication operation to pixel value of the input image172 according to each node point.173 2.4 Loss function174 Our dataset has one-to-many problems between input and target data . In dataset pair175 ( I1 , T1 ) , ( I2 , T2 ) , . . . , ( In , Tn ) , for example , the red color in I1 can be targeted to purple color in176 T1 , and the red color in I2 can be targeted to orange color in T2 . With these one-to-many dataset177 structures , we design the loss function L that expresses the potential and the diversity of predicted178 image in equation 4.179 L = N∑ i=1 Labloss ( V ( Φ ( Îi ) ) ) + Hloss ( Φ ( Îi ) ) ( 4 ) Stencil Masking As explained in the dataset refining criteria , we do not proceed with color180 conversion for all areas in the target images , but only for areas with color combinations that are181 invisible to the deuteranopia ( non-CUD object ) . For this reason , the input image has color regions of182 converting color and unconverting color , which also can be referred to as non-CUD object and CUD183 object . To imply the color bound to model , the stencil masking method is introduced.184 Φ ( Îi ) = Îij || ( Iij · T ij ) ( 5 ) We consist a stencil maps through the logical and operations ’ · ’ of each pixel value Iij , Tij . Stencil185 map can specify the non-CUD area and be computed with predicted image Îij of logical or operation186 ’ || ’ in equation 5 . Consequently , CUD object of the image adjusted with a stencil mask does not carry187 out the neural filter computation , such as the same way we refine the target image . This refined image188 is calculated on the loss function.189 CIELab Loss We use the CIELab channel loss function to maximize the contrast of color on190 deuteranopia vision . To stabilize the contrast and brightness of the predicted image , we calculate the191 MS· SSIM ( multi-scale structural similarity [ WSB03 ] ) of L channel.192 Labloss = ∥∥∥Lab ( Îrgbi ) − Lab ( T rgbi ) ∥∥∥ 1 +MS·SSIM ( Lab ( ÎLi ) , Lab ( TLi ) ) ( 6 ) The Lab ( . ) expression in equation 6 returns the CIELab channel corresponding to the RGB channel,193 and all calculations are made only on the L channel.194 Histogram Loss We use the histogram loss function to comply with the color preservation of the195 image . The RGB channel is used to preserve its color , contrary to using only the L channel in other196 loss functions . Handling the RGB channel as a loss function rather than using Lab ’ s ab channels has197 shown better results on color preservation.198 Hloss = −ωhist ∫ N ( Îrgbi ; σ ) − N ( T rgbi ; σ ) ( 7 ) When simply designing a loss function with the L1 distance of the RGB channel pixel values , it was199 very sensitive to certain values and the gradients are diverged , resulting in an untrainable experiment.200 Therefore , we used a gaussian expansion method [ SAC∗17 ] denoted by N ( . ) to infer a differentiable201 histogram loss function in equation 7 . We compute the difference of the RGB channel of the202 differentiable histogram function , which can be altered to mean squared error or cosine similarity.203 The scaler ωhist is determined in inverse proportion to the size of the input image . By maintaining204 the RGB similarity between the predicted image and the target image , we can comply with the color205 preservation.206 Variational Prediction There are various ways to maximize difference of the L channel in the207 image . And the target image is converted at least two colors compared to the input image . However,208 the predicted image of the model is generated by the neural filter , so it is unpredictable which area209 of color is modified . Therefore , if the color in predicted image is over-shifted or in the color value210 of opposite shifts to the target , the loss will rather increase . In addition to one-to-many problem211 that the data pair itself does not matches one-to-one in a particular color , it is necessary to generate212 alternative predicted image with the same aspect of the data pair . We calculate the loss function with213 a variational prediction based on the predicted image for potential color shifts.214 The first potential is the case of excessive shifts . Assume that ILi = { 74 , 41 , 79 } , Î L i = { 97 , 10 , 70 } ,215 TLi = { 50 , 41 , 80 } in L channel value . The first and third components of each image is non-CUD216 objects , and second component is CUD object . Therefore , we refined data paired with a difference of217 15 on L channel . Here , we clip the excessive L channel value in ÎLi by equation 8 . Up to this point,218 no calculation is made as no value is exceeded in this example . The second potential is the case of219 opposite shifts . It can be said that a complete neural filter has been proceeded for value 97 , 10 , 70220 where L channel difference is 27 . However , if we actually calculate the mean square error between221 ÎLi and T L i , it will be an large value over 1k . Here we can generate alternative predicted image from222 equation 9 and 10.223 clip ( Îij ) = max ( Îij , Tij ) , Iij > Tij min ( Îij , Tij ) , Iij ≤ Tij ( 8 ) R1 = 2Iij − Îij , R2 = Îij ( 9 ) V ( Îij ) = argmin ( ‖clip ( R1,2 ) − Tij‖2 ) ( 10 ) As mentioned above , we define thresholds by the maximum and minimum value of each corresponding224 pixel position of ÎLi and T L i . By computing a difference of residual map and the input image , we225 induce the alternative two images R1 , R2 . As a result , ÎLi with a smaller L2 distance is selected to226 alternative predicted image in equation 10 , and it is finally computed with loss function compared to227 the TLi . The above equation establishes V ( Φ ( Îi ) ) = { 54 , 41 , 80 } and the mean square error to the228 target image is approximately 5 , which is agreeable loss value respect to ÎLi itself.229 Identity Loss [ ZPIE17 , TPW16 ] We use Lidentity ( Ti ) to apprehend the CUD object to the model.230 In the case of target image that already satisfy the CUD , the filter should be relatively weakly applied231 than input image . The input of identity loss is target image Tij instead of input image Iij , and the232 reference of the loss function is also target image Tij to maintain the value itself . In computing233 identity loss , we do not require variational prediction as we can not judge the potential region by234 equation 10.235 3 Experiments236 The experiment was performed with Tesla V100 SXM2 and Intel Xeon Gold 5120 and the computation237 speed was about 40 images per minutes . We refined a dataset with Adobe Photoshop to maximize238 contrast in the L channel by adjusting saturation and brightness for areas that require color conversion239 based on deuteranopia vision simulation . Color experts has refined about 1,500 vectorized image for240 the training data and 300 publication images for the validation data . All the comparative experimental241 models used the same train , test , validation data in this paper . We used the inference data in242 publications , which is almost composed of vectorized images , as colors often appear distorted in a243 gradation-rich image . The Figure 4 is arranged in descending order of the number of combinations in244 colors from the top image.245 Both structure similarity ( SSIM ) [ ZBSS04 ] and peak signal to noise ratio ( PSNR ) in Table 1 can246 indicate whether the image is suitable for CUD or not . As a notable aspect , the result has shown that247 comparative models with lower metrics are sensitive to high-gradation input images , which generated248 color-heterogeneous image . SSIM and PSNR itself can determine the increase in contrast compared249 to the target image but do not determine whether the color preservation complied . Therefore , we250 evaluated SSIM and PSNR with three references , inputs images I , predicted images Î , and target251 images T . The higher the estimation of the Î and I , the more color preservation factor worked.252 The higher the estimation of the Î and T , the more increase in contrast can be considered . We also253 define SSIM mean absolute error , PSNR mean absolute error to measure the extent of the conversion254 between the F : I → T and F : I → Î in equation 11 and 12 , respectively . The N is total number255 of inference data.256 SSIM ·MAE = 1 N N∑ i=1 ∣∣∣ SSIM ( Îi , T i ) − SSIM ( Ii , Ti ) ∣∣∣ ( 11 ) PSNR·MAE = 1 N N∑ i=1 ∣∣∣ PSNR ( Îi , T i ) − PSNR ( Ii , Ti ) ∣∣∣ ( 12 ) Cycle-GAN and Zero-DCE showed worse result than others . Cycle-GAN model had difficulty257 reconstructing a geometry of a particular object , and overall color had low saturation and brightness,258 resulting in color conversion into an almost grey scale image . Zero-DCE is faded in color , and the259 contrast was not much different from the input image . The overall image lost its color preservation,260 which we focused to solve in this paper.261 DeepLPF meets both the color preservation and contrast that we deal with for . However , DeepLPF262 tends to color be over-stably filtered for the images with fewer color combinations . Although the263 color preservation has complied better than other experiments , there were many failed results from264 the perspective of contrast , which the over-stable filter leads to by DeepLPF.265 Remarkably , predicted images of Enlighten-GAN showed reasonable results . However , simple color266 combinations or the images with already satisfying the CUD often showed results degenerated with267 low CUD suitability . Enlighten-GAN was able to generate the results we targeted , but its deviation of268 filter is so high that it sometimes failed to satisfy the contrast even on simple images or decreased269 the contrast . As the problem of GAN-based method including Enlighten-GAN , moreover , model270 fixes the width and height of the predicted image . If width and height of T and I down-scaled271 to size of Enlighten-GAN Î ( approximately 25K pixels in this experiment ) , the SSIM ( Î , I ) and272 PSNR ( Î , T ) showed higher estimation in some metrics than CUD-NET . In the opposite case of Î273 up-scaled to size of T and I , the significantly low estimation was recorded due to the information274 loss of up-scaling problem.275 CUD-NET showed stable and robust predicted images in both color preservation and increase in276 the contrast compared to other experiments . In comparing the values in the same region of I and277 Î , the model scaled two L channel values with opposite side in the most of case , the one goes up278 and the other goes down . When we reduced the number of bottle-neck feature of model , it tends to279 record relatively high deviation of filter scales according to the number of combinations of colors . In280 summary , the CUD-NET showed the highest estimations for 4 evaluation metrics . Moreover , as our281 model adopted a neural filter unlike generation models , there is no loss of information regarding the282 scaling of predicted images.283 The figure 5 shows the evaluation of the deuteranopia and the protanopia . The evaluation metrics284 consist of object distinguishability and color harmony in order of input image I , predicted image285 of Enlighten GAN , DeepLPF , and CUD-NET . User study has tested upon the total of 6 subjects , 4286 deuteranomaly and 2 protanomaly . The subjects were asked to list the ranks of object distinguishability287 and color harmony of 4-paired-image for each model-blinded item . As the experimental results,288 the deuteranopia subject ranked the 1-st in the object distinguishability of CUD-NET at an average289 rank of 1.821 , followed by Enlighten-GAN at an average rank of 2.512 . Similarly , the protanopia290 subject also ranked the 1-st in CUD , followed by Enlighten-GAN , DeepLPF , and input images . The291 evaluation of color harmony showed that the subjects tend to assume that the image with a good292 object distinguishability has good color harmony preferentially . For a total of six subjects , the five293 subjects chose the CUD-NET , with the exception of one who ranked Enlighten-GAN by a subtle gap294 4 Conclusion295 In this paper , we proposed deep network to generate CUD images from non-CUD input images . The296 pre-processing and multi-modal fusion layer could comprehend the information for color weakness,297 and the variational loss function makes the model further adapt to CUD dataset . Compared to other298 research , we are able to maintain high-resolution images and both stable color preservation and299 contrast with neural filter per images.300 Our current research shows a robust filter for a single color , such as vectorized images , but it is301 difficult to expect stable results in the case of a real-world image with high gradation in hues . We302 consider the same limitation of our work when the certain pixel values react sensitively , making noise303 appear more prominent in the predicted image . In the future , we plan to create additional datasets304 with gradation on the vectorized image and focus on the fusion layer to improve performance of the305 model.306 References307 [ AHB∗18 ] ANDERSON P. , HE X. , BUEHLER C. , TENEY D. , JOHNSON M. , GOULD S. , ZHANG L. : Bottom-308 up and top-down attention for image captioning and visual question answering . In Proceedings of309 the IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) ( June 2018 ) .310 [ BCPS19 ] BIANCO S. , CUSANO C. , PICCOLI F. , SCHETTINI R. : Content-preserving tone adjustment for311 image enhancement . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern312 Recognition ( CVPR ) Workshops ( June 2019 ) .313 [ DLT18 ] DENG Y. , LOY C. C. , TANG X. : Aesthetic-driven image enhancement by adversarial learning.314 In Proceedings of the 26th ACM International Conference on Multimedia ( New York , NY , USA,315 2018 ) , MM ’ 18 , Association for Computing Machinery , p. 870–878 . URL : https : //doi.org/316 10.1145/3240508.3240531 , doi:10.1145/3240508.3240531.317 [ FPY∗16 ] FUKUI A. , PARK D. H. , YANG D. , ROHRBACH A. , DARRELL T. , ROHRBACH M. : Multimodal318 compact bilinear pooling for visual question answering and visual grounding , 2016. arXiv:319 1606.01847.320 [ GLG∗20 ] GUO C. , LI C. , GUO J. , LOY C. C. , HOU J. , KWONG S. , CONG R. : Zero-reference deep curve321 estimation for low-light image enhancement . In Proceedings of the IEEE/CVF Conference on322 Computer Vision and Pattern Recognition ( CVPR ) ( June 2020 ) .323 [ HMKO19 ] HIRA S. , MATSUMOTO A. , KIHARA K. , OHTSUKA S. : Hue rotation ( hr ) and hue blending ( hb ) :324 Real-time image enhancement methods for digital component video signals to support red-green325 color-defective observers . Journal of the Society for Information Display 27 , 7 ( 2019 ) , 409–426.326 URL : https : //onlinelibrary.wiley.com/doi/abs/10.1002/jsid.758 , arXiv : https:327 //onlinelibrary.wiley.com/doi/pdf/10.1002/jsid.758 , doi : https : //doi.org/10.328 1002/jsid.758.329 [ IZZE17 ] ISOLA P. , ZHU J.-Y. , ZHOU T. , EFROS A . A. : Image-to-image translation with conditional330 adversarial networks . In Proceedings of the IEEE Conference on Computer Vision and Pattern331 Recognition ( CVPR ) ( July 2017 ) .332 [ JGL∗21 ] JIANG Y. , GONG X. , LIU D. , CHENG Y. , FANG C. , SHEN X. , YANG J. , ZHOU P. , WANG Z.:333 Enlightengan : Deep light enhancement without paired supervision . IEEE Transactions on Image334 Processing 30 ( 2021 ) , 2340–2349 . doi:10.1109/TIP.2021.3051462.335 [ JSZK16 ] JADERBERG M. , SIMONYAN K. , ZISSERMAN A. , KAVUKCUOGLU K. : Spatial transformer336 networks , 2016. arXiv:1506.02025.337 [ KWK21 ] KUMAR M. , WEISSENBORN D. , KALCHBRENNER N. : Colorization transformer . In International338 Conference on Learning Representations ( 2021 ) . URL : https : //openreview.net/forum ? id=339 5NA1PinlGFu.340 [ KZG∗17 ] KRISHNA R. , ZHU Y. , GROTH O. , JOHNSON J. , HATA K. , KRAVITZ J. , CHEN S. , KALANTIDIS341 Y. , LI L.-J. , SHAMMA D. A. , BERNSTEIN M. S. , FEI-FEI L. : Visual genome : Connecting342 language and vision using crowdsourced dense image annotations . Int . J. Comput . Vision 123 , 1343 ( May 2017 ) , 32–73 . URL : https : //doi.org/10.1007/s11263-016-0981-7 , doi:10.1007/344 s11263-016-0981-7.345 [ LCY14 ] LIN M. , CHEN Q. , YAN S. : Network in network , 2014. arXiv:1312.4400.346 [ LYL∗20 ] LI X. , YIN X. , LI C. , ZHANG P. , HU X. , ZHANG L. , WANG L. , HU H. , DONG L. , WEI F.,347 CHOI Y. , GAO J. : Oscar : Object-semantics aligned pre-training for vision-language tasks . In348 Computer Vision – ECCV 2020 ( Cham , 2020 ) , Vedaldi A. , Bischof H. , Brox T. , Frahm J.-M.,349 ( Eds . ) , Springer International Publishing , pp . 121–137.350 [ MMM∗20 ] MORAN S. , MARZA P. , MCDONAGH S. , PARISOT S. , SLABAUGH G. : Deeplpf : Deep local351 parametric filters for image enhancement . In Proceedings of the IEEE/CVF Conference on352 Computer Vision and Pattern Recognition ( CVPR ) ( June 2020 ) .353 [ MMS19 ] MORAN S. , MCDONAGH S. , SLABAUGH G. : Curl : Neural curve layers for global image354 enhancement , 2019. arXiv:1911.13175.355 [ PEZZ20 ] PARK T. , EFROS A . A. , ZHANG R. , ZHU J.-Y . : Contrastive learning for unpaired image-to-image356 translation . In Computer Vision – ECCV 2020 ( Cham , 2020 ) , Vedaldi A. , Bischof H. , Brox T.,357 Frahm J.-M. , ( Eds . ) , Springer International Publishing , pp . 319–345.358 [ RFB15 ] RONNEBERGER O. , FISCHER P. , BROX T. : U-net : Convolutional networks for biomedical image359 segmentation . In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015360 ( Cham , 2015 ) , Navab N. , Hornegger J. , Wells W. M. , Frangi A. F. , ( Eds . ) , Springer International361 Publishing , pp . 234–241.362 [ RG19 ] RIBEIRO M. , GOMES A. J. P. : Recoloring algorithms for colorblind people : A survey . ACM363 Comput . Surv . 52 , 4 ( Aug. 2019 ) . URL : https : //doi.org/10.1145/3329118 , doi:10.1145/364 3329118.365 [ SAC∗17 ] SCHÜTT K. T. , ARBABZADAH F. , CHMIELA S. , MÜLLER K. R. , TKATCHENKO A. : Quantum-366 chemical insights from deep tensor neural networks . Nature Communications 8 , 1 ( 2017 ) .367 [ TPW16 ] TAIGMAN Y. , POLYAK A. , WOLF L. : Unsupervised cross-domain image generation , 2016.368 arXiv:1611.02200.369 [ TSC20 ] TAO A. , SAPRA K. , CATANZARO B. : Hierarchical multi-scale attention for semantic segmentation,370 2020. arXiv:2005.10821.371 [ VZCR20 ] VARIKUTI V. N. , ZHANG C. , CLAIR B. , REYNOLDS A. L. : Effect of enchroma glasses on color372 vision screening using ishihara and farnsworth d-15 color vision tests . Journal of American Associ-373 ation for Pediatric Ophthalmology and Strabismus 24 , 3 ( 2020 ) , 157.e1–157.e5 . URL : https:374 //www.sciencedirect.com/science/article/pii/S1091853120301002 , doi : https : //375 doi.org/10.1016/j.jaapos.2020.03.006.376 [ WEG87 ] WOLD S. , ESBENSEN K. , GELADI P. : Principal component analysis . Chemo-377 metrics and Intelligent Laboratory Systems 2 , 1 ( 1987 ) , 37–52 . Proceedings of378 the Multivariate Statistical Workshop for Geologists and Geochemists . URL : https:379 //www.sciencedirect.com/science/article/pii/0169743987800849 , doi : https : //380 doi.org/10.1016/0169-7439 ( 87 ) 80084-9.381 [ Won11 ] WONG B. : Points of view : Color blindnesss , 2011. doi : https : //doi.org/10.1038/nmeth.382 1618.383 [ WPLK18 ] WOO S. , PARK J. , LEE J.-Y. , KWEON I. S. : Cbam : Convolutional block attention module . In384 Proceedings of the European Conference on Computer Vision ( ECCV ) ( September 2018 ) .385 [ WSB03 ] WANG Z. , SIMONCELLI E. P. , BOVIK A. C. : Multiscale structural similarity for image quality386 assessment . In The Thrity-Seventh Asilomar Conference on Signals , Systems Computers , 2003387 ( 2003 ) , vol . 2 , pp . 1398–1402 Vol.2 . doi:10.1109/ACSSC.2003.1292216.388 [ WZF∗19 ] WANG R. , ZHANG Q. , FU C.-W. , SHEN X. , ZHENG W.-S. , JIA J. : Underexposed photo389 enhancement using deep illumination estimation . In Proceedings of the IEEE/CVF Conference on390 Computer Vision and Pattern Recognition ( CVPR ) ( June 2019 ) .391 [ ZBSS04 ] ZHOU WANG , BOVIK A. C. , SHEIKH H. R. , SIMONCELLI E. P. : Image quality assessment:392 from error visibility to structural similarity . IEEE Transactions on Image Processing 13 , 4 ( 2004 ) ,393 600–612 . doi:10.1109/TIP.2003.819861.394 [ ZGL∗20 ] ZOPH B. , GHIASI G. , LIN T.-Y. , CUI Y. , LIU H. , CUBUK E. D. , LE Q. V. : Rethinking395 pre-training and self-training , 2020. arXiv:2006.06882.396 [ ZPIE17 ] ZHU J.-Y. , PARK T. , ISOLA P. , EFROS A . A. : Unpaired image-to-image translation using397 cycle-consistent adversarial networks . In Proceedings of the IEEE International Conference on398 Computer Vision ( ICCV ) ( Oct 2017 ) .399 [ ZWZ∗20 ] ZHANG H. , WU C. , ZHANG Z. , ZHU Y. , LIN H. , ZHANG Z. , SUN Y. , HE T. , MUELLER400 J. , MANMATHA R. , LI M. , SMOLA A. : Resnest : Split-attention networks , 2020. arXiv:401 2004.08955.402 Checklist403 1 . For all authors ... 404 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s405 contributions and scope ? [ Yes ] 406 ( b ) Did you describe the limitations of your work ? [ Yes ] See on conclusion section407 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] This work408 is for positive societal impacts409 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to410 them ? [ Yes ] 411 2 . If you ran experiments ... 412 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-413 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] See on github:414 https : //github.com/Anonymous68864576/CUD-NET-anonymous415 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they416 were chosen ) ? [ Yes ] on github repository417 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-418 ments multiple times ) ? [ Yes ] See on section 2.3419 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type420 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] See on Experiment section421 3 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 422 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] Cooperated with423 Co-author424 ( b ) Did you mention the license of the assets ? [ Yes ] on github repository425 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 426 on github repository427 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re428 using/curating ? [ Yes ] on github repository429 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable430 information or offensive content ? [ N/A ] 431

Summary: The paper proposes a method to generate CUD (color universal design) images with a deep learning based image enhancement method. The output CUD images are more discernible to those challenged by red/green color blindness (protanopia/deuteranopia). The network architecture proposed to generate CUD images consists of three steps -- i) feature layer wherein 512 dimensional features are extracted from concatenated input images, ii) fusion layer to combine the 3 features, iii) Finally an attention layer to generate an enhanced feature. Subsequently, 64 values are regressed from the enhanced feature to estimate a piecewise linear function for S/V channels of the output image. As stated in Sec 2.4, different loss functions are used -- stencil masking (only change areas that challenge deuteranopia vision), CIELab loss to enhance contrast, histogram loss to preserve original colors etc. Quantitative evaluation is performed in Tab. 1 which measures color preservation and contrast enhancement by comparing SSIM/PSNR metrics between predicted image (I^\hat) and original image (I), target image (T). In Fig. 5, protanopia/deuteranopia rank different methods based on distinguishability of output CUD image.