<figure><img src="image_1.png" /></figure>

HAL
open science

# Using Neural Networks for Fake Colorized Image Detection

Yuze Li, Yaping Zhang, Liangfu Lu, Yongheng Jia, Jingcheng Liu

► To cite this version:

Yuze Li, Yaping Zhang, Liangfu Lu, Yongheng Jia, Jingcheng Liu. Using Neural Networks for Fake Colorized Image Detection. 15th IFIP International Conference on Digital Forensics (DigitalForensics), Jan 2019, Orlando, FL, United States. pp.201-215, 10.1007/978-3-030-28752-8_11. hal-02534604

HAL Id: hal-02534604

https://inria.hal.science/hal-02534604

Submitted on 7 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Chapter 11

USING NEURAL NETWORKS FOR
FAKE COLORIZED IMAGE DETECTION

Yuze Li, Yaping Zhang, Liangfu Lu, Yongheng Jia and Jingcheng Liu

**Abstract** Modern colorization techniques can create artificially-colorized images that are indistinguishable from natural color images. As a result, the detection of fake colorized images is attracting the interest of the digital forensics research community. This chapter tackles the challenge by introducing a detection approach that leverages neural networks. It analyzes the statistical differences between fake colorized images and their corresponding natural images, and shows that significant differences exist. A simple, but effective, feature extraction technique is proposed that utilizes cosine similarity to measure the overall similarity of normalized histogram distributions of various channels for natural and fake images. A special neural network with a simple structure but good performance is trained to detect fake colorized images. Experiments with datasets containing fake colorized images generated by three state-of-the-art colorization techniques demonstrate the performance and robustness of the proposed approach.

**Keywords:** Image forensics, fake colorized image detection, neural networks

1. Introduction

Digital image forensics is the process of collecting, identifying, analyzing and presenting evidence derived from digital image resources [2, 6]. Rapid advancements in image tampering techniques have made it increasingly difficult to distinguish between natural and fake images. Farid [6] divides image tampering techniques into six categories: (i) compositing; (ii) morphing; (iii) re-touching; (iv) enhancing; (v) computer-generating; and (vi) painting. While these categories cover most image tampering techniques, other more specific image tampering techniques such as colorization [13] and splicing [1, 5] have been proposed.
<figure><img src="image_3.png" /><figcaption>Figure 1. ColorDet-NN approach.</figcaption></figure>

Colorization is the process of transforming grayscale images to col-
orized images by adding color features. Colorization techniques are fre-
quently used to add color to greyscale photographs or black-and-white
films to restore historical scenes. These techniques are also used to col-
orize black-and-white CT, X-ray and MRI images to enhance medical
diagnosis and treatment. However, colorization can also be used for ma-
licious purposes, for example, to create doctored photographs and videos
that appear legitimate to the naked eye.

Guo et al. [8] were among the first researchers to focus on fake col-
orized image detection. They proposed two classification techniques,
FCID-HIST and FCID-FE, that rely on support vector machines [3].
Difficulties in choosing appropriate kernel functions for the support vec-
tor machines limit the performance of the techniques. Additionally, the
computing resources required by support vector machines render them
infeasible for large datasets.

To address these challenges, this research employs the ColorDet-NN
neural network [15] to detect fake colorized images. Figure 1 shows the
ColorDet-NN approach. An initial feature analysis step compares the
statistical differences in the color distributions of natural images and fake
colorized images. A feature extraction step then captures contributing
features from raw image data for pre-processing. The final training step
utilizes the extracted features to create a ColorDet-NN neural network
that detects fake colorized images.

## 2. Background

Colorization is the process of transforming grayscale images to col-
orized images by adding color features. Several colorization techniques
have been proposed over the past two decades. The colorization tech-
niques differ in how they obtain and handle the data used to model
the correspondence between grayscale and colorized images. As a re-
sult, colorization techniques are broadly divided into three categories:
(i) scribble-based; (ii) transfer-based; and (iii) fully automated.

Scribble-based methods require users to specify the colors in grayscale images in advance based on their experience. The first scribble-based method, developed by Levin et al. [16], utilizes a quadratic cost function of the differences between a pixel and its neighboring pixels under the assumption that adjacent pixels with similar intensities should have similar colors. Several researchers have developed more effective techniques. For example, Luan et al. [17] have developed an interactive system for colorizing natural images that uses texture similarity to obtain effective color propagation. Sykora et al. [20] have created a flexible, interactive tool for painting hand-drawn cartoons. However, these techniques rely on – and are therefore limited by – the user’s experience, and require a large number of experiments to achieve good performance.

Transfer-based colorizing techniques establish mappings between refer-ence colorized images and grayscale images, following which they trans-fer colors to the target greyscale images from analogous regions of the reference colorized images. Reinhard et al. [18] have done pioneering research on transferring colors between images. Ironi et al. [12] have presented a novel color transfer technique that analyzes the low-level feature space using a robust supervised classification scheme. However, in transfer-based colorization, the choice of appropriate reference col-orzed images is crucial to obtaining good performance.

Several researchers have applied deep learning techniques [14] to col-
oration. These fully-automated techniques have better performance
than scribble-based and transfer-based methods. Larsson et al. [13] have
developed a fully-automated image colorization technique that predicts
per-pixel color histograms utilizing low-level and semantic representa-
tions. Iizuka et al. [11] have employed a neural network that combines
global priors and local image features to automatically colorize grayscale
images. Zhang et al. [22] have proposed a fully-automated technique
that increases the diversity of colors in images by posing colorization as
a classification problem.

Guo et al. [8] were among the first researchers to leverage machine
learning to detect fake colorized images. They proposed two classifica-
tion methods, FCID-HIST and FCID-FE, that compute the statistical
differences in the hue, saturation, dark and bright channels in different
ways; they then employ support vector machines to distinguish between
natural and fake colorized images.
<table><caption>Table 1. Maximum absolute differences for natural and fake image distributions.</caption><thead><tr><th>Colorization Technique</th><th>Red</th><th>Green</th><th>Blue</th><th>Hue</th><th>Saturation</th><th>Value</th></tr></thead><tbody><tr><td>Larsson et al. [13]</td><td>70%</td><td>42%</td><td>188%</td><td>264%</td><td>4,393%</td><td>174%</td></tr><tr><td>Iizuka et al. [11]</td><td>32%</td><td>211%</td><td>92%</td><td>1,093%</td><td>1,654%</td><td>52%</td></tr><tr><td>Zhang et al. [22]</td><td>105%</td><td>154%</td><td>70%</td><td>774%</td><td>5,083%</td><td>49%</td></tr></tbody></table>

## 3. Detection Methodology

Research in deep learning has significantly enhanced colorization techniques. It has become very difficult for humans to distinguish fake colorized images from natural images. The proposed ColorDet-NN approach for detecting fake colorized images effectively analyzes the statistical differences between natural images and fake colorized images generated by three state-of-the-art techniques developed by: (i) Larsson et al. [13]; (ii) Iizuka et al. [11]; and (iii) Zhang et al. [22]. The ColorDet-NN neural network is then trained to detect fake colorized images.

### 3.1 Statistical Analysis and Testing

Statistical differences exist in the color distributions of natural images and fake colorized images. The RGB color space is defined by three chromaticities of the red, green and blue primary color channels (range is from 0 to 255), which can produce any chromaticity in the triangle defined by the primary colors. The HSV color space is an alternative representation of the RGB color space, which has hue, saturation and value channels. The RGB color space has more redundant information, which leads to insufficient feature differentiation. Therefore, the HSV color space is employed to obtain more features.

Normalized histograms were computed for the red, green, blue, hue, saturation and value channels in 10,000 natural images from the ImageNet LSVRC 2012 Validation Set [19]. The corresponding fake colorized images were generated using the colorization techniques of Larsson et al. [13], Iizuka et al. [11] and Zhang et al. [22].

The absolute differences between the distribution values of natural images and those of fake colorized images divided by the distribution values of the natural images were computed for the red, green, blue, hue, saturation and value channels. Table 1 shows the maximum values of the percentages obtained for the six channels. Clearly, a statistical difference exists in each channel between the natural and fake colorized images generated by each of the three colorization techniques.
<table><caption>Table 2. Two-sample Kolmogorov-Smirnov test results for the RGB channels.</caption><thead><tr><th>Colorization Technique</th><th>Red</th><th>Green</th><th>Blue</th></tr></thead><tbody><tr><td>Larsson et al. [13]</td><td>1</td><td>1</td><td>0</td></tr><tr><td>Iizuka et al. [11]</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Zhang et al. [22]</td><td>0</td><td>0</td><td>1</td></tr></tbody></table>

<table><caption>Table 3. Two-sample Kolmogorov-Smirnov test results for the HSV channels.</caption><thead><tr><th>Colorization Technique</th><th>Hue</th><th>Saturation</th><th>Value</th></tr></thead><tbody><tr><td>Larsson et al. [13]</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Iizuka et al. [11]</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Zhang et al. [22]</td><td>0</td><td>1</td><td>0</td></tr></tbody></table>

Note that significant differences exist in the saturation channel. In this channel, all the percentages are more than 1,600%, which means that significant color biases exist at some channel values between the natural and fake colorized images. In addition, the minimum percentage reached 32%, which suggests that there are statistical differences that can be utilized for detection.

The two-sample Kolmogorov-Smirnov test [7] is employed to determine whether the distributions of natural images and fake colorized images are different. The test checks whether the two data samples have the same distributions in order to measure their differences.

The null hypothesis $H_0$ is defined as:

$H_0$: The two data samples satisfy the same distribution.

Let $KSTest_m^c$ be the two-sample Kolmogorov-Smirnov test result between the distribution of natural images and the distribution of fake colorized images generated by a colorization method $m$ in channel $c$. Then, the null hypothesis is rejected at the 0.05 level of significance if $KSTest_m^c = 1$.

Tables 2 and 3 show that at least one channel will reject the null hypothesis for each colorization method in each color space. In the case of fake colorized images generated using the technique of Iizuka et al. [11], the red, green, blue, hue, saturation and value channels all reject the null hypothesis. On the other hand, for fake colorized images generated using the technique of Zhang et al. [22], only the blue and saturation channels
reject the null hypothesis, but this still means that the features of at least two channels can be used to distinguish between natural and fake colorized images. Simply put, there are statistical differences in the color distributions of natural and fake colorized images.

## 3.2 Feature Extraction

The statistical differences in the red, green, blue, hue, saturation and value channels are used for feature extraction.

Specifically, to distinguish between natural and fake colorized images the following six features are employed: (i) red channel feature $F_r$; (ii) green channel feature $F_g$; (iii) blue channel feature $F_b$; (iv) hue channel feature $F_h$; (v) saturation channel feature $F_s$; and (vi) value channel feature $F_v$.

Each channel feature is computed in a similar manner. For each feature $F_{ch}$, let $Hist_{n,ch}^{total}$ denote the normalized histogram distribution for all the natural images for channel $ch$. Let $Hist_{ch}^{\alpha}$ denote the $ch$ channel histogram distribution for an input image $\alpha$. The feature computation also leverages the first-order derivative of the normalized channel histogram distributions. These first-order derivatives are $Deri_{n,ch}^{total}$ for natural images. The $Deri_{ch}^{\alpha}$ for an input image $\alpha$ is computed as:

$$Deri_{ch}^{\alpha}(i) = Hist_{ch}^{\alpha}(i+1) - Hist_{ch}^{\alpha}(i), \quad i \in [0, 254] \quad (1)$$

where $Hist_{ch}^{\alpha}(i)$ and $Deri_{ch}^{\alpha}(i)$ are components of the vectors $Hist_{ch}^{\alpha}$ and $Deri_{ch}^{\alpha}$, respectively.

Since a natural image has a closer similarity to the natural image distributions than a fake colorized image, the cosine similarity $cos$ is used to measure the overall similarity between $Hist_{ch}^{\alpha}$ and $Hist_{n,ch}^{total}$ and $Deri_{ch}^{\alpha}$ and $Deri_{n,ch}^{total}$:

$$cos(\theta) = \frac{A \cdot B}{\|A\| \|B\|} = \frac{\sum_{i=0}^{n} A_i B_i}{\sqrt{\sum_{i=0}^{n} A_i^2} \sqrt{\sum_{i=0}^{n} B_i^2}} \quad (2)$$

where $A_i$ and $B_i$ are components of vectors $A$ and $B$, respectively.

The feature computations $F_r^{\alpha}$, $F_g^{\alpha}$, $F_b^{\alpha}$, $F_h^{\alpha}$, $F_s^{\alpha}$ and $F_v^{\alpha}$ for input image $\alpha$ are given by:
$$ 
\begin{aligned}
F_{ch}^{\alpha}(1) &= \frac{\text{Hist}_{n,ch}^{\text{total}} \cdot \text{Hist}_{ch}^{\alpha}}{\|\text{Hist}_{n,ch}^{\text{total}}\| \|\text{Hist}_{ch}^{\alpha}\|} \\
&= \frac{\sum_{i=0}^{255} \text{Hist}_{n,ch}^{\text{total}}(i) \times \text{Hist}_{ch}^{\alpha}(i)}{\sqrt{\sum_{i=0}^{255} \text{Hist}_{n,ch}^{\text{total}}(i)^2} \times \sqrt{\sum_{i=0}^{255} \text{Hist}_{ch}^{\alpha}(i)^2}}, \quad (3) \\
ch &= r, g, b, h, s, v
\end{aligned}
$$

$$ 
\begin{aligned}
F_{ch}^{\alpha}(2) &= \frac{\text{Deri}_{n,ch}^{\text{total}} \cdot \text{Deri}_{ch}^{\alpha}}{\|\text{Deri}_{n,ch}^{\text{total}}\| \|\text{Deri}_{ch}^{\alpha}\|} \\
&= \frac{\sum_{i=0}^{254} \text{Deri}_{n,ch}^{\text{total}}(i) \times \text{Deri}_{ch}^{\alpha}(i)}{\sqrt{\sum_{i=0}^{254} \text{Deri}_{n,ch}^{\text{total}}(i)^2} \times \sqrt{\sum_{i=0}^{254} \text{Deri}_{ch}^{\alpha}(i)^2}}, \\
ch &= r, g, b, h, s, v
\end{aligned}
$$

$$ F_{ch}^{\alpha} = [F_{ch}^{\alpha}(1), F_{ch}^{\alpha}(2)], \quad ch = r, g, b, h, s, v \quad (5) $$

After all the features are obtained, the feature vector $F_{HIST}^{\alpha}$ for an input image $\alpha$ is:

$$ F_{HIST}^{\alpha} = [F_{r}^{\alpha}, F_{g}^{\alpha}, F_{b}^{\alpha}, F_{h}^{\alpha}, F_{s}^{\alpha}, F_{v}^{\alpha}] \quad (6) $$

Let $L_{HIST}^{\alpha}$ denote the binary label of $F_{HIST}^{\alpha}$. $L_{HIST}^{\alpha}$ has a value of one if input image $\alpha$ is a fake colorized image and $L_{HIST}^{\alpha}$ has a value of zero if input image $\alpha$ is a natural image.

Thus, the final detection data $D_{HIST}^{\alpha}$ is:

$$ 
\begin{aligned}
D_{HIST}^{\alpha} &= [F_{HIST}^{\alpha}, L_{HIST}^{\alpha}] \\
&= \begin{cases} [F_{r}^{\alpha}, F_{g}^{\alpha}, F_{b}^{\alpha}, F_{h}^{\alpha}, F_{s}^{\alpha}, F_{v}^{\alpha}, 1], & \text{if image } \alpha \text{ is fake} \\ [F_{r}^{\alpha}, F_{g}^{\alpha}, F_{b}^{\alpha}, F_{h}^{\alpha}, F_{s}^{\alpha}, F_{v}^{\alpha}, 0], & \text{if image } \alpha \text{ is natural} \end{cases} \quad (7)
\end{aligned}
$$

## 3.3 Neural Network Construction

An artificial neural network is an algorithm that models computations using graphs of artificial neurons, mimicking how neurons work in the brain. Artificial neural networks are well-suited to solving complex non-linear problems. Unlike traditional machine learning algorithms such as support vector machines, artificial neural networks have flexible structures that can be adapted according to the problem that is to be solved. This work uses an artificial neural network to differentiate natural images from fake colorized images.
<figure><img src="image_4.png" /><figcaption>Figure 2. Neural network structure.</figcaption></figure>

The artificial neural network employed for detecting fake colorized images is based on the dense convolutional network (DenseNet) model [10]. DenseNet has a relatively simple structure, in which every layer of the network is connected to every other layer in a feed-forward manner. Compared with other neural network models, DenseNet strengthens feature propagation while reducing the number of parameters.

Figure 2 shows the structure of the neural network used for fake colored image detection. The neural network has six layers – an input layer, an output layer and four hidden layers. Each hidden layer is fully connected to the previous layers. For each hidden layer, the input of the layer is the sum of the outputs of the other hidden layers.

The relationships of the hidden layers are given by:

$$X_i = Y_1 + \cdots + Y_{i-1}, \quad i \ge 2 \qquad (8)$$

where $X_i$ and $Y_i$ are the input and output of layer $i$, respectively.

The selection of an appropriate activation function is an important aspect when designing a neural network. The proposed technique employs a parametric rectified linear unit (PReLU) [9], an activation function with parameters that can be trained. This activation function is used in the hidden layers of the network. Table 4 shows the details of the neural network. Hidden layers 1 through 3 have 32 neurons each whereas hidden layer 4 has 128 neurons.

The joint supervision of the softmax loss function and center loss function [21] was used to train the neural network. The softmax loss function is one of the most widely used loss functions. The center loss function has been demonstrated to minimize intra-class variations while keeping the features of different classes separable.

The softmax loss function $L_S$ is:

$$L_S = - \sum_{i=1}^{m} \log \frac{e^{W_{y_i}^T x_i + b_{y_i}}}{\sum_{j=1}^{n} e^{W_j^T x_i + b_j}} \qquad (9)$$

where $x_i$ is the $i^{th}$ deep feature, which belongs to the class $y_i$; $m$ is the mini-batch; and $n$ is the number of classes.
<table><caption>Table 4. Neural network details.</caption><thead><tr><th>Layer</th><th>Neurons</th><th>Activation</th><th>Connected Layer</th></tr></thead><tbody><tr><td>Input Layer</td><td>12</td><td></td><td></td></tr><tr><td>Hidden Layer 1</td><td>32</td><td>PReLU</td><td>Input Layer</td></tr><tr><td>Hidden Layer 2</td><td>32</td><td>PReLU</td><td>Hidden Layer 1</td></tr><tr><td>Hidden Layer 3</td><td>32</td><td>PReLU</td><td>Hidden Layer 1, 2</td></tr><tr><td>Hidden Layer 4</td><td>128</td><td>PReLU</td><td>Hidden Layer 1, 2, 3</td></tr><tr><td>Output Layer (Softmax loss)</td><td>2</td><td></td><td>Hidden Layer 4</td></tr><tr><td>Output Layer (Center loss)</td><td>1</td><td></td><td>Hidden Layer 4</td></tr></tbody></table>

The center loss function is:

$$L_C = \frac{1}{2} \sum_{i=1}^{m} \|x_i - c_{y_i}\|_2^2 \qquad (10)$$

where $c_{y_i}$ is the center of $y_i$ of the deep feature and is updated as the deep feature changes. The joint supervision of the softmax loss function and center loss function are used to train the neural network.

The final loss function is:

$$ \begin{aligned} L &= L_S + \lambda L_C \\ &= -\sum_{i=1}^{m} \log \frac{e^{W_{y_i}^T x_i + b_{y_i}}}{\sum_{j=1}^{n} e^{W_j^T x_i + b_j}} + \frac{\lambda}{2} \sum_{i=1}^{m} \|x_i - c_{y_i}\|_2^2 \end{aligned} \qquad (11) $$

# 4. Experiments and Results

This section describes the datasets used in the experiments, the experimental measurements and the performance evaluation results.

## 4.1 Datasets

Six benchmark datasets based on the ImageNet LSVRC 2012 Validation Set [19] were employed in the experiments. The datasets, which are widely used in image colorization and fake image detection research, contain many categories of images, including images of people, animals, buildings and landscapes.

The D1 dataset corresponds to the ctest10k dataset [13], which has 10,000 fake colorized images and their corresponding 10,000 natural images from the ImageNet LSVRC 2012 Validation Set. Datasets D2 and D3 each contain the 10,000 natural images in dataset D1 as well as 10,000 fake colorized images generated from the natural images using
the colorization techniques of Iizuka et al. [11] and Zhang et al. [22], respectively. Thus, datasets D1, D2 and D3 each have 20,000 images.

The D4 dataset contains 2,000 fake colorized images randomly selected from the ctest10k dataset [13] and their corresponding 2,000 natural images from the ImageNet LSVRC 2012 Validation Set, resulting in a total of 4,000 images. The D5 dataset also has 4,000 images – 2,000 natural images selected randomly from dataset D1 and their corresponding fake colorized images generated by the colorization technique of Iizuka et al. [11]. The D6 dataset also has 4,000 images – 2,000 natural images selected randomly from dataset D1 and their corresponding fake images generated by the colorization technique of Zhang et al. [22].

## 4.2 Measurements

The accuracy, precision, recall and F1 score were used to evaluate the performance of ColorDet-NN. In addition, the half total error rate (HTER) and area under the curve (AUC) measurements were used to compare the performance of ColorDet-NN against the performance of FCID-HIST and FCID-FE developed by Guo et al. [8].

## 4.3 Performance Evaluation

Several experiments were designed to evaluate the performance of ColorDet-NN. The experiments use all six datasets, D1 through D6.

The first set of experiments evaluated the ability of ColorDet-NN to detect fake colorized images. Datasets D1, D2 and D3 were used to assess the performance of ColorDet-NN at detecting fake colorized images generated using the colorization techniques of Larsson et al. [13], Iizuka et al. [11] and Zhang et al. [22]. Each dataset D1, D2 and D3 was randomly divided into a training set corresponding to 75% of the dataset and a testing set corresponding to 25% of the dataset.

Nine cross-validation experiments were conducted using the three training sets and three testing sets. The results in Table 5 demonstrate that ColorDet-NN can effectively distinguish between natural images and the fake colorized images generated by the three colorization techniques. All the accuracy values are greater than 88% when the training and testing sets come from the same original dataset. However, the accuracy values fall when the training and testing sets come from different datasets. Most of the experiments have accuracy values greater than 73%, except for the third experiment; this is likely due to large differences in the image features for fake images generated by the colorization techniques.
<table><caption>Table 5. Detection results in the cross-validation experiments.</caption><thead><tr><th>Training</th><th>Testing</th><th>Accuracy</th><th>Precision</th><th>Recall</th><th>F1 Score</th><th>HTER</th></tr></thead><tbody><tr><td rowspan="3">D1</td><td>D1</td><td>88.46%</td><td>85.46%</td><td>92.67%</td><td>88.92%</td><td>11.54%</td></tr><tr><td>D2</td><td>74.32%</td><td>74.24%</td><td>74.48%</td><td>74.36%</td><td>25.68%</td></tr><tr><td>D3</td><td>65.94%</td><td>69.17%</td><td>57.52%</td><td>62.81%</td><td>34.06%</td></tr><tr><td rowspan="3">D2</td><td>D1</td><td>77.82%</td><td>84.52%</td><td>68.12%</td><td>75.44%</td><td>22.18%</td></tr><tr><td>D2</td><td>88.20%</td><td>87.59%</td><td>88.99%</td><td>88.28%</td><td>11.80%</td></tr><tr><td>D3</td><td>80.82%</td><td>87.46%</td><td>71.96%</td><td>78.96%</td><td>19.18%</td></tr><tr><td rowspan="3">D3</td><td>D1</td><td>73.76%</td><td>78.29%</td><td>65.76%</td><td>71.48%</td><td>26.24%</td></tr><tr><td>D2</td><td>81.16%</td><td>80.22%</td><td>82.72%</td><td>81.45%</td><td>18.84%</td></tr><tr><td>D3</td><td>89.58%</td><td>86.28%</td><td>94.12%</td><td>90.02%</td><td>10.42%</td></tr></tbody></table>

<table><caption>Table 6. Area under curve results in the cross-validation experiments.</caption><thead><tr><th>Training</th><th colspan="3">Testing</th></tr><tr><th></th><th>D1</th><th>D2</th><th>D3</th></tr></thead><tbody><tr><th>D1</th><td>0.95972</td><td>0.82466</td><td>0.73605</td></tr><tr><th>D2</th><td>0.87773</td><td>0.95323</td><td>0.89848</td></tr><tr><th>D3</th><td>0.82149</td><td>0.89930</td><td>0.96504</td></tr></tbody></table>

Table 6 shows the area under the curve results in the cross-validation experiments. All the area under the curve results are greater than 95% when the training and testing sets come from the same dataset. The results imply that ColorDet-NN is effective at detecting the fake colorized images.

The next set of experiments were conducted to compare the detection performance of ColorDet-NN against state-of-the-art techniques for detecting fake colorized images. The FCID-HIST and FCID-FE fake colorized image detection techniques developed by Guo et al. [8] were used in the comparisons. Datasets D4, D5 and D6 were divided equally into training sets and testing sets in order to evaluate the performance of ColorDet-NN versus FCID-HIST and FCID-FE.

Nine experiments were performed using testing and training sets drawn from the same and different datasets. Table 7 compares the area under the curve results for ColorDet-NN, FCID-HIST and FCID-FE. ColorDet-NN has better performance than FCID-HIST and FCID-FE in most situations, especially when the training and testing sets are drawn from the same dataset (area under the curve values greater than 93%). A small decline in performance is seen when the training and testing sets
<table><caption>Table 7. Comparison of area under the curve results.</caption><thead><tr><th>Training</th><th>Testing</th><th>ColorDet-NN</th><th>FCID-HIST</th><th>FCID-FE</th></tr></thead><tbody><tr><td rowspan="3">D4</td><td>D4</td><td>0.93654</td><td>0.85687</td><td>0.85762</td></tr><tr><td>D5</td><td>0.81590</td><td>0.74112</td><td>0.77320</td></tr><tr><td>D6</td><td>0.76650</td><td>0.78525</td><td>0.84100</td></tr><tr><td rowspan="3">D5</td><td>D4</td><td>0.83701</td><td>0.63106</td><td>0.68758</td></tr><tr><td>D5</td><td>0.93613</td><td>0.85927</td><td>0.90382</td></tr><tr><td>D6</td><td>0.88755</td><td>0.56928</td><td>0.71286</td></tr><tr><td rowspan="3">D6</td><td>D4</td><td>0.82165</td><td>0.79804</td><td>0.82669</td></tr><tr><td>D5</td><td>0.88527</td><td>0.63953</td><td>0.72447</td></tr><tr><td>D6</td><td>0.94972</td><td>0.83215</td><td>0.84832</td></tr></tbody></table>

come from different datasets, but ColorDet-NN still outperforms FCID-HIST and FCID-FE, except in the third and seventh experiments. The negative results in these two cases arise because dataset D6 has more complex features than dataset D4 and the features extracted by FCID-HIST and FCID-FE are more sensitive than the features extracted by ColorDet-NN.

<table><caption>Table 8. Comparison of half total error rate results.</caption><thead><tr><th>Training</th><th>Testing</th><th>ColorDet-NN</th><th>FCID-HIST</th><th>FCID-FE</th></tr></thead><tbody><tr><td rowspan="3">D4</td><td>D4</td><td>13.85%</td><td>22.50%</td><td>22.30%</td></tr><tr><td>D5</td><td>27.00%</td><td>33.95%</td><td>31.70%</td></tr><tr><td>D6</td><td>30.45%</td><td>28.00%</td><td>23.65%</td></tr><tr><td rowspan="3">D5</td><td>D4</td><td>25.45%</td><td>38.15%</td><td>38.50%</td></tr><tr><td>D5</td><td>13.85%</td><td>22.35%</td><td>17.30%</td></tr><tr><td>D6</td><td>20.95%</td><td>43.55%</td><td>36.15%</td></tr><tr><td rowspan="3">D6</td><td>D4</td><td>25.80%</td><td>26.95%</td><td>25.10%</td></tr><tr><td>D5</td><td>20.55%</td><td>41.85%</td><td>34.25%</td></tr><tr><td>D6</td><td>12.35%</td><td>24.45%</td><td>22.85%</td></tr></tbody></table>

Table 8 compares the half total error rate results for ColorDet-NN, FCID-HIST and FCID-FE. ColorDet-NN has lower values than those of FCID-HIST and FCID-FE, which implies that ColorDet-NN outperforms FCID-HIST and FCID-FE in detecting fake colorized images.

In summary, the experiments demonstrate that ColorDet-NN has better performance than FCID-HIST and FCID-FE in distinguishing natural images from fake colorized images.
## 5. Conclusions

The ColorDet-NN neural-network-based technique for detecting fake colorized images has three steps. The first step analyzes and validates the statistical differences existing between fake colorized images and their corresponding natural counterparts. The second step employs the cosine similarity of normalized histogram distributions between fake and natural images in various channels to extract features for detection. The third step designs and trains ColorDet-NN to detect fake colorized images. Experiments with six datasets containing fake colorized images generated by three state-of-the-art colorization techniques demonstrate that ColorDet-NN significantly outperforms existing detection methods.

The ColorDet-NN technique exhibits reduced performance when its training and testing sets are drawn from different datasets. This occurs because different colorization techniques with large differences in the statistical information of color distributions significantly impact the extraction of features used for fake image detection. Future research will focus on the common features of colorization techniques and leveraging auxiliary features such as texture to enhance detection. Additionally, efficient neural network structures will be investigated as a means to improve performance.

## References

[1] K. Bahrami, A. Kot, L. Li and H. Li, Blurred image splicing localization by exposing blur type inconsistency, *IEEE Transactions on Information Forensics and Security*, vol. 10(5), pp. 999-1009, 2015.

[2] R. Bohme and M. Kirchner, Counter-forensics: Attacking image forensics, in *Digital Image Forensics*, H. Sencar and N. Memon (Eds.), Springer, New York, pp. 327-366, 2013.

[3] C. Chang and C. Lin, LIBSVM: A library for support vector machines, *ACM Transactions on Intelligent Systems and Technology*, vol. 2(3), article no. 27, 2011.

[4] H. Farid, Creating and Detecting Doctored and Virtual Images: Implications to The Child Pornography Prevention Act, Technical Report TR2004-518, Department of Computer Science, Dartmouth College, Hanover, New Hampshire, 2004.

[5] H. Farid, Exposing digital forgeries from JPEG ghosts, *IEEE Transactions on Information Forensics and Security*, vol. 4(1), pp. 154-160, 2009.

[6] H. Farid, Image forgery detection, *IEEE Signal Processing*, vol. 26(2), pp. 16-25, 2009.
[7] G. Fasano and A. Franceschini, A multidimensional version of the Kolmogorov-Smirnov test, *Monthly Notices of the Royal Astronomical Society*, vol. 225(1), pp. 155-170, 1987.

[8] Y. Guo, X. Cao, W. Zhang and R. Wang, Fake colorized image detection, *IEEE Transactions on Information Forensics and Security*, vol. 13(8), pp. 1932-1944, 2018.

[9] K. He, X. Zhang, S. Ren and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification, *Proceedings of the IEEE International Conference on Computer Vision*, pp. 1026-1034, 2015.

[10] G. Huang, Z. Liu, L. van der Maaten and K. Weinberger, Densely connected convolutional networks, *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pp. 2261-2269, 2017.

[11] S. Iizuka, E. Simo-Serra and H. Ishikawa, Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification, *ACM Transactions on Graphics*, vol. 35(4), article no. 110, 2016.

[12] R. Ironi, D. Cohen-Or and D. Lischinski, Colorization by example, *Proceedings of the Sixteenth Eurographics Conference on Rendering Techniques*, pp. 201-210, 2005.

[13] G. Larsson, M. Maire and G. Shakhnarovich, Learning representations for automatic colorization, *Proceedings of the Fourteenth European Conference on Computer Vision*, Part IV, pp. 577-593, 2016.

[14] Y. LeCun, Y. Bengio and G. Hinton, Deep learning, *Nature*, vol. 521(7553), pp. 436-444, 2015.

[15] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard and L. Jackel, Backpropagation applied to handwritten ZIP code recognition, *Neural Computation*, vol. 1(4), pp. 541-551, 1989.

[16] A. Levin, D. Lischinski and Y. Weiss, Colorization using optimization, *ACM Transactions on Graphics*, vol. 23(3), pp. 689-694, 2004.

[17] Q. Luan, F. Wen, D. Cohen-Or, L. Liang, Y. Xu and H. Shum, Natural image colorization, *Proceedings of the Eighteenth Eurographics Conference on Rendering Techniques*, pp. 309-320, 2007.

[18] E. Reinhard, M. Adhikhmin, B. Gooch and P. Shirley, Color transfer between images, *IEEE Computer Graphics and Applications*, vol. 21(5), pp. 34-41, 2001.
[19] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg and F. Li, ImageNet Large-Scale Visual Recognition Challenge, *International Journal of Computer Vision*, vol. 115(3), pp. 211–252, 2015.

[20] D. Sykora, J. Dingliana and S. Collins, LazyBrush: Flexible painting tool for hand-drawn cartoons, *Computer Graphics Forum*, vol. 28(2), pp. 599–608, 2009.

[21] Y. Wen, K. Zhang, Z. Li and Y. Qiao, A discriminative feature learning approach for deep face recognition, *Proceedings of the Fourteenth European Conference on Computer Vision*, Part VII, pp. 499–515, 2016.

[22] R. Zhang, P. Isola and A. Efros, Colorful image colorization, *Proceedings of the Fourteenth European Conference on Computer Vision*, Part III, pp. 649–666, 2016.