A MULTI-SCALE STRUCTURE-PRESERVING HETEROLOGOUS IMAGE TRANSFORMATION ALGORITHM BASED ON CONDITIONAL ADVERSARIAL NETWORK LEARNING
Authors: Rui Xiang
Guo-yo Wang
Pan Yu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Heterologous Image Transformation
Multi-scale feature encoding
Generative Adversarial Networks

Abstract: Image transformation model learning is a basic technology for image enhancement, image super-resolution, image generation, multimodal image fusion, etc. which uses deep convolutional networks as a representation model for arbitrary functions, and uses fitting optimization with paired image training sets to solve the transformation model between images in the different sets. Affected by the complex and diverse changes of the 3D shape of the actual scene and the pixel-level optical properties of materials, the solution of the heterologous image conversion model is an ill-posed problem. In recent years, most of the proposed conditional adversarial learning methods for image transformation networks only consider the overall consistency loss constraint of the image, and the generated images often contain some pseudo-features or local structural deformations. In order to solve this problem, using the idea of multi-scale image coding and perception, this paper proposes a multi-scale structure-preserving heterologous image transformation method based on conditional adversarial network learning. First, using the idea of multi-scale coding and reconstruction, a multi-scale, step by step generator lightweight network structure is designed. Then, two image multi-scale structure loss functions are proposed, and combined with the existing overall consistency loss, a loss function for generative adversarial learning is designed. Finally, test experiments are performed on the KAIST-MPD-set1 dataset. The experimental results show that, compared with the state-of-the-art algorithms, the proposed algorithm can better suppress the local structural distortion, and has significant advantages in evaluation indicators such as RMSE, LPIPS, PSNR, and SSIM.

Under review as a conference paper at ICLR 2023
A MULTI-SCALE STRUCTURE-PRESERVIVG HETEROL- OGOUS IMAGE TRANSFORMATION ALGORITHM BASED ON CONDITIONAL ADVERSARIAL NETWORK LEARN- ING
Anonymous authors Paper under double-blind review
ABSTRACT
Image transformation model learning is basic technology for image enhance- ment, image super-resolution, image generation, multimodal image fusion, etc. which uses deep convolutional networks functions, and uses fitting optimization with paired image training sets to solve the transformation model between images the different sets. Affected by the com- plex and diverse changes of the 3D shape of the actual scene and the pixel-level optical properties of materials, the solution of the heterologous image conversion model is an ill- ill-posed problem. In recent years, most of the proposed conditional adversarial learning methods for image transformation networks only consider the overall consistency loss constraint of the image, and the generated images often contain some pseudo-features or local structural deformations. order to solve this problem, using the idea of multi-scale image coding and perception, this pa- per proposes a multi-scale cruucture-preserviig heterologous image transforma- tion method based on conditional adversarial network learning. First, using the idea of multi-scale coding and reconstruction, a multi-scale, step by step genera- tor lightweight network structure designed. Then, image multi-scale struc- ture loss functions are proposed, and combined with the existing overall consis- tency loss, loss function for generative adversarial learning is designed. Finally, test experiments are performed on the KAIST-MPD-SCR dataset. The experimen- tal results show that, compared with the state-of-the-art algorithms, the proposed algorithm can better suppress the local structural distortion, and has significant advantages in evaluation indicators such as RMSE, LPIPS, PSNR, and SSIM.
representation model for arbitrary
INTRODUCTION
Heterologous image conversion is process of generating modal B image of the same scene from a modal A image by constructing a pixel conversion model from modality to modality B. Early heterogencous image conversion requires 3D scene models and complete material information to manually design pixel conversion models. However, in practical applications, the construction of 3D models of actual scenes is difficult, the composition of materials is complex, and the acquisition of imaging characteristics of materials is difficult, which makes the solution of heterogeneous image conversion models an ill-posed problem with low efficiency and low applicability.
In recent years, deep learning has led the trend in the field of computer vision, and image transla- tion model learning is a new solution for solving heterogeneous image translation models. Image transformation model learning is an image generation technique for image super-resolee resolutio.. image enhancement, and multimodal image fusion. uses deep convolutional network as representa- tion model for an arbitrary function, and solves a transition model between images of two modalities by using a fitting optimization on training set of paired images. The most mainstream way to learn image conversion models is Conditional Generative Adversarial Nets (CGAN) proposed by Mirza & Osindero (014)). Since CGAN can model the semantic information of images and constrain the output results, CGAN is currently used for Image transformation has become research hotspot in the field of computer vision and the latest heterogeneous image transformation methods are based on it.

---

Under review as a conference paper at ICLR 2023
Image conversion model learning relies large amount of training data. This paper uses the Mul- tispectral Pedestrian Detection (MPD) dataset published by the Korea Advanced Institute of Science and Technology (KAIST)(Hwan) & Jaesik Park 2013). referred to the KAIST-MPD dataset, which is currently the only publicly available dataset that contains a large number of visual and in- frared approximate common aperture images of multiple scenes. Although the heterogenous image pairs of this dataset have viewing angle deviation and the view field of visual image is smaller, the viewing angle deviation is small and the size of the view field remains fixed, so it is feasible to use this dataset to solve the transformation model. The research in this paper is to solve the heteroge- neous image conversion model from infrared visual based on the KASIT-MPD dataset, as shown in the figure below. The contributions of this paper are summarized as:
1.A new four-layer multi-scale encoder generator proposed, in which the images are down- sampled three times, and then they are encoded and decoded from small to large in turn. Such a network structure can make full use of the structural information in the input images;
2. Two dructure-sensitiv loss functions proposed to solve the problem that LI loss and genera- tive adversarial loss cannot effectively constrain the structural information of a single image.
Image
Simulated visual Image
Infrared sensor array
Heterologous image pransorratioo model
Real world
Common aperture imaging system
Model porameter learning
visual Image
Visual
function
array
Figure 1: Heterologous image conversion from infrared to visual.
RELATED WORK
Following the development of deep learning, generative adversarial learning has also achieved rapid development. The original Generative Adversarial Nets (GAN) (Goodfellow et al., (2014) only used the feedback of the discriminator D to adjust the generator G parameters. The discriminator D replaced the method of manually designing the loss function and became a dynamic measure, but the training process of the discriminator itself is uncontrollable and has few constraints, making it difficult to generate high-quality images. MMrza & Osindero (2014) then proposed Conditional Generative Adversarial Nets (CGAN), which sends the input images to the discriminator, and the discriminator needs to distinguish whether the two input images constitute mapping relationship. This improvement enables the generation of the results are more purposeful. Howeeer, the details of the images generated by CGAN are still not clear enough, and there obvious distortions in the images. In 2018, [sola et al. (017)) sorted out the idea of the image conversion model at that time, and proposed a universal pix2pix model, which uses the widely recognized ResNet (He al.. 2016) and Uoone(Ronnerrrger et al. 2013) networks as general generators, added L1 loss, and used the PatchGAN discriminator. Different from the previous single-output discriminator, the PatchGAN discriminator will output discrimination result for each pixel block, and finally form a discriminant surface. Experiments show that this can effectively improve the texture features of the output image, and the picture is clearer. However, pix2pix inevitably has distortion and artifacts, which cannot be effectively constrained by LI loss and discriminative loss.The evolutionary version of pix2pix, pix2pixHD (Wang et al. 2018), is dedicated to generating larger resolution images. The model has several contributions to improve the quality of generated images: First, a new two-layer generator is built, which down-samples the input image to the lower-layer generator, and generates images from small to large; Second, the LI loss is removed, and the Perceptual Loss (PL) (Zhang et al.) 2018a) and the Feature Match (FM) loss are introduced, both of which calculate the 1.2 distance of
2

---

Under review as a conference paper at ICLR 2023
the feature block inside the discriminator. The difference is that the PL loss is used as a "third party inspection" " with a discriminator pretrained on other large datasets, the mapping of the discriminator depends the pretrained dataset, and the discriminator parameters are fixed and unchanged, while The FM loss is a "self-check", the mapping of the discriminator depends on the task training set, the parameters of the discriminator change continuously with the learning process, and the mapping also changes; Finally, the discriminator is multi-scaled, the output image is down-sampled twice, and the three images are sent the three discriminators respectively. The experimental results show that pix2pixHD can generate clear images with resolutions above 1080P, but there are also inevitable artifacts and distortions.
At present, some domestic and foreign scholars have applied generative adversarial learning to solve heterologous image transformation models. For the application of infrared face recognition, ZZang et al. (2018b) introduced the face recognition information as part of the loss function to increase the recognizability of the generated face image. However, compared with the real visual image, the generated face image has obvious differences in facial features, which reduces the reliability of face recognition. Babu[11] 11] added the cycle consistency constraints of infrared and visual faces on the basis of pix2pixHD, and trained both forward and reverse generators at the same time. Although the generation quality has been improved, the problems of distortion and artifacts are still unavoidable. Based the pix2pix model, Kuang al. (2020) introduced the PL loss and added the Total Variation (TV) loss as a regular term to make the generated image smoother, but the image distortion and artifacts were not improved.
In general, most of the existing generative adversarial methods consider texture authenticity and image consistency constraints, but ignore the loss constraints on image structure, resulting in the generated images often contain some pseudo features or local structure deformation. Therefore, how to design a loss function that pays more attention to the image structure and allow the generator to better capture the structural information of the image has become the key to overcoming the pseudo-feature and local distortion of the generated image.
3
ALGORITHM
Since multi-scale images contain local structures at different scales, this paper proposes improve- ments from two aspects, generator model and loss function, form a multi-scale structure- preserving heterogeneous image transformation scheme: First, using the idea of multi-scale encod- ing and decoding, more reasonable multi-scale encoding generator with fewer parameters is pro- posed to fully guarantee the structural reliability of the generated image; Then, two losses sensitive to the local structure of the image are designed and introduced into the loss function effectively suppress the deformation of the local structure of the generated image.
3.1 GENERATOR
The human visual system's 's cognition of target images is a process of multi-scale structural cogni- tion. In the research of image super-resolution, this idea is often used to design network of gradual synthesis from small scales to large scales to obtain high-resolution images et al. 2020). In- spired by such ideas, four-layer generator structure is designed in this paper, as shown in Figure 2. Cone in the figure refers to the convolutional layer; The parameter l: refers to the width of the convolution kernel; d refers to the dilation parameter, its default value is refers stride param- eter, default 1: TranCone refers to transposed convolution, which enlarges the length and width of the feature block
times. The output feature blocks of all modules are 64 channels, and by default all convolutional layers contain instance NarmalzationnUlyaaga functions. It is noted that [Sitzmann et al. (2020) used the sine activation function improve the effect of generating images. In this paper, the sine activation function with learning parameters is used as the activation function, and the formula is as follows:
et al. (016)) and activation
STNF(*) sin(wx + 0)
where x is the input feature block, and angular frequency and phase 0 are learnable parameters. Input the image into the generator, the generator firstly down-samples the image three times in a row to obtain three images of smaller size, and then starts from the image of the smallest size, the generator extracts the features and encodes them. The input of every layer of encoding network is also used to generate heterogenous images of the same scale. The generated images are up-sampled
3

---

Under review as a conference paper at ICLR 2023
using the nearest neighbor method and pixel-wise added to the higher-layer generated image. The hown-sampling process of the image can reduce noise and image texture, and preserve the structural information of different scales. Multi-scale coding information of each scale in the input image. single pixel in the 1/8-size image corresponds the 8*8 pixel area in the original image, so the bottom-up working mode of the generator is actually synthesis process of constructing large-scale features first, and then gradually adding small-scale features. The generator proposed in this paper strengthens the correlation between multi- scale structures, and experiments in subsection 4.2
make full use of the structural
will also verify the superiority of the generator.
Irput
LAyer
Laye festure
LAVE Image
CUES-E
festune
visuol
eocacing
PROOUITT JTTIVS
LiVe
Loye festure
LassT
goncrattdd
Enyer fest.une
Laye-
Tayer Image
1/8 groveated maage
Infraned Imag
FME
ReckvtBiock
Convid
Conu 13
34
ratch
IransCory
supar
IUZU
Figure 2: Model structure of generator.
3.2 LOSS FUNCTION
Common loss functions for image generation are used in this paper, including loss, PI, loss, FM loss, GAN loss and CGAN loss. 1.1 loss, PI. loss, and FM loss are recorded I.L1- LPL. LFM. Both GAN loss and CGAN loss adopt the dual form in pix2pixHD, that is, when training the generator, the generated image is True, and when training the discriminator, the real image is True, and the optimization goal is to reduce the loss at any time. The GAN loss is recorded as LGAN .G when training the generator and LGAN D when training the discriminator. Similarly, the CGAN loss is recorded as LCGAN and LCGAN respectively. Other common loss formulas are the same as those in pix2pix and pix2piXHD, and will not be repeated in this article.
3.2.1 COSINE LOSS
The LI loss calculates the pixel-level image distance, and lacks the correlation constraint between pixels; GAN and CGAN loss is based on probability to distinguish the authenticity of the image, and it is more inclined to generate high-frequency image content, resulting in severe distortion of dynamic targets. is worth noting that when the human visual cognition system observes an image, no matter the overall color of the image is bright dark, long the ratio of the values between pixels remains unchanged, people can obtain almost the same cognition, so the ratio of pixel values represent the content information of the image data. Therefore, in this paper, each image is abstracted into a vector with the dimensions of the number of pixels x the number of channels, and then the cosine angle of the vector is used to measure the similarity between the images. The cosine
4

---

Under review as a conference paper at ICLR 2023
loss is defined as follows:
Lcos (T G(Iinf))
(2)
Lesel [
where Tois represents the visual image; finf represents the infrared image; G(.) represents the generator; I- represents the vector modulus of image; Mp(+) represents multiplying the images in pixels, and then summing pixel values of the result. When the vector angle between the generated visual image and the target visual image is O, the cosine loss reaches the minimum value, that is, the generated image a pixel-wise scaling of the real image. Both cosine loss and LI loss directly calculate the image distance, but unlike the LI loss, where each pixel is calculated independently, the cosine loss calculates the overall structure of the image, and the feedback gradient information represents the global optimization direction. It can guide the network parameter adjustment more effectively.
3.2.2 TOTAL VARIATION LOSS OF DIFFERENCE
Although the Total Variation (TV) loss deseribe the noise level of the image and is added to the loss function as regular term for suppressing noise, the image can only be a completely flat monochrome image when the TV loss obtains the optimal solution. Therefore, if the ratio parameter of TV loss is too large, it will inevitably lead to a lot of blurs in the generated image, and the edge information in the image lost, which can describe the target better than color, the introduction of TVV loss is obviously flawed. In order to make the image smooth and keep the edge structure of the image better, total variation of difference (TVD) proposed, which is defined as follows:
LTVD = (G(V)(((( Jeis)) + E, (V(V - Ivis)
(3)
where 2 represents calculating the difference map of the image in the direction, and then squar- ing by pixels; V represents calculating the difference map of the image in the y direction, and then squaring by pixels; F.,.) represents calculating the average pixel value. Figure shows the calculation method of the difference maps in the and y directions this paper.
1 to n-1
input image (n pixels)
sub image (n-1 m pixels]
sub image
x-direction difference map m pixels]
(n-1 pixels)
2 to
column to column m-1
Input image (nx pixels)
image In m-1 pixels)
image m-1 pixels)
y-direction difference map (n x m-1 pixels)
column to column
Figure 3: Get the difference map.
3.2.3 TOTAL LOSS FUNCTION
The total loss function of the algorithm is as follows:
LG = (L + Loos + .LVD) ) + >2 + LOGAN.G) + Aa(Lras + Lr)
(4)
Lp >2 LOGANDD)
(5)
where LG represents the loss function used for training the generator; Lp represents the loss function for training the discriminator; In this paper, losses are classified into three categories, and losses of the same kind use the same scaling factor: X1 is scaling factor of pixel class; X is the scaling factor of discriminator class; Ag is the scaling factor feature distance class.
5

---

Under review as a conference paper at ICLR 2023
Table Pseudo code of algorithm
Code
G generator in this paper;
D 5layers multi-scale PatchGAN;
C = 5layers multi- scale PatchGAN;
P VGG19(layer 1_1 to layer 5_1); Initialize G, D, C. P:
Import pretraining data;
While(Do not meet training termination condition)
Import image pairs: L and Lussi
Generate image I = G(
Calculate generator loss Lg:
Backpropagates and updates the parameters of G;
Generate image I = G(Jnn)):
Calculate discriminator loss Lp:
Backpropagates and updates the parameters of D and C;
3.3 OVERALL STRUCTURE AND PROCESS
The overall structure of the algorithm is shown in Figure below, and the pseudocode the al- gorithm flow is shown in Table [ The generator G adopts the generator model in this paper. The discriminators D and both use the multi-scale PatchGAN network, the used code which the same as the code in pix2pixHD, and the perception network P adopts the layer 1 1_1, 3_1, 4_1 and 5_1 of pre-trained VGG19 to calculate perceived loss of output features.
real
generated
discriminator
infrared
generator
loss
visual image
image
discriminator
CGAN loss
TVD loss
FM loss
perceptual network
real visual image
PL loss
Figure 4: Overall structure of the algorithm.
4 EXPERIMENT
The experiments in this paper use the set01 subset of the KASIT-MPD dataset, and the 10th pair is selected as the test set for every 10 pairs of images, and the rest are used as the training set. There are 7227 pairs of infrared-vins visual images the training set and 802 pairs of images in the test set. To reduce computational power consumption, all infrared and visual images were down-sampled and resized to 256x256 pixels.
In the experiment, all networks are trained for 200 epochs, using the ADAM optimizer, the param- eters of which are (0.5, 0.999). The learning rate is 2e-4 initially, which in the first 100 epochs remains unchanged, but decreases lincarly 1/101 of the initial learning rate in the last 100 epochs. Å₁ Xz, and Ag are respectively 100, 1, and 10. In the experiment, the first layers of the VGG19 model provided by pytorch used to calculate IPIPS, which is the same the network structure

---

Under review as a conference paper at ICLR 2023
Table 2: Evaluation metrics for different algorithms. The bold number indicates the best result and the number with asterisk indicates the second best result.
The smaller the better
The bigger the better
Net
Param(M)
RMSE
LPIPS
PSNR
SSIM
pix2pix
11.37
10.88*
12.96
17.03*
0.62
pix2pixHD
45.59
13.70
11.76
16.01
0.60
TICCGAN
45.85
12.29
11.39*
16.57
0.61
PCSGAN
11.37
10.93
12.63
16.98
0.61
Ours
10.81
8.61
10.30
18.11
0.69
for calculating perceptual loss, and all use the pre-training data provided by pytorch. The metries of the generated image quality are RMSE, LPIPS, PSNR and SSIM.
4.1 COMPARATIVE EXPERIMENTS
The algorithm this paper is compared with the general algorithms pix2pix (solla et al. 2017) pix2pixHD Wang al. (0188) and the open source algorithms TICCGAN(Kuang al.) (020) and PCSGAN(Babu & Dubey 2020) for infrared to visual image conversion. The generated image com- parison is shown in Figure the index calculation is shown in Table and the detailed comparison is shown in Figure In Figure 5 the scenes in rows 2, 3, and 4 have a lot of repetition in the dataset, so all existing methods can achieve good background generation results in these scenes. But when the scene changes drastically (the scene in the first row is obtained when the car turns quickly), or when there are dynamic objects such as people and cars, the images generated by the existing algo- rithms suffer from a lot of distortions and false features. The reason for this image degradation is that the constraints designed in the reference algorithm do not fully utilize the pixel-related informa- tion within single image. Algorithms that rely heavily the discriminator to improve image quality tend to generate more frequent information, such as static scenes, and ignore less frequent information, such pedestrians and cars. It be from the comparison that the algorithm in this paper focuses on the utilization of the information between the pixel values within a single im- age, better maintains the structural information, and makes the edges of moving objects clearer and more reliable. The algorithm in this paper outperforms the reference algorithm in every evaluation index, and the generator proposed in this paper has fewer parameters.
infrared
visuel
pix2pix
pic2oxHD
POSGAN
Figure Comparison of multi-scene generation of different algorithms.

---

Under review as a conference paper at ICLR 2023
real visual
real visual
pix2pix
pix2pixHD
TICCGAN
PCSGAN
Ours
Figure 6: Comparison of vehicle generation of different algorithms.
4.2 ABLATION EXPERIMENT
The ablation experiments are carried out for the effectiveness of the network and loss functions proposed this paper. The experimental results are shown in Figure 7 the evaluation indicators are shown in Table and the detailed comparison is shown in Figure g In the ablation experiments discussing the effectiveness of the network structure, the 9-layer ResNet generator adpoted in pix2pix is used as an alternative generator G, which is similar to the network proposed in this paper in the number parameters
According to the comparison in Figure 7 and Figure g the lack of any of the three improvements proposed in this paper will make the cars and people in the image more distorted, and the structural information of the image will be lost more. The evaluation indicators also prove the effectiveness of the improvements.
real
GReeNeet
withou
without
Figure Comparison of Ablation Experiment.
5 CONCLUSION
This paper proposes network structure for multi-scale encoding and two loss functions sensitive to image structure information. Experiments on paired infrared-visual dataset show that the algorithm proposed in this paper generates fewer image distortions and pseudo-features than the
8

---

Under review as a conference paper at ICLR 2023
Table 3: Evaluation indicators for ablation experiment. The bold number indicates the best result.
The smaller the better
The bigger the better
Net
Param(M)
RMSE
LPIPS
PSNR
SSIM
Ours
10.81
8.61
10.30
18.11
0.69
use ResNet G
11.37
9.35
11.54
17.68
0.66
without COS
10.81
9.16
10.71
17.84
0.67
without TVD
10.81
9.01
10.68
17.9
0.67
real visual
real visual
Ours
use ResNet
without COS
without TVD
Figure 8: Comparison of vehicle generation in Ablation Experiment.
statc-of-the-arr algorithm, and has obvious ad advantages in all evaluation indicators of image quality.
REFERENCES
Kancharagunta Kishan Babu and Shiv Ram Dubey. Pesgan: Perceptual synthsssytteeizeed genera- tive adversarial networks for thermal and nir to visible image transformation. Neurocomputing, 413:41-50, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. A fast learning algorithm for deep belief nets. Advances in neural information processing systems, 27, 2014.
Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao, Zeshuai Deng, Yanwu Xu, and Mingkui Tan. Closed-loop matters: Dual regression networks for single image super-resolution. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5407- 5416, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Soonmin Hwang and Yukyung Choi Jaesik Park, Namil Kim. Multispectral pedestrian detection: Benchmark dataset and baseline. Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 1037-1045, 2015.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.

---

Under review as a conference paper at ICLR 2023
Xiaodong Kuang, Jianfei Zhu, Xiubao Sui, Yuan Liu, Chengwei Liu, Qian Chen, and Guohua Gu. Thermal infrared colorization via conditional generative adversarial network. Infrared Physics & Technology, 107, 2020.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1700 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, 2015.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 2016.
Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2018.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 586-595, 201
Teng Zhang, Amoll Wiliem, Siqi Yang, and Brian Lovell. Tv-gan: Generative adversarial network based thermal to visible face recognition, 2018 international conference on biometrics (ICB), pp- 174-181, 2018b.
10


Reviews:
[{"review_id": "mXrrAUkBlT", "reviewer": "Reviewer_5dFs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work solves the problem of transforming images from infrared to real visual spectrum that too solely focuses on the KAIST-MPD dataset. To this end, the authors propose to use a multi-scale generator architecture and new loss functions such as cosine loss and total variation loss of the differences. ", "summary_of_the_review": "The work suffers from poor writing quality, lack of novelty, and too narrow scope keeping it well below the acceptance level.", "strength_and_weaknesses": "Strengths\n \n1. Though I have seen the use of cosine loss in the literature associated with the classification or metric learning, I haven’t encountered using the same for image transformation. This can be an exciting proposal to look at.\n \n \nWeaknesses\n \n1. Poor writing standard: The paper is poorly written. Below are some examples\n\n- Input the image into the generator, the generator firstly down-samples the image three times in a row to obtain three images of smaller size, and then starts from the image of the smallest size, the generator extracts the features and encodes them. => poorly framed sentence, written as if one is talking to another in-person\n\n- Both GAN loss and CGAN loss adopt the dual form in pix2pixHD, that is, when training the generator, the generated image is True, and when training the discriminator, the real image is True, and the optimization goal is to reduce the loss at any time. The GAN loss is recorded as LGAN G when training the generator and LGAN D when training the discriminator\n- Table 1 seems to be redundant. \n- Poorly showcased results. It is extremely difficult to compare and understand the differences among the results shown in Figs. 5-7\n \n2. Lack of novelty: Multi-scale-based network architectures are very common in image transformation and restoration. Some of these works are cited by the authors themself. Some other works, for example, includes: {Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution CVPR 2017, Scale-recurrent network for deep image deblurring CVPR 2018, Msg-gan: Multi-scale gradients for generative adversarial networks, CVPR 2020} I do not see any novelty in terms of the design of the generator as such.\n \n3. The paper's objective doesn’t have too much of an impact: This work aims to improve the image transformation performance specifically for the KASIT-MPD dataset. As such, the advantages of the proposed method for general image transformations are unclear. \n \n4. Experiments can be biased: Though authors have compared with some state-of-the-art methods, there is no clear description indicating if authors have used pre-trained networks or newly trained (on the dataset under consideration) versions of these state-of-the-art methods. \n \nMinor issue:\n\nSection 2: Babu[11] - erroneous citation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "KnsecoBiLl", "reviewer": "Reviewer_KAPg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is interesting. It summarizes varios image processing tasks as image transformation trained on adversersial networks.", "summary_of_the_review": "This paper can be accepted if more experiments are evaluated.", "strength_and_weaknesses": "Strength:\n1. SOTAs on the KASIT-MPD dataset\n\nWeaknesses:\n1. Only one dataset and infrared-rgb task is evaluated.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3Lh45iRfYNg", "reviewer": "Reviewer_o1H8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A multi-scale structure-preserving heterologous image transformation method based on conditional adversarial network learning is proposed in this paper. The experimental results show that the proposed algorithm can better suppress the local structural distortion and has significant advantages in evaluation indicators such as RMSE, LPIPS, PSNR, and SSIM.", "summary_of_the_review": "See the above", "strength_and_weaknesses": "Strengths:\n1.\tThe proposed loos function is capable to consider the correlation constraint between pixels.\n2.\tTotal variation of difference (TVD) is proposed to make the image smooth and keep the edge structure of the image better.\n\nWeaknesses:\n1.\tThe novelty is limited. As a major contribution to the author's summary, what are the advantages of multi-scale feature extraction over existing methods? It looks similar to most multi-scale feature extraction methods.\n\n2.\tIt will be better if the results of more state-of-the-art methods proposed in 2021 and 2022 are provided.\n\n3.\tMore descriptions should be given in the titles of the figures and tables.\n\n4.\tThe layout of the subfigures should be improved, e.g., Fig.6 and Fig.8. \n\n5.\tThe results of runtime should be added in the paper.\n\n6.\tThe number of references is too small. References can be further improved.\n\n7.\tThe format of the tables should be revised. These tables should contain three lines.\n\n8.\tIt will be better if the results of some non-reference metrics are provided.\n\n9.\tThere are some grammar mistakes, e.g., “the state-of-the-art algorithm” in Conclusion should be “the state-of-the-art algorithms”.\n\n10.\tSome results of the proposed loss functions on other image transformation methods should be added to validate the effectiveness.\n\n11.\tThe selection of hyperparameters used in the combination of loss functions should be analyzed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject