Data Generation Scheme for Thermal Modality with Edge-Guided Adversarial Conditional Diffusion Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 04:01:20

Abstract: In challenging low-light and adverse weather conditions, thermal vision algorithms, especially object detection, have exhibited remarkable potential, contrasting with the frequent struggles encountered by visible vision algorithms. Nevertheless, the efficacy of thermal vision algorithms driven by deep learning models remains constrained by the paucity of available training data samples.
To this end, this paper introduces a novel approach termed the edge-guided conditional diffusion model (ECDM). This framework aims to produce meticulously aligned pseudo thermal images at the pixel level, leveraging edge information extracted from visible images. 
By utilizing edges as contextual cues from the visible domain, the diffusion model achieves meticulous control over the delineation of objects within the generated images. To alleviate the impacts of those visible-specific edge information that should not appear in the thermal domain, a two-stage modality adversarial training (TMAT) strategy is proposed to filter them out from the generated images by differentiating the visible and thermal modality. 
Extensive experiments on LLVIP demonstrate ECDM’s superiority over
existing state-of-the-art approaches in terms of image generation quality.
The pseudo thermal images generated by ECDM also help to boost the performance of various thermal object detectors by up to 7.1 mAP.

Main paper: 
 related work: 
 To solve the lacking of thermal images, some studies attempt to employ the domain adaptation techniques [17] , by fine-tuning the pretrained visible object detectors into the thermal domain. The main promising ideas are multi-level feature alignments [29] and style consistency constraints [30] . Nonetheless, while domain adaptation methods may mitigate the issue of insufficient annotations, they continue to face challenges in the absence of thermal images.Another mainstream of studies rely on generative-based methods to generate synthetic thermal images. In [2, 3, 28] , there is a discussion of the use of virtual environments to create synthetic thermal images. These methods rely on intricate 3D models now focusing only on objects rather than whole scenes and employ infrared physics-based rendering. In [5, 18] , generative models are discussed to create synthetic thermal images. But these generative models-based methods can not generate pixel-level alignments of thermal images from visible images.Deep generative models (DGMs) are neural networks trained to approximate the probability distributions of data. After training successfully, we can generate new samples from the underlying distribution. Generative Adversarial Networks (GANs) [11] , as a type of DGMs, have been extensively employed in the image-to-image translation tasks [15, 21, 24, 48] . Basic GANs consist of a generator and a discriminator under an adversarial training framework. The adversarial training process can be modeled as a min-max game. However, they have drawbacks such as poor convergence characteristics, especially on the thermal modality with rare textures.Recently, diffusion models (DMs) [14] as a novel paradigm in the generative model, were shown impressive generative capabilities in high level of details [4] . Compared to GANs, this approach has a more stable training process and produces a greater range of diverse images. Recent advancements in DMs have demonstrated the ability to control the generation process, including details, through various conditions like image [20, 35, 45] , class [6] , and text [33] . DMs and their variants possess intriguing properties, such as stable training, generative diversity in images, and details control through conditions. These properties make them suitable for generating training data from visible images for the thermal object detection task.Diffusion-GAN [40] attempt utilize the advantage of the flexible diffusion model to stability the training process of GANs. In contrast to [40] which injects adaptive noise via diffusion at various time steps to provide higher training stability over strong GAN baselines, our two-stage modality adversarial training (TMAT) strategy utilizes adversarial training to mitigate the distribution mismatch in generating images under diverse conditions. In the second stage, we use unpaired 𝑥 𝑣𝑖𝑠 and 𝑥 𝑡𝑖𝑟 as input and utilize GANs to reduce the gap between visible and thermal domains. This helps us learn the distribution of 𝑝 𝑚𝑜𝑑𝑒𝑙 (𝑥 𝑡𝑖𝑟 |𝜁 𝑣𝑖𝑠 ) for approximating 𝑝 𝑡𝑖𝑟 (𝑥 𝑡𝑖𝑟 ). 
methodology 3.1 framework overview: 
 In this section, we first formalize the problem of generating pseudo training data for thermal object detection. As visible object detection datasets typically exhibit greater scale than their thermal counterparts, we leverage existing visible datasets to craft pseudo training samples for thermal object detection. Given a real visible object detection dataset D 𝑣𝑖𝑠 = {𝑥 𝑣𝑖𝑠 𝑖 , 𝑦 However, different from the task of image generation or imageto-image translation, our generated pseudo images are prepared for thermal object detection. Consequently, generative thermal images necessitate pixel-level alignment with their corresponding visible images to accurately represent objects. To attain this precise alignment, we introduce the Edge Condition Diffusion Model (ECDM) rooted in conditional diffusion models. In our approach, edge images play a crucial role as the guiding condition during the sampling process for the creation of training samples.Although edge information can bridge the thermal and visible domains, some discrepancies persist between the corresponding edge images in these domains. To address this challenge, we introduce a two-stage modality adversarial training strategy instead of a direct end-to-end training approach for the ECDM. Initially, we utilize thermal edge images to train the ECDM, enabling it to translate thermal edge images into thermal images. Subsequently, we leverage the trained ECDM as a generator and devise a discriminator. Through adversarial training resembling a GAN approach, we work towards minimizing the disparity between synthetically generated thermal images under visible edge conditions and authentic thermal images.Texture, shape, and color stand as the paramount visual cues in visual recognition [9] . However, due to the substantial differences in primary radiation, a considerable gap exists between thermal infrared and visible images. Specifically, thermal images have lack color information and exhibit lower texture than their visible counterparts. Shape information exhibits notable similarity between thermal and visible images within the same scene. We posit that this shape information serves as a bridge, mitigating the gap between the thermal and visible domains to some degree. The shape information in the images can be roughly extracted by high-frequency filteringEQUATIONwhere 𝑚 ∈ {𝑣𝑖𝑠, 𝑡𝑖𝑟 } is a modality indicator superscript, and H (•) is an edge extracted operator contains a fast Fourier transform, a high pass filtering, and an inverse fast Fourier transform. The extracted shape information is also termed as edge images. Different from image generation task in vanilla diffusion models [14, 37, 40] or conditioned on the class label, text, or natural images, the proposed ECDM introduces prior knowledge related to the fine-granularity content of objects, i.e., edge images. The ECDM comprises both the diffusion process and the reverse process. It destroys the input thermal image 𝑥 𝑡𝑖𝑟 0 ∈ D 𝑡𝑖𝑟 to a standard Gaussian noise 𝑥 𝑡𝑖𝑟 𝑇 ∼ N (0, I) by gradually adding small Gaussian noise in 𝑇 diffusion steps in forward process. The forward process of ECDM is presented as followsEQUATIONEQUATIONwhere 𝛽 𝑡 is a small positive constant to control the variants of the added noise in diffusion step 𝑡 (0 ≤ 𝑡 ≤ 𝑇 ). Following [14] , the variance schedule is predefined linearly increasing from 𝛽 1 = 10 -4 to 𝛽 𝑇 = 0.02 and diffusion steps is seted 𝑇 = 1000. The noised sample at diffusion step 𝑡 can be directly calculated byEQUATIONwhere 𝛼 𝑡 = 1 -𝛽 𝑡 , ᾱ𝑡 = 𝑡 𝑖=1 𝛼 𝑖 . The reverse process of ECDM is conditioned on the edge images to bridge the gap between the thermal domain and the visible domain while capturing the fine-granularity content of objects. Note that the different training stages use edge images in different domains. The reverse process is written asEQUATIONEQUATIONThe parameterizations of 𝜇 𝜃 and 𝜎 𝜃 are defined byEQUATIONEQUATIONwhere 𝜅 𝜃 is a neural network parameterized by 𝜃 , implemented by a modified UNet [14, 34] . In our work, 𝜅 𝜃 takes the noised image 𝑥 𝑡𝑖𝑟 𝑡 , the conditional edge image 𝜁 𝑚 , and the diffusion step 𝑡 as input. The the diffusion step 𝑡 are fed into a Transformer sinusoidal position embedding layer [39] with a given embedding dimension, followed by a Linear + sigmoid layer. Then, for each downsample and upsample block, an additional Linear + sigmoid layer is employed to align the channel dimensions of the feature maps. Figure 4 shows the architecture of ECDM.Our goal is to learn the distribution of 𝑝 𝑚𝑜𝑑𝑒𝑙 (𝑥 𝑡𝑖𝑟 |𝜁 𝑣𝑖𝑠 ). However, attempting to manually train the ECDM in a straightforward endto-end manner, such as 𝜁 𝑚 = 𝜁 𝑣𝑖𝑠 in (7), while theoretically feasible, becomes challenging in practice due to the substantial divergence between the thermal and visible domains.To address this challenge, we propose a two-stage modality adversarial training strategy. As illustrated in Figure 3 , in the first stage, we set 𝜁 𝑚 = 𝜁 𝑡𝑖𝑟 and train the model to learn the distribution of 𝑝 𝑚𝑜𝑑𝑒𝑙 (𝑥 𝑡𝑖𝑟 |𝜁 𝑡𝑖𝑟 ). The condition 𝜁 𝑡𝑖𝑟 and generative images 𝑥 𝑡𝑖𝑟 all remain within the thermal domain, simplifying the distribution learning process. In the second stage, we incorporate the principles of GANs, employing the ECDM previously trained in the first stage as the generator and employing a thermal modality authenticity indicator as the discriminator. At this stage, we begin by generating images from the generator under a distinct condition 𝜁 𝑚 = 𝜁 𝑣𝑖𝑠 , which introduces modality bias due to the incongruity between 𝑝 (𝜁 𝑡𝑖𝑟 ) and 𝑝 (𝜁 𝑣𝑖𝑠 ). We then mitigate this modality bias through adversarial training. The complete two-stage modality adversarial training procedure is detailed in Algorithm1.Specifically, in the first stage, we focus on training the ECDM by optimizing the standard variational bound on negative loglikelihood. Following the reparameterization trick in [14] , the training objective of ECDM in the first stage training process isEQUATIONIn the second stage, we leverage the ECDM as a generator 𝐺 and introduce a discriminator 𝐷 by PatchGAN [15] , where 𝐺 aims to generate synthetic thermal images x𝑣𝑖𝑠 ∼ 𝑝 𝑡𝑖𝑟 (𝑥 𝑡𝑖𝑟 ) the condition of visible edge images, 𝐷 aims to distinguish the real and synthetic thermal images. To ensure that the generated thermal images are indistinguishable from authentic ones, we employ an adversarial loss [11] EQUATIONwhere 𝜆 𝑟𝑒𝑎𝑙 is a hyper-parameter to balance the adversarial loss components. We set 𝜆 𝑟𝑒𝑎𝑙 = 10 in our experiments. To further narrow the gap between the generated thermal images and actual ones, we incorporate a style-consistency loss defined asEQUATIONand design a modality-consistency loss for 𝑘 = 0 to 𝑆 𝐺 do 10:EQUATIONUpdate 𝜅 𝜃 by descending its gradient:11: for 𝑙 = 0 to 𝑆 𝐷 do 17:▽ 𝜅 𝜃 ∥ 𝜅 -𝜅 𝜃 (𝑥 𝑡𝑖𝑟 𝑡 , 𝜁 𝑡𝑖𝑟 , 𝑡) ∥ 2 2 12: ▽ 𝜅 𝜃 ∥ x𝑡𝑖𝑟 -𝑥 𝑡𝑖𝑟 ∥ 2 2 13: ▽ 𝜅 𝜃 ∥1 -𝐷 ( x𝑣𝑖𝑠 )∥ 2 2 14: ▽ 𝜅 𝜃 ∥H (𝑥 𝑣𝑖𝑠 ) -H ( x𝑣𝑖𝑠 )∥ 2Update 𝐷 by descending its gradient:18: ▽ 𝐷 [𝜆 𝑟𝑒𝑎𝑙 ∥ log(𝐷 (𝑥 𝑡𝑖𝑟 ))∥ 2 2 19: + ∥ log(1 -𝐷 ( x𝑣𝑖𝑠 ))∥ 2 2 ] 20:end for 21: end for 22: return ECDM (𝜅 𝜃 ).where x𝑡𝑖𝑟 := 𝐺 (𝜅, 𝜁 𝑡𝑖𝑟 ) and x𝑣𝑖𝑠 := 𝐺 (𝜅, 𝜁 𝑣𝑖𝑠 ) represent synthetic thermal images under conditions of thermal edge images or visible edge images, respectively. Additionally, we utilize an edge loss to preserve accurate boundaries and highly detailed shapes, which is defined asEQUATIONFinally, we express the objective functions to optimize 𝐺 and 𝐷, respectively, as followsEQUATIONEQUATIONwhere 𝜆 𝑑𝑖 𝑓 𝑓 , 𝜆 𝑠𝑡 𝑦𝑙𝑒 , 𝜆 𝑚𝑜𝑑 , 𝜆 𝑒𝑑𝑔𝑒 are hyper-parameters to control the weight of different loss. We use 𝜆 𝑑𝑖 𝑓 𝑓 = 0.1, 𝜆 𝑠𝑡 𝑦𝑙𝑒 = 100, 𝜆 𝑚𝑜𝑑 = 1, 𝜆 𝑒𝑑𝑔𝑒 = 1000 in all our experiments. Moreover, we use dpm-solver++ [27] to improve the sampling speed of ECDM. The sampling parameters of DPM-solver++ are listed in the supplementary material. 
experiments 4.1 experimental settings: 
 4.1.1 Datasets. Our experiments are mainly on the Low-Light Visible-Infrared Paired Dataset (LLVIP) [16] , FLIR thermal dataset (FLIR) [8] , and Person Re-identification in the Wild Dataset (PRW) [47] . LLVIP consists of 15,488 pairs of visible-thermal images, captured under low-light conditions using a binocular surveillance camera.These paired images are precisely aligned both spatially and temporally. For brevity, we refer to this dataset asD 𝑙𝑙 𝑣𝑖𝑝 . FLIR has two versions: v1.3 (2019) and v2.0 (2021). We utilize v2.0 for its expanded training dataset. Our analysis focuses on five categories: person, bike, car, light, and sign. PRW comprises 11,816 frames captured during the summer months using a visible camera. These frames were extracted from the Market-1501 dataset [46] . This dataset is annotated for both person re-identification and pedestrian detection tasks. We refer to this dataset D 𝑝𝑟 𝑤 .4.1.2 Metrics. We use Fréchet Inception Distance (FID) [13] , Learned Perceptual Image Patch Similarity (LPIPS) [44] , peak-noise-to-signal ratio (PSNR), the structural similarity index measure (SSIM), and Kernel Inception Distance (KID) [1] to measure the quality of generated thermal images. FID is a widely adopted non-reference perceptual metric that assesses the similarity between two sets of images. We denote the standard implementation in [36] as FID. Besides, the FID implemented in [31] is FID-C, and the FID implemented using CLIP instead of InceptionV3 is FID-C 𝑐𝑙𝑖𝑝 . Additionally, we employ KID, a metric similar to FID but with a polynomial kernel for an unbiased estimator.We use standard mean Average Percision [23] (mAP) under different Intersection over Union (IoU) thresholds as the metrics to measure the gain of generated pseudo training data to the performance of thermal detectors.We train the ECDM on four NVIDIA 3090 24GB GPUs, utilizing a batch size of 4 and resizing the input images to a resolution of 512 × 640. The generator 𝐺 and discriminator 𝐷 are optimized using Adam with 𝛽 1 = 0.9 and 𝛽 2 = 0.999. The learning rate is set to 0.00002. Our training procedure involves setting 𝑆 𝑑𝑖 𝑓 𝑓 = 70, 𝑆 𝑎𝑑𝑣 = 20, 𝑆 𝐺 = 2, and 𝑆 𝐷 = 1. Further details can be found in the supplementary material.Our training process in Sec. 4.7 utilizes two NVIDIA 3090 24GB GPUs with a batch size of 32 and employs SGD as the optimizer. We set the base learning rate to 0.0002, momentum to 0.99, and weight decay to 0.0001. To ensure a fair comparison, we train them for 40k iterations and use a multistep learning rate scheduler with steps at 12000, 18000, and 32000 iterations.We demonstrate that ECDM can deliver competitive results in visible-to-thermal translation tasks. We compare our method with several state-of-the-art methods: pix2pixGAN [15] , CycleGAN [48] , UGATIT [19] , LPTN [21] , VSAIT [38] , DDIM [37] and BBDM [20] .We also benchmark some energy-based/flow-based models in our experiments, but their performance was inferior to the GAN-based methods presented in Table 1 . We report the performance under 512 × 640 resolution which specifically addresses thermal object detections. For DDIM, we modified the original architecture as shown in Figure 4 . However, in our case, we used visible images from the LLVIP dataset as the conditioning input for the visible-to-thermal image translation task. We evaluate BBDM's performance by upsampling generated images from 256×256, due to its difficulty in converging at 512×640.As can be seen in Table 1 , ECDM achieves superior performance to SOTA thermal image generators on the LLVIP dataset. It suggests that by introducing visible edge conditions to guide diffusion modeling, our proposed ECDM effectively generates high-quality pixel-level aligned pseudo thermal images with visible edge images. We also show qualitative comparison with other methods in Figure 5 . Our qualitative comparison is based on two key principles: object shape reconstruction and object thermal radiation characteristic reconstruction. The latter principle is based on the thermal imaging mechanism. Thermal imaging relies on emitting radiation from objects. The higher the temperature, the brighter the object in the thermal image.In Figure 5 , although some methods can reconstruct full objects like cars, they still fail to capture the thermal radiation characteristic of objects in generated images. For example, the tires of moving vehicles and the exposed skin of pedestrians should appear brighter in thermal images. GAN-based methods, such as CycleGAN and UGATIT, excel in object shape reconstruction but fall short in object thermal radiation characteristic reconstruction. These GAN-based methods also tend to reconstruct the object texture rather than the object thermal radiation characteristics, such as stationary vehicles and sidewalks. Conversely, diffusion-based methods, such as DDIM and BBIM, demonstrate proficiency in thermal radiation characteristic reconstruction but struggle with shape reconstruction. Our proposed method stands out by achieving both shape reconstruction and thermal radiation characteristic reconstruction simultaneously. More visual results can be found in supplementary material.We compare the complexity of ECDM with other methods in terms of the number of parameters (#params) and FLOPs. All results computed using the calflops package [41] .As can be seen in Table 2 , our method is comparable to GANbased method except LPTN in #params. Compared with DDIM, our method introduces an additional discriminator, resulting in an increase of 2.765M #params. Diffusion-based methods generally require higher FLOPs because of multisteps reversal process compared with GAN-based methods. However, our method benefits We also report that generating 512 × 640 of one thermal image needs about 14.7s on a NVIDIA 3090 GPU. However, we can apply some acceleration schemes of diffusion to tackle the efficiency problem. Besides, the data generation is a one-time operation, and any other downstream tasks can use them.In the experimental setup described above, the visible edge images and target generated thermal images are from the same dataset (all in the LLVIP dataset), resulting in a small gap between them. In practice, generating pseudo thermal images that can yield gains for downstream tasks such as thermal object detection poses a challenging problem: will the generated training data be useful when the conditions are far from the target domain? This raises the issue of model transferability. 3 , substantial degradations in various metrics are evident due to domain and dataset disparities. Nonetheless, welltrained ECDM attains a minimal FID-C score under cross-domain and cross-dataset sampling conditions.To validate the effectiveness of our TMAT strategy in the face of transferability, we train the ECDM with TMAT under diverse edge inputs. As shown in Table 4 , the trends of various metrics are consistent, demonstrating that TMAT diminishes the gap between disparate domains and datasets.To investigate the impact of the number of pseudo thermal images on thermal object detection performance, we select RetinaNet [22] as our baseline and train it under the same settings, except for the training data. The training data consists of two parts: real thermal We experiment with diverse augmentation multiple ratios, namely 0, 0.2, 0.4, 0.6, 0.8, and 1.0, and observe the impact on mAP. As depicted in Figure 2 , the mAP improves gradually from 0 to 7.1, with the most significant enhancement occurring at a ratio of 1.0.To further effective the generalization of pseudo training data generated by our ECDM, we train various object detectors, including Faster RCNN [32] , RetinaNet [22] , CenterNet [7] , VFNet [43] , and DINO [42] . For a fair comparison, we maintain an augmentation multiple ratio of 1.0 throughout this experiment.Our generated pseudo training data yield mAP improvements ranging from 1.1 (CenterNet) to 7.1 (RetinaNet) across different detectors on the LLVIP dataset. Notably, most detectors, excluding VFNet, exhibit mAP improvements between 0.7 and 1.8, demonstrating the effectiveness of our pseudo training data on the FLIR dataset. The Varifocal Loss in VFNet focuses the training on those highquality positive examples that are more important for achieving a higher AP than those low-quality ones [43] . Considering the dataset characteristics, FLIR comprises objects of different scales, with a predominant presence of small-scale objects. LLVIP primarily consists of medium-scale objects, while PRW contains a mix of medium-and large-scale objects. The unique design of the Varifocal Loss in VFNet places more emphasis on the generated pseudo objects compared to other detectors since large-scale objects are more easily identified as positive examples than small-scale objects. This design also explains the performance decrease of VFNet on the FLIR dataset and its performance improvement on the LLVIP dataset.We also evaluate the performance of pseudo-training data generated by ECDM or other image-to-image translation techniques on the RetinaNet. As shown in Table 6 , when compared to the absence of any additional training data, the utilization of generated pseudotraining data proves effective in enhancing the performance of RetinaNet. Notably, our ECDM achieves the highest improvement (+7.1 mAP) compared with other methods.  
limitations and future works: 
 Our ECDM currently requires high-quality edge images, which limits its applicability in certain scenarios. Furthermore, we also noticed that the generated images often have global color levels error, especially on the ground, traffic lights, and woods, which may account for the large FID score in Table 1 . Additionally, as illustrated in Fig 5 , our method and DDIM tend to generate brighter thermal images compared to the ground truth or the CycleGAN result. Notably, BBDM, also a diffusion-based method, does not exhibit this trend. Considering the comparison of model parameters in Table 2 . We attribute this issue to the U-Net's limitations in parameter count and learning capacity.Considering the relationship between the thermal object detection and other downstream tasks which also suffer the lacking of training samples like thermal object tracking, thermal semantic segmentation, and thermal image super resolution, our method is expected to bring performance improvements to them. However, tracking tasks typically require image sequences as input, necessitating the design of additional modules to ensure frameinter consistency for generated images. It also exhibits potential applicability to image-to-image translation where edges can act as connectors between the source and target domain, particularly when the texture information in the target domain is relatively sparse. 
conclusion: 
 In this paper, we introduce a novel data generation scheme for the thermal modality, called ECDM, which leverages a diffusion model to generate pixel-level aligned thermal images. Our approach utilizes edge images extracted from visible images as a condition to guide the diffusion model in learning the fine control of object boundaries in the generated image. To address the domain gap between thermal and visible images, we propose TMAT, a method that trains our ECDM to generate thermal images from visible edge images. Our extensive experiments demonstrate the promising performance of ECDM, and we conduct an exhaustive ablation study to analyze its effectiveness. 
: 
 thermal sensors capitalize on thermal radiation and temperature differentials. This sensitivity renders them particularly proficient in detecting temperature-related entities, notably livings and vehicles, within obscured settings. The superiority of thermal vision motivates numerous studies [10, 12, 17] dedicated to thermal vision applications, particularly thermal object detection, and yield noteworthy enhancements in this domain. However, the efficacy of thermal vision applications remains notably curtailed by the paucity of available training samples. For example, the LLVIP dataset [16] only contains a mere 12,000 training thermal images, constituting only one-ninth of the visible samples contained within the COCO dataset [23] . The procurement of expansive training data and the meticulous labeling of precise annotations necessitate extensive human labor and substantial time investment. To address this challenge, extant studies mainly harness two methodologies for augmenting training datasets to facilitate deep model training: 3D synthesis and deep generative models. The 3D synthesis methods [3, 28] commence by generating a subset of 3D objects, followed by the application of a rudimentary thermal shader to render these objects, thereby engendering synthetic thermal images. Nevertheless, the images produced by the latest thermal sensor simulators still exhibit significant disparities when compared to those captured using real equipments. Recent studies in the domain of generative models, particularly within the realm of Generative Adversarial Networks (GANs) [15, 25] , serves as an impetus for the generation of training data for thermal object detection [18, 26] . However, existing GAN-based methods necessitate the availability of paired visible and thermal images for training deep models. The constraint often proves challenging to meet in practical contexts. This study delves into the applicability of the diffusion model to the task of generating thermally-aligned pixel-level images, obviating the need for supervisory guidance from paired visible-thermal counterparts. To this end, we propose an edge-guided adversarial condition diffusion model (ECDM) for thermal data generation. The basic idea of ECDM is learning the conditional probabilistic density of thermal images under the condition of the given visible image edge information. By incorporating the idea of adversarial learning to ease detrimental impacts stemming from extraneous and irrelevant edge details in the visible domain. As illustrated in Figure 1d , our ECDM can simultaneously reconstruct object shape and object thermal radiation characteristics while other methods are only good in one aspect. As shown in Figure 1b , GAN-based methods often tend to reconstruct object shapes but introduce irrelevant edges and abnormal details in thermal modality. As shown in Figure 1c , other Diffusion-based methods often tend to reconstruct object thermal radiation characteristics. In summary, the main contributions of ECDM are threefold:(1) ECDM engenders pixel-level thermally-aligned images through the generation process, bypassing the necessity for annotated visible-thermal pairs. This innovation augments the available datasets for thermal object detection by effectively generating thermal images from their visible counterparts. (2) We devise a conditional diffusion model to estimate the conditional probabilistic density of thermal images under the constraints of the given visible image edges. Furthermore, we develop an adversarial training strategy to filter out the extraneous edge information from the visible domain that should not appear in the thermal domain.(3) Extensive experiments on LLVIP demonstrate ECDM's superiority over existing state-of-the-art approaches in terms of image generation quality. The applicability of ECDM in generating training samples is also evaluated on the classical object detection task, wherein ECDM brings up to 7.1% mAP improvement for the detectors (as shown in Figure 2 ). 
 
 Appendix:  


Decision: The paper originally received mixed ratings of BA, BR, BR and BA respectively. The reviewers were generally appreciative of the innovative method and the novel TMAT strategy, promising results and effectiveness of the generated data for downstream tasks, potential of wider application beyond object detection, adequate experimental details etc. However, they also raised concerns on the lack of comparisons with more SOTA diffusion models, clarity of motivation and aim of the proposed work, missing performance comparisons between DPM solver and other sampling accelerators, missing details of sampling process employed and possibility of anti-facts in the generated images, missing important details in Figure 3 etc. The authors have provided a detailed rebuttal with an effort to clarify most of the concerns raised. The reviewers w9Da, CReY and AWBx comment that the rebuttal has been able to partially/fully clarify their concerns, whereas reviewer ZDWB still has concern regarding comparison with more SOTA methods. Finally, three out of the four reviewers have retained their original rating, whereas reviewer CReY has upgraded the rating from BR to BA. 

Upon careful consideration of the original reviews, authors' rebuttal and the post-rebuttal ratings (BA, BA, BR, BA), the AC feels that the merit in the contributions of the paper as well as its potential wide utility outweigh the remaining limitations and hence recommends acceptance. The authors are strongly encouraged to take all the reviewers' comments into account while preparing the camera-ready submission.

Reviews: review: paper_summary: paper_summary: This paper generates thermal images using edge-guided conditional diffusion model and two-stage modality adversarial training, in spirit combining diffusion models and GANs.
strength_weakness: strength_weakness: 1. The results are promising. Experimental results show that the proposed method is effective and brings evident gains.

2. Table 5 shows the effectiveness of the generated data for downstream tasks. 

3. The two-stage modality adversarial training is interesting.
limitations: limitations: 1. In the experiments, the proposed method should be compared to more state-of-the-art diffusion models. Only comparing to DDIM and BBDM can be a bit limited.

2. In Table 6, the gain margin of the proposed method is not very evident.

3. Figure 3 can be improved in terms of clarity of the illustration.

4. The edge-guided conditional diffusion model is not very novel.
paper_summary: paper_summary: This paper introduces a novel infrared data generation method based on diffusion models, and employs a Generative Adversarial Network adversarial training strategy to mitigate the issue of excessive domain disparity during the generation process. The generated infrared images from this method exhibit a superior recovery of thermal map information compared to existing generation techniques, and additionally, they enhance the performance of downstream perception tasks.
strength_weakness: strength_weakness: This paper introduces a novel infrared data generation method based on diffusion models, and employs a Generative Adversarial Network adversarial training strategy to mitigate the issue of excessive domain disparity during the generation process. The generated infrared images from this method exhibit a superior recovery of thermal map information compared to existing generation techniques, and additionally, they enhance the performance of downstream perception tasks.
limitations: limitations: 1. Firstly, I believe the motivation of the paper needs reconsideration. If the objective is to enhance perception tasks in adverse environments (low-light, rain, haze) where infrared imaging is necessitated, then the available visible image texture information would presumably be minimal. On the other hand, if the aim is to enhance the performance of downstream perception tasks under infrared vision algorithms, why not consider fusion the details of visible image texture information during generation? A generation approach akin to multimodal fusion could potentially bolster the performance of downstream tasks more effectively.
2. Why only utilize datasets of infrared and visible light under low-light conditions? There are now more diverse environments captured in datasets like M3FD and FMB, which could provide a broader basis for evaluation. And this kind of data set is more suitable for motivation.
3. Regarding the comparison with diffusion-based methods such as DDIM and BBDM, are these strategies trained using the proposed TMAT approach? Utilizing DDIM solely for accelerating sampling should not lead to significant performance degradation. Additionally, I am curious about the performance comparisons between DPM solver and other sampling accelerators.
4. This paper should give some visible results of detection to provide a clearer demonstration of the effectiveness of the proposed methods.
paper_summary: paper_summary: This article proposes an edge guided conditional diffusion model (ECDM) for generating pseudo thermal images aligned with visible light images at the pixel level to enhance the performance of thermal imaging algorithms in areas such as object detection. A two-stage Modal Adversarial Training (TMAT) strategy was introduced, which trains the model on hot edge images first, and then uses adversarial training to reduce the distribution difference between visible light and thermal imaging domains, thereby improving the quality of generated images. The superiority of ECDM in image generation quality has been verified through extensive experiments.
strength_weakness: strength_weakness: 1. The TMAT strategy gradually reduces the distribution difference between the generated domain and the target domain through phased training, which is a novel and effective idea.
2. The experimental design of the article is relatively sufficient and has achieved significant improvements in multiple indicators, verifying the superiority of ECDM in image generation quality.
3. The article has the potential for widespread application, as this method is not only suitable for object detection, but also has the potential to be applied to other scenarios that require thermal imaging data, such as thermal imaging super-resolution, thermal target tracking, etc.
limitations: limitations: 1. The article lacks comparison with professional SOTA papers, making it difficult to demonstrate its reliability. Pix2pix and cyclegan are very early articles, and recent gan-based articles such as LPTN and VSAIT are not specifically targeted for image translation in the fields of infrared thermal and visible light image conversion. This greatly undermines the superiority of the article and makes it difficult for practitioners in the infrared field to fully believe it.
2. The Figure 3 of the article lacks sufficient textual explanations or legends, and the lines and various modules lack explanations. Readers may need to put in more effort to understand each component in the figure and their relationships.
3. There may be an error in Figure 3 of the article. Formula (11) indicates that in order to further narrow the gap between the generated thermal image and the actual thermal image, the style consistency loss in Figure 3 is illustrated as the binomial between the actual thermal image after high-frequency sampling and the generated thermal image.
paper_summary: paper_summary: This paper investigates the generation of thermal modality data. The author proposes to utilize the edge information to bridge the domain gap between real vision images and thermal modality data, and designs an adversarial training strategy to filter out irrelevant edge information. The overall logic of the paper is smooth, and the presentation is clear. The model design is scientifically reasonable. The experiments are enough and effectively validate the effectiveness of the method design. The generated results show significant improvements compared to previous methods.
strength_weakness: strength_weakness: 1.	The paper's logic is smooth, and the expression is clear. The motivation is reasonable, providing a comprehensive analysis of the existing problems in the current field.
2.	The design demonstrates a certain level of innovation, effectively addressing the issues raised in the article. Additionally, it shows significant improvement compared to other works in this field.
3.	The experiments are adequate and complete, strongly demonstrating the effectiveness and reasonableness of the model design, effectively supporting the article's conclusions.
limitations: limitations: 1.	The authors propose a two-stage method, with the first stage conducted entirely on thermal modality data and the second stage incorporating real vision data through adversarial training. A potential issue is whether the training in the second stage might degrade the model obtained from the first stage. This point is not addressed in the paper, and it would be helpful if the authors could provide clarification on this matter.
2.	The paper primarily describes the training process but does not detail the sampling process. Therefore, I assume the sampling process follows the sampling way of the basic diffusion model, from random noise to images. Since there is no condition in the diffusion process to control generation, it raises the question of whether the content generated by the proposed method might include some anti-fact elements. It would be beneficial if the authors could provide more generated samples to illustrate this.