ThermalGaussian: Thermal 3D Gaussian Splatting
Authors: Rongfeng Lu
Hangyu Chen
Zunjie Zhu
Yuhang Qin
Ming Lu
Le zhang
Chenggang Yan
anke xue
Venue: ICLR 2025 Poster (2025)
Keywords: 3D reconstruction; Thermal fild reconstruction; 3D Computer Vision; Machine learning approaches;
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: The use of infrared thermal imaging generates new perspective images and realistic 3D reconstructions, improving both thermal and color image quality while significantly reducing memory requirements.

Abstract: Thermography is especially valuable for the military and other users of surveillance cameras. Some recent methods based on Neural Radiance Fields (NeRF) are proposed to reconstruct the thermal scenes in 3D from a set of thermal and RGB images. However, unlike NeRF, 3D Gaussian splatting (3DGS) prevails due to its rapid training and real-time rendering. In this work, we propose ThermalGaussian, the first thermal 3DGS approach capable of rendering high-quality images in RGB and thermal modalities. We first calibrate the RGB camera and the thermal camera to ensure that both modalities are accurately aligned. Subsequently, we use the registered images to learn the multimodal 3D Gaussians. To prevent the overfitting of any single modality, we introduce several multimodal regularization constraints. We also develop smoothing constraints tailored to the physical characteristics of the thermal modality.
Besides, we contribute a real-world dataset named RGBT-Scenes, captured by a hand-hold thermal-infrared camera, facilitating future research on thermal scene reconstruction. We conduct comprehensive experiments to show that ThermalGaussian achieves photorealistic rendering of thermal images and improves the rendering quality of RGB images. With the proposed multimodal regularization constraints, we also reduced the model's storage cost by 90\%. Our project page is at https://thermalgaussian.github.io/.

Published as a conference paper at ICLR 2025
THERMALGAUSSIAN! SPLATTING
THERMAL 3D GAUSSIAN
Rongfeng Lu 13.e.f Hangyu Chen Zunjie Zhu Yuhang Qin' Ming Lu Le Zhang' Chenggang Yan Anke Xue
'Hangzhou Dianzi University Intel Labs China Lishui Institute of Hangzhou Dianzi University
PS 77, MeMCISTAM
PENR:19.1 Mon. 39.36
PSNK 25 AN, Mcncl42MM
PSNR-25 2OMS
(a) RGB
(b) Thermal (c) ThermoNeRF
(d) 3DGS
(e) Ours
(f) Ours MR
Compared NeRF-based methods (Hassan et 2024 and methods that directly use thermal images for training 3DGS, our approach not only improves the thermal image rendering quality but also significantly reduces the model's storage through multimodal regularization (MR).
Figure 1:
ABSTRACT
Thermography is especially valuable for the military and other users of surveil- lance cameras. Some recent methods based on Neural Radiance Fields (NeRF) are proposed to reconstruct the thermal scenes in 3D from set of thermal and RGB images. However, unlike NeRF, 3D Gaussian splatting (3DGS) prevails due to its rapid training and real-time rendering. sian, the first thermal 3DGS approach capable of rendering high-quality images in RGB and thermal modalities. We first calibrate the RGB camera and the thermal camera ensure that both modalities use the registered images to learn the multimodal 3D Gaussians. To prevent the overfitting of any single modality, we introduce several multimodal regularization constraints. We also develop smoothing constraints tailored to the physical char- acteristics the thermal modality. Besides, we contribute real-world dataset named CGBBT-ccena captured by a hand-hold nhermal-infrared camera, facilitat- ing future research on thermal scene reconstruction. We conduct comprehensive experiments to show that ThermalGaussian achieves photorealistic rendering of thermal images and improves the rendering quality RGB images. With the pro- posed multimodal regularization constraints, we also reduced the model's storage cost by 90%. Our project page is at https://hhermalgassinnn github..o.....
this work, we propose ThermalGaus-
accurately aligned. Subsequently, we
INTRODUCTION
Thermal imaging is widely used in fields such military (e et al. 2021), healthcare (Lahiri al., 2012], industry (Glowacz), 2021), agriculture (Zhou et al. 2021). building inspection (El Masri & Rakha 2020), and search and rescue (Yeom) (024) because it converts temperature information an important physical modality not visible to the human eye-into interpretable images. 3D reconstruc- tion technology, which involves lifting multi-view 2D images into 3D scenes, is foundational for key technologies such as the metaverse, digital twins, autonomous driving, and robotics. Any image with valuable 2D applications can be lifted into 3D to view the captured scene from a new view and in greater detail. Thermal images are no exception (Abreu de Souza et al. 2023 Liu et al. 2024).
Previous 3D thermal scene reconstruction (Rangel et al. 2014 Zhao al.) 2017 Müller 2019 et al. 2023) involves a two-stage process. In the first stage, RGB images and traditional multi-viww
"Equal contribution
1 "ooresponnence fongfeng-1 1uhdu.. edu.en en, akxue@hdu.c edu.en

---

Published as a conference paper at ICLR 2025
ourtherof
Number Pointe
Poine 1184)
(a) RGB
(b) Thermal
(c) Mix (3 0.5)
(d) MSX
Figure 2: Top: camera poses and point cloud generated by SfM. Bottom: input images for SfM.
geometry methods (Newcombe et al.) 2011 are used to achieve 3D geometrie reconstruction. In the second stage, thermal images are mapped onto the reconstructed 3D scene. However, these methods not only fail to fully exploit thermal information but are also constrained by the limitations of traditional 3D reconstruction techniques, which impede their ability to render high high-quality images from view. This hinders the practical application of thermal reconstruction.
Neural Radiance Fields (NeRF) (Mildenhall et al., 2021) can render photorealistic images from a new view, thus revolutionizing novel-view synthesis and 3D reconstruction. Recently, several NeRF- based methods (Hassan et al.) 2024 Ye et al. (024) have been proposed to reconstruct thermal scenes in 3D using thermal images. However, NeRF's slow rendering speed and implicit scene representation limit its practical applications. In contrast, 3D Gaussian Splitting (3DGS) (Kerbl et 2023) not only maintains photorealistic rendering quality from new views but also significantly improves rendering speed. Additionally. the explicit 3D scene representation of 3DGS, akin to point clouds, facilitates integration with downstream tasks, establishing it as a leading approach in research. Building on 3DGS, we propose ThermalGaussinnn a multimodal Gaussian technique that renders high-quality RGB and thermal images from new views.
The absence of open-source datasets dedicated to thermal scene reconstruction significantly impedes progress this domain. Several researchers (Hassan et al., 2024 Ye et al. 2024) have recognized this issue and have released some datasets. However, these datasets suffer from problems such as lack color images registered with thermal images, inconsistencies thermal information from different views, and watermarked images. To address these issues, contribute a real-world dataset named RGBTT-Scenes.
Unlike RGB images, thermal images possess unique low-texture and ghosting characteristics (Bao eet al.) 2023) that hinder accurate camera pose estimation using Structure-from -ooor-roooooooon (SfM) (Schönberger & Frahm 2016), illustrated in Fig. [2. Consequently, thermal images cannot directly replace RGB images for running 3DGS. To address this issue, we first register the RGB and thermal images and then fuse them (Fig. 2b, use Multi-Spectral Dynamic Imaging (MSX)
(Abdullah 2023 (Fig. 2d) to localize the thermal image camera. Additionally, We design a thermal loss adapt the unique characteristics of thermal images.
Introducing new modality, such as thermal imaging, into 3D reconstruction should enable the model to understand the scene from a more comprehensive perspective. However, ThermoNeRF (Hassan et al. 2024) reduces the RGB rendering quality after implementing thermal reconstruction. In contrast, our method not only improves thermal rendering quality but also enhances RGB render- ing quality by dB. Furthermore, to prevent overfitting of any single modality during multimodal Gaussian training, we introduce a multimodal regularization coefficient. This approach significantly reduces model storage requirements and accelerates rendering speed. In summary, the main contri- butions as follows:
(1)We propose ThermalGaussian, the first multimodal 3DGS capable of simultaneously rendering photorealistic thermal and RGB images scene.
We propose series of strategies for multimodal Gaussian reconstruction, including multimodal initialization, three different thermal Gaussians, constraints specific to thermal modalities, and mul- timodal regularization.
2

---

Published as a conference paper at ICLR 2025
We) introduce view synthesis. The dataset consists of paired RGB and thermal images captured from multiple viewpoints across 10 different scenes.
new dataset designed for thermal 3D reconstruction and novel-
(4)Finally, experimental results show that our multimodal method not only improves the rendering quality of both thermal and RGB images but also reduces storage space by 90% compared training each modality separately, while also improving rendering speed.
RELATED WORK
2.1
THERMAL IMAGING AND 3D RECONSTRUCTION
All objects with temperatures above absolute zero emit energy in the form of electromagnetic waves, phenomenon known as thermal radiation. Through Planck's law, thermal imaging converts the wavelength and intensity of electromagnetic waves radiated from an object's surface into thermal information, which is then used to create images. Common thermal imaging devices operate in the mid-to-long-want wave infrared range. To intuitively display the temperature distribution in thermal imaging, pseudo-color rendering is often applied using a cool-to-warm color scheme. Originally, thermal imaging was developed for military purposes to enable visualization under extreme lighting conditions, such as nighttime or smoke. As costs have decreased in recent years, it has also been widely applied in fields of healthcare, industry, agriculture, building inspection, and on.
By capturing 2D images from various angles and applying 3D reconstruction, a 3D model can be created. Unlike fixed 2D viewpoints, the 3D model enables intuitive analysis from any perspec- tive and scale, allowing detailed exploration. The advent of KinectFusion (Newcombe et al. 2011) marked the beginning of an era of high-preciste
dense 3D reconstruction. Subsequent develop- ments (NieBner et al. 2013 Kähler et al.) 2015 Dai et al. 2017 Gong et al. 2021 Zhang et al.) 202T) have optimized 3D reconstruction in terms of accuracy, efficiency, and unconstrained camera movement. Some works (Rangel et al. 2014] Zhao et al. 2017 Müller 2019; Li et al. (023) have been made integrate thermal imaging with the aforementioned methods, leading to the develop- ment of thermal 3D reconstruction. However, these traditional multi-view geometry-based methods do not perform as well in rendering new views the more recent deep learning- based approaches.
With the rapid development of artificial intelligence Yan et al. 2020a Wang et al. 2024a), NeRF MMildenhall et al., 2021) has emerged as significant milestone the field of 3D reconstruction due to its impressive ability to render highly realistic images from a new view. Recently, ReRF (Hassan et al. 2024) and Thermal-NeRF (Ye et al.) 2024) have been proposed to reconstruct thermal scenes by combining set of thermal images with NeRF Although these approaches successfully generate images from new perspectives, they are limited by the slow rendering speed and implicit scene representation of NeRF, which hampers their practical application.
2.2
3DGS AND MULTIMODALITY
3DGS Kerbll et al. 2023 represent a revolutionary technology in the fields of 3D reconstruction. Distinct from methods like NeRF, 3DGS employs millions of explicit Gaussians, fundamentally al- tering its approach. This technology merges the advantages of neural network-based optimization with structured data representation, enabling photorealistic rendering from new views, significantly enhancing real-time rendering capabilities, and introducing the ability to manipulate and edit 3D scenes. These features make 3DGS highly compatible with broad range of downstream appli- cations, establishing it as the baseline for next-generation 3D reconstruction technologies (Chen) & Wang, (224)). Although 3DGS is constrained and trained using only RGB modality images, it ultimately generates millions of Gaussians, resembling point cloud. This characteristic makes 3DGS particularly suitable for multimodal fusion with other devices that directly capture scene point clouds, such as depth cameras and LiDAR. Studies (Matsuki et al. 2024, Yan et al.) [2224] ( Keetha ct al. 2024) have effectively integrated depth cameras to implement 3DGS-based simultaneous lo- calization and mapping. Studies (Li et al., 2024 Chung et al. 2024) have effectively combined depth images (Yan et al. 2020b) estimated from a pre-trained Monocular Depth estimation model (Bhat et al. 2023) with the RGB modality, resulting in improved rendering quality and more accu- rate geometric structures. 3D scenes contain not only RGB and geometric modalities but also other important modalities relevant various applications, such thermal, material, and pressure.
3

---

Published as a conference paper at ICLR 2025
Los
Mabiincal Lornd
Pewwart Paes
Malticodel Regabazzatios
Malimobl Maiimiaaliittta Amiiakann
Mallinodia Ruca
Readend
Dornity Comnal
Meni View Mutimodaa Soane
Matinoii Spirting
Figure 3: Caussal Overview, We simultaneously construct Gaussians for RGB and ther- mal modalities using the point cloud obtained from multimodal initialization. Each modality's 's Gaus- sians used render images in their respective modality. However, the losses from different modalities are combined to jointly constrain the optimization of both sets of Gaussians. Addition- ally, establish multimodal regularization based the number of Gaussians in each modality, which dynamically adjusts the training coefficients for both modalities.
3 METHOD
Fig. shows the overview of the proposed ThermalGaussian, which is based on the 3DGS (Kerbl al. 2023), aiming to extend its capability to simultancously render images of color and temper- ature. this section, we first briefly introduce the background of the 3DGS. Then, we provide a detailed description of our method's specific implementation details, including multimodal initial- ization, three types of multimodal thermal Gaussians, thermal loss, and multimodal regularization.
3.1 PRELIMINARY: 3D GAUSSIAN SPLATTING
3DGS (Kerbl et al. 20233 represents a 3D reconstruction scene using a large number of anisotropic 3D Gaussians. This representation not only provides differentiabilityy which offers advantages in learning-basse based methods, but also enables explicit spatial expression, enhancing the editability and controllability of 3D scenes. Furthermore, allows for rapid and efficient rasterization rendering through splatting. Initially. a set of unordered images of objects to be reconstructed is processed using SfM obtain the camera poses and sparse point clouds. 3DGS then initializes these sparse point clouds as the position of 3D Gaussian:
G(x) = caa 2==--)"
where represents the covariance matrix of the 3D Gaussian, and x denotes any point in the 3D scene. Σ is defined using scaling matrix S and rotation matrix R:
Σ= RSS R.
The 3D Gaussian G(æ)) is projected onto the imaging plane using the camera's intrinsic parameters, transforming it into a 2D Gaussian. Subsequently, the image is rendered through alpha-blending:
1-1
C = Σ CHaR II (1-ay (1-a))
KEN
where represents the queried pixel position, denotes the number of 2D Gaussians correspond- ing this pixel, denotes the opacity of each Gaussian and the color on each Gaussian is modeled spherical harmonics. All attributes of the 3D Gaussians are learnable and optimized directly in an end-to-end manner during training.
4

---

Published as a conference paper at ICLR 2025
(a) Special hoard
(h) Standard board
Figure 4: Different calibration boards for thermal Cameras.
3.2 MULTIMODAL INITIALIZATION
Previously, methods for calibration RGB and thermal images (Zhang et al., 2023) often involve designing specialized, non-standard metallic calibration boards with uniformly sized circular holes, as shown in Fig. 4a. The calibration relies on the temperature difference between the board and the background to compute thermal features, enabling calibration. However, the high complexity and stringent requirements for producing these calibration boards make them difficult to obtain and lack a universal standard. We find that standard chessboard pattern, as shown in Fig 4b, commonly used for RGB camera calibration, can effectively be used for calibrating both thermal and color cameras, with a mean reprojection error of less than 0.5 pixels. Initially, we heat the calibration board using devices like an infrared heater, black regions, absorbing heat faster due to their material properties, exhibit relatively higher temperatures. We capture simultaneous color and thermal images before thermal equilibrium, which occurs when two systems reach a balanced state with equal temperatures, halting heat flow. Subsequently, conventional camera calibration (Zhang 1999) performed.
Using the calibrated intrinsic parameters KRGB for the color camera, Km for the thermal camera, and the rotation R and translation from the temperature camera to the color camera, we computed the corresponding positions (um, eT.) the thermal image mapped to the registered positions the color image:
URGB URGB
K RGB
R
K
WTh UTh
+
As shown in Fig b)). directly using thermal images, which exhibit low texture and ghosting char- acteristics, makes it difficult to successfully run SfM (Schonberger & Frahm| 2016). Therefore, to obtain the thermal camera poses, tested three different multimodal StM strategies. The first uti- lizes registered high-texture RGB images directly for camera pose estimation. These poses serve simultaneously for both the RGB and thermal cameras. However, practical scenarios that require thermal scene reconstruction often occur under dim lighting conditions or in scenes lacking distinct color features. Therefore, relying solely on color images may impede the precise camera pose es- timation necessary for thermal scene reconstruction. The second approach, illustrated in Fig c), involves blending registered color and thermal images using the following formula:
Iiix = 3m + -1- -1-33)ARa B)BB
(5)
where in the above equation, 3 represents the opacity of the thermal image. This method produces blended images containing both rich color and thermal information, catering various practical applications of thermal scene reconstruction. The third strategy. depicted in Fig 2[d) d), maps high- frequency color variations from the color images onto the thermal images. This approach mitigates the lack of feature points caused by thermal images' low texture and ghosting characteristics.
3.3 THERMAL GAUSSIAN
We utilize three different multimodal training strategies to construct the thermal Gaussian.
Multimodal Fine- Fine-Tuning Gaussians (MFTG): Inspired by the fine-tuning approach used in large- scale models, our first multimodal training strategy is training a basic Gaussian with RGB images and then refining this Gaussian with thermal images to generate thermal Gaussian. This is a two- stage process. In the first stage, similar to 3DGS, we utilize multimodal camera poses and initial
5

---

Published as a conference paper at ICLR 2025
point clouds obtained from multimodal initialization inputs. The training is supervised using RGB images, with L1 combined with a D-SSIM term:
CRGB = (1 A)2, + ALD.SSIM
(6)
This stage enables us to render high-quality RGB modality images from a new view and establish basic 3D Gaussian with preliminary geometry. In the second stage, we fine-tum tune this basic Gaussian model with thermal images and multimodal camera poses obtained from initialization. Since the first stage constraints were based on texture-rich color images rather than thermal images, which results in a better geometry. Therefore, training on this geometry yields better results than training thermal Gaussian directly from the initial point cloud derived from multimodal initialization.
Multiple Single-Modal Gaussians (MSMG): The training of MFTG initially utilizes RGB modal information followed by thermal modal information. Although both modalities are employed, they are not used simultaneously. Since only thermal images were utilized for supervision in stage two, the information from the color modality in Fig. 3) we constrain the training with information from both color and thermal modalities si- multaneously. We train two single-modal Gaussians initialized by point clouds from multimodal initialization. The thermal Gaussian renders thermal images, while the RGB Gaussian renders RGB images. Subsequently, these rendered images of both modalities are compared separately with the ground truth of their respective inputs using loss functions:
not fully leveraged. Therefore, in MSMG (as shown
L LRGB + Lehermaal
(7)
The details Lihermat constraint will be elaborated below. Each Gaussian model is influenced not only by its corresponding input modality but also by others. Experimental results indicate that joint constraints across multiple modalities enhance the training outcomes for both color and ther- mal modalities. Moreover, since these modalities jointly optimize the entire scene from different perspectives, redundant points are pruned to some extent, reducing the number of points in the point cloud and lowering the model's storage requirements.
One Multi-Modal (DMM);:) OMMG extends MSMG by not only employing dual- modal loss constraints in Eq. (7) but also integrating multiple modalities onto single Gaussian. This integration ensures that information from diverse modalities is unified within a single geometric structure. Specifically, we construct a multimodal Gaussian comprising positional coordinates r, scaling matrix S, rotation matrix R. opacity a, spherical harmonics e for RGB representation, and spherical harmonics for thermal representation. RGB rendering is achieved using Formula 3, while thermal rendering follows the equation below:
( = Σ LoOK II (1 (1-a))
(8)
KEN
j=1
3.4 THERMAL LOSS & MULTIMODAL REGULARIZATION
The loss function for RGB modality images is directly given by Equation 6. The same loss func- tion can also be applied to thermal modality images. However, because thermal images exhibit unique low-texture and ghosting characteristics, we design specific thermal loss function to better accommodate these features.
The RGB modality may exhibit abrupt changes. However, because all objects above absolute zero continuously engage in heat transfer and thermal radiation, eventually reaching thermal equilibrium with their surroundings, significant abrupt changes typically not observed in thermal images. Additionally, most regions of objects in thermal equilibrium have similar temperatures, resulting in smoother thermal images. Therefore, we introduce smoothness term for regularization:
Esmooth
([T==,
Ti,5l + [T - Te,s))
4M
where Tiy represents rendered thermal values pixel position (i.j). M denotes the number of rendering pixels. Similarly to the color modality, we also incorporate C₁ and LP-SSIM Thus, our final temperature loss is:
Lihermal (1 --)Li ALSSMM + Asmoouh Camooth
(10)

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of our collected dataset with others.
Mode T RGB
Bimodal Calibration
Multiview Consistency
Content
Dataset
Richness
Thermal-NeR3
X1 1
Simple
ThermoNeRF
Moderate
Ours
1 v
V
V
Rich
where Asmooth is the coefficient of Lumoath
When training multiple modalities of the same object simultaneously, is often undesirable for one modality to dominate at the expense of others. Therefore, a regularization strategy is needed to dynamically adjust the weight of cach modality's loss during training. the training of MSMG, observe that the weight coefficients of modality align linearly with the Gaussian it ultimately generates. higher weight for a modality results in more Gaussian generated by that modality, while fewer Gaussian generated by another modality. Hence, we design multi-modal regularization coefficients y based on the number of Gaussian generated by each modality during training.
Nihermal
(11)
Nthermal + NRGB
where Nihermal represents the number of Gaussian for the thermal modality during training. When the number of one modality's Gaussian increases, we increase the training weight of the other modal- ity. This dynamic balancing of weights ultimately prevents overfitting to any single modality, The final design of this loss is: L LLRB (1 7) Lihermal
(12)
SELF-COLLECTED THERAML DATASET
We introduce dataset, named RGBT-Scenes, which consists of aligned collections of thermal and RGB images captured from various viewpoints of scene. The images are collected using the commercial-gree -gade handheld -herral-infrared camera FLIR E6 PRO (Teledyne FLIR 2024). which can simultaneously capture RGB and thermal images. The basic specifications of this camera in- clude a resolution of 240x 180, a field of view of 339x25 temperature range from -20 550CC, and temperature accuracy of +2% of the reading. Our dataset includes over 1 1,000 RGB and thermal images from different scenes. These scenes encompass both indoor and outdoor environments, various object sizes (from large structures everyday items), different temperature variations (rang- ing from 300 difference to a 4 difference), and include both 360-degree and forward-facing scenarios. We provide the raw images captured by the thermal camera, as well as the RGB images, thermal images, MSX images, and camera pose data. In Table we compare our dataset with those from concurrent works, Thermal-NeRF (Ye et al. 2024 and ThermoNeRR (Hassan et al.) 2024). Our dataset includes both RGB and thermal images and applies multimodal calibration methods align these images. The images used for calibration will also be made available. Compared to ThermoNeRF, our dataset ensures consistent thermal measurements across views and encompasses richer variety of scenes. Detailed descriptions of each scene provided in the supplementary.
EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
Our method is improvement upon the 3DGS framework, with all experimental settings (e.g., A) remaining consistent with the reference 3DGS. The specifie hyperparameter Asmooth is set to 0.6. Each comparative experiment trained for 30K iterations. All experiments conducted a single NVIDIA 3090 GPU. The resolution of the rendered RGB images and thermal images is 640x480.
7

---

Published as a conference paper at ICLR 2025
Quantitative evaluation of thermal image using our method compared to previous work from test views. "x" indicates failure to localize using only thermal images in the scene, making it impossible to success with 3DGS. 3DGS+MI represents the results obtained by directly training 3DGS after Multimodal Initialization.
Table 2:
Metric
Method
Dimsum Daily Stuff
Ebike
Road Block
Truck
Rotary Kiln
Building Tron
Parterre
Land Scape
Avg.
Ingot
3DGS
25.38
20.97
23.79
23.75
ThermoNeRF
24.27
17.34
19.70
17.17
23.53
26.40
23.31
22.97
17.88
18.79
21.13
PSNR
3DGS+MI
26.35
18.77
20.89
26.75
26.17
26.59
25.76
29.57
22.09
20.17
24.31
OutSMPrG
26.94
20.52
22.51
24.96
25.02
26.91
26.11
30.41
23.55
20.03
24.70
OuTN
26.73
21.35
23.23
26.52
26.27
27.15
26.83
30.06
25.01
20.61
25.38
OuTSOMAre:
26.46
22.28
23.31
27.17
25.88
26.33
26.72
29.86
26.16
22.27
25.64
3DGS
0.860
X
0.717
0.872
0.810
ThermoNeRF
0.747
0.759
0.694
0.781
0.750
0.916
0,804
0.717
0.709
0.774
0.765
SSIM
3DGS+MI
0.889
0.789
0.806
0.917
0.872
0,922
0,872
0.887
0.843
0.794
0,859
OUrS
0.890
0.798
0.845
0.906
0.880
0,920
0,886
0.895
0.859
0.808
0.869
OuISMSMG
0.891
0.829
0.857
0,909
0.879
0.926
0.897
0.898
0.860
0.832
0.878
OurSOMaN
0.886
0.835
0.862
0.919
0.874
0.922
0.888
0.896
0.883
0.850
0.882
3DGS
0.157
X
0,281
0.193
0.299
ThermoNeRF
0.312
0.494
0.290
0.293
0.291
0.170
0.234
0.152
0.309
0.264
0.280
LPIPS
3DGS+MI
0.124
0.274
0.313
0.204
0.139
0.125
0.211
0.093
0.252
0.328
0.206
OTT
0,121
0.258
0.235
0,210
0.133
0.129
0.199
0.091
0.232
0.317
0.192
0.124
0.208
0.220
0.213
0.133
0.130
0.189
0.086
0.227
0.293
0.182
OuTSOMAN
0.129
0.210
0.203
0.198
0.136
0.124
0.177
0.091
0.181
0.248
0.170
5.2 THERMAL VIEW SYNTHESIS
Similar to 3DGS, we employ image quality assessment metrics including Peak Signal-to-Noise Ratio (PSNR) (Hore & Ziou| 2010 Wang et al. 2024b), Structural Similarity Metric (SSIM) (Wang et al.) 2004.. and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang al. 2018) to evaluate the quality of reconstructed thermal and RGB images from new views.
As shown in Table 2 even in scenes with pronounced thermal variations. specifically targeting low- texture thermal characteristics, direct application of thermal data proves challenging for 3DGS. In very few successful cases, inadequate precision in thermal camera positioning has compromised the quality of thermal reconstructions. 3DGS+MI denotes training the original 3DGS using thermal images instead of RGB images after obtaining accurate thermal poses through our multimodal ini- tialization. Compared to 3DGS, 3DGS+MI adapts to wider range of scenarios and achieves higher reconstruction quality. Given the higher reconstruction quality 3DGS (Kerbl et al. 2023 com-
RGB GT
(b) Thermal GT (c) ThermoNerf
(d) 3DGS
(c) OutsoMmG
(f) Ours (MSX)
Figure 5: We present qualitative thermal image comparisons between our method, previous ap- proaches (Hassan et al. 2024 Kerbl et al. 2023), and the corresponding ground truth images from test views. We also show the training results of the MSX images, which easier to apply.
8

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative evaluation RGB image using our method compared to 3DGS.
Metric
Method
Dimsum Daily Stuff
Bbike
Road Block
Truck
Rotary Kiln
Building
Iron Ingot
Parteme
Land Scape
Avg.
3DGS
23.91
26.77
27.
22.30
20.79
20.95
23.96
24.91
20.20
23,20
PSNR
ThermoNeRF
19.74
16.79
17.75
18.32
18.77
18.89
17.12
15.07
23.13
19.13
18,46
24.42
21.71
27.34
28.22
23.57
22.23
23.08
25.69
25.57
20.91
24.27
OuG
24.38
21.76
26.85
28.12
24.17
23.14
24.19
24.55
25.48
21.71
24.34
3DGS
0.847
0.748
0.901
0.910
0.807
0.772
0.872
0.859
0.820
SSIM
ThermoNeRF
0,688
0.639
0.540
0.619
0.688
0.600
0.460
0.293
0.756
0.583
0.586
0.858
0.793
0.917
0.916
0.833
0.811
0.844
0.891
0.874
0.715
0.845
OuTSOMMG
0.858
0.797
0.905
0.920
0.840
0.822
0.849
0.884
0.855
0.739
0.847
3DGS
0.194
0.299
0.171
0.201
0.232
0.217
0.228
0.188
0.183
0,280
0.219
LPIPS
ThermoNeRF
0.228
0.465
0.244
0.548
0.311
0.207
0.291
0.301
0.167
0.275
0,303
OuG
0.194
0.262
0.156
0.221
0.217
0.190
0.168
0.172
0.184
0.275
0.204
OuTSOMAG
0.194
0.253
0.169
0.199
0.211
0.184
0.170
0.186
0.195
0.268
0,203
pared to NerfStudio (Tancik al. 2023). 3DGS+MI and method naturally outperforms Ther- moNeRF. Our three thermal Gaussian methods outperform 3DGS+MI across all scenes in PSNR, SSIM, and LPIPS. Among them, MU shows an average PSNR improvement of 1.3 dB. As shown in Fig. 5 our method's i qualitative rendering of thermal images is clearly superior. Addi- tionally, as depicted in Fig. 5t. we enhance thermal image readability by training with MSX images using thermal Gaussian. This hierarchical and easily recognizable thermal Gaussian further pro- motes the application of thermal scene reconstruction. We provide more qualitative comparison results in the supplementary materials.
5.3 RGB VIEW SYNTIEESIS
Our method not only achieves high-quality thermal image rendering but also significantly enhances RGB image rendering quality. As shown quantitatively Table our multimodal constraints im- prove RGB rendering quality in nearly all scenarios, with an average PSNR improvement of 1.1 dB compared the original 3DGS. This improvement is particularly evident in scenarios where the RGB modality struggles to identify the environment, while the thermal modality can recognize it clearly. shown in the top of 6 where distinguishing between foreground and background is challenging in the RGB modality but straightforward in the thermal modality due to temperature differences, constraints from the thermal modality aid in the accurate learning of the RGB modality. Additionally, depicted in the bottom of Fig 6 the assistance from thermal images enables accu- rate color rendering in low-light scenes for the RGB modality. Our results demonstrate that, under multimodal constraints, when the other modality to enhance the model's understanding of the scene, thus facilitating the correct learning of the failing modality. This enables method to advance 3D) reconstruction in low-light scenes and enhances the robustness of 3D reconstruction techniques to some extent.
modality fails, our approach leverages accurate information from
3DGS(RGB)
(b) MG
(c) RGB GT
(d) Thermal GT
Figure 6: We present qualitative RGB image comparisons between method and 3DGS.

---

Published as a conference paper at ICLR 2025
Table 4: Ablation Study, We conducted ablation experiments by gradually adding each compo- nent to the baseline 3DGS model. We then performed a comprehensive comparison across various dimensions, including rendering capability, the quality of rendered color and thermal images, train- ing time, model memory usage, and the number of Gaussians. "-" indicates that the model lacks the specified capability or metric, "x" denotes reconstruction failure. Since multimodal regularization relies on the Gaussians from multiple modalities, it only applies to OurS
Mode
Thernal
RGB
Methods
Train
FPS
Mem.
T/RGB
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
3DGS
"/
X
X
x
23.20
0.820
0.219
507s
231
159MB
+MI
41- - 24.31
0.859
0.206
367s
277
65MB
Cul+Ch
41-
24.65
0.867
0.198
603s
292
61MB
OUTSMFTG
81-
24.70
0.871
0.191
491s
51MB
OUSSMSAG
25.38
0.883
0.180
24.27
0.845
0.204
873s
330/
18MB+66MB
OUSOOMI
VIV
25.64
0.883
0.169
24.34
0.800
0.203
838s
271
/ 242
136MB
OUSSMSMG
K/V
25.09
0.880
0. 189
24.21
0.840
0.235
760s
390
420
9MB+9MM
23.0
7.10
Number of Gauzeisai 467K
Number Gauzsian 25K
(a) MR (?) vs. Fixed coefficient
(b) Gaussian distributions. Left: 3DGS; Right: MURS
Figure 7: Effectiveness the multimodal regularization term.
5.4
ABLATION STUDY
We separate different contributions and algorithm choices to test their effectiveness. As shown in Table and Table 4 after incorporating multimodal initialization, allows 3DGS to achieve thermal reconstruction across various environments. Our multimodal thermal Gaussian models, MSMG and OMMG, only render both thermal and RGB images simultaneously but also improve rendering quality for both modalities in all scenes, with an average increase of over 1.2 dB. We also observed that multimodal constraints mitigate the generation of excessive redundant Gaussians. Later, we introduced regularization term to dynamically adjust the coefficients of both modalities. As shown Table directly training RGB modality Gaussians with 3DGS results in an average storage requirement of 159 MB. On the other hand, directly training thermal Gaussians with MI requires an average 65 MB. The RGB Gaussians for MSMG+MR average only 9 MB in storage, with thermal Gaussians averaging the same. Our method requires only 8%
9-9 0.08) of the storage space (19++63
compared to directly using 3DGS. Due to the reduction in the number of Gaussians, the rendering speed has also significantly increased. Additionally, the rendering quality for both modalities has also improved. MFTG, MSMG+MR, and OMMG excel in different aspects: training speed, storage efficiency, and rendering quality. Fig a, we compare our multimodal regularization with manually adjusting the thermal constraint coefficients in truck scene. The comparison shows that our multimodal regularization approach reduces storage space for both RGB and thermal modalities while maintaining high image quality. In Fig 7 visually present the Gaussian distributions of the original 3DGS method and our method with multimodal regularization.
CONCLUSION AND FUTURE WORK
We are the first to implement thermal reconstruction based on 3DGS. We not only achieve simulta- neous rendering of thermal and RGB images but also significantly improve the rendering quality of both color and thermal images. Additionally, we greatly reduce the model's storage requirements. In the appendix, we discuss the limitations of this work and potential directions for future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Key Project of National Nature Science Foundation of China (U22A2047), the "Pioneer" and "Leading Goose" R&D Program of Zhejiang Province(2233010044, 2024C01107, 2023C01030, 2023C03012), the National Nature Science Foundation of China (62301198, 62371173).
REFERENCES
Sarwin Abdullah. Low-energy radar for handheld ir-cameras: Using mmwave radar as complement for multi spectral dynamic imaging in handheld cameras, 2023.
Mauren Abreu de Souza, Daoana Carolaine Alka Cordeiro, Jonathan de Oliveira, Mateus Ferro An- tunes de Oliveira, and Beatriz Leandro Bonafini. 3d multi-modality medical imaging: Combining anatomical and infrared thermal images for 3d reconstruction. Sensors, 23(3):1610, 2023.
Fanglin Bao, Xueji Wang. Shree Hari Sureshbabu, Gautam Sreekumar, Liping Yang, Vaneet Ag- garwal, Vishnu Boddeti, and Zubin Jacob. Heat-assisted detection and ranging. Nature, 619 (7971):773-745) 2023.
Shariq Farooq Bhat. Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 2023.
Guikun Chen and Wenguan Wang.
survey on 3d gaussian splatting. arXiv preprint
arXiv:2401.03890, 2024.
Jaeyoung Chung, Jeongtaek Oh, and Kyoung Mu Lee. Depth-regularized optimization for 3d gaus- sian splatting few-shot images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 1-1-820, 2024.
Angela Dai. Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundle- fusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG), 1,
Yasser El Masri and Tarek Rakha. A scoping review of non- dentrestructive testing (ndt) techniques in building performance diagnostic inspections. Construction and Building Materials, 205:12, 2020.
Adam Glowacz. Fault diagnosis of electric impact drills using thermal imaging. Measurement, 171: 108815, 2021.
Bingjian Gong, Zunjie Zhu, Chenggang Yan, Zhiguo Shi, and Feng Xu. Planefusion: Real-time indoor scene reconstruction with planar prior. IEEE Transactions on Visualization and Computer Graphics, 2021.
Mariam Hassan, Florent Forest, Olga Fink, and Malcolm Mielle. Thermonerf: Multimodal neural radiance fields for thermal novel view synthesis. arXiv preprint arXiv:2403.12154 2024.
Yunze He, Baoyuan Deng, Hongjin Wang, Liang Cheng, Ke Zhou, Siyuan Cai, and Francesco Ciampa. Infrared machine vision and infrared thermography with deep learning: A review. In- frared physics & technology, 103:103754, 2021.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp. 2366-2361 IEEE, 2010.
O. Kähler, V. Prisacariu, C. Y. Ren, X. Sun, P. H. S Torr, and D. W. M.Mury. Very High Frame Rate Volumetric Integration of Depth Images on Mobile Device. IEEE Transactions on Visualization and Computer Graphics (Proceedings International Symposium Mixed and Augmented Reality 2015, 22(1 2015.
Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat track & map 3d gaussians for dense rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21357-21366, 2024.
11

---

Published as a conference paper at ICLR 2025
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
Barid Baran Lahiri, Subramaniam Bagavathiappan, Jayakumar, and John Philip. Medical ap- plications of infrared thermography: review. Infrared physics & technology, 55(4))221-25) 2012.
Huaizhou Li, Shuaijun Wang, Zhenpeng Bai, Hong Wang, Sen Li, and Shupei Wen. Research on 3d reconstruction of binocular vision based on thermal infrared. Sensors, 23(1))77372, 2023.
Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimiz- ing sparse-view 3d gaussian radiance fields with global-local depth normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20775-20785, 2024.
Yuxiang Liu, Rouwan Wu, Shen Yan, Xiaoya Cheng, Juelin Zhu, Yu Liu, and Maojun Zhang. Atloc: Aerial thermal images localization via view synthesis. IEEE Transactions on Geoscience and Remote Sensing, 2024.
Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 18039-18048, 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Antonio Rafael Ordóñez Müller. Close range 3D thermography: real-time reconstruction of high fidelity 3D thermograms, volume 7, kassel university press GmbH, 2019.
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on, pp. 127-136. IEEE, 2011.
Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Mare Stamminger. Real-time 3d recon- struction at scale using voxel hashing. ACM Transactions on Graphics (ToG), 32(6):1-11, 2013.
Johannes Rangel, Samuel Soldan, and A Kroll. 3d thermal imaging: Fusion of thermography and depth cameras. In International Conference on Quantitative InfraRed Thermography, volume 3, 2014.
Johannes L. Schonberger and Jan-Michael Frahm. Stractureffrom-matiin revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113, 2016.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-frome frotuurrrroommooiion revisited. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016.
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Nerfstudio: A modular frame- work for neural radiance field development. ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-12, 2023.
Teledyne FLIR.
FLIR E6 Pro Thermal Imaging Camera, 2024.
URL https:
7/ /www. flir. cOm/ products/estteeeeee http:p//eoeeaaaaoooooooooooooooo
secment=solution
Tingyu Wang, Zhedong Zheng, Yaoqi Sun, Chenggang Yan, Yi Yang, and Tat-Seng Chua. Multiple- environment self-adaptive network for aerial-view geo-localization. Pattern Recognition, 152: 110363, 2024a. doi: https://doi.org//
Tingyu Wang, Zhedong Zheng, Zunjie Zhu, Yaoqi Sun, Chenggang Yan, and Yi Yang. Learning cross-view cco.localization embeddings via dynamic weighted decorrelation regularization. IEEE Transactions on Geoscience and Remote Sensing, 62:1-12, 2024b. doi: 10. 109//TTRSSS20024 3491757.
12

---

Published as a conference paper at ICLR 2025
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4))600- 612, 2004.
Chenggang Yan, Biao Gong, Yuxuan Wei, and Yue Gao. Deep multi-view enhancement hashing for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(4): 1445-1451, 2020a.
Chenggang Yan, Zhisheng Li, Yongbing Zhang, Yutao Liu, Xiangyang Ji, and Yongdong Zhang. Depth image denoising using nuclear norm and learning graph model. ACM Transactions Multimedia Computing, Communications, and Applications (TOMM), 16(4):1-17, 2020b.
Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li. Gs-slam: Dense visual slam with 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19595-19604, 2024.
Tianxiang Ye, Qi Wu, Junyuan Deng, Guoqing Liu, Liu Liu, Songpengcheng Xia, Liang Pang, Wenxian Yu, and Ling Pei. Thermal-nerf: Neural radiance fields from an infrared camera. arXiv preprint arXiv:2403. 10340, 2024.
Scokwon Yeom. Thermal image tracking for search and rescue missions with drone. Drones, 8 (2):53, 2024.
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, and Kai Xu. Rosefusion: random optimization for online dense reconstruction under fast camera motion. ACM Transactions Graphics (TOG), 40(4):1-17, 2021.
Jun Zhang, Yiyao Liu, Mingxing Wen, Yufeng Yue, Haoyuan Zhang, and Danwei Wang. L 2 2 calib: Automatic and unified extrinsic calibration toolbox for different 3d lidar, visual camera and thermal camera. In 2023 IEEE Intelligent Vehicles Symposium (IV), pp. 1-7. IEEF, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Zhengyou Zhang. Flexible camera calibration by viewing a plane from unknown orientations. In Proceedings of the seventh ieee international conference on computer vision, volume 1, pp. 666- 673. Ieee, 1999.
Shibo Zhao, Zheng Fang, and Shiguang Wen. real-time handheld 3d temperature field recon- struction system. In 2017 IEEE 7th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pp. 289-22922999 IEEE, 2017.
Zheng Zhou, Yaqoob Majeed, Geraldine Diverres Naranjo, and Elena MT Gambacorta. Assessment for crop water stress with infrared thermal imagery in precision agriculture: A review and future prospects for deep learning applications. Computers and Electronics in Agriculture, 182: 106019, 2021.
13


Reviews:
[{"review_id": "BmQYINuY7Z", "reviewer": "Reviewer_B1kF", "summary": "This paper introduce ThermalGaussian, a multimodal Gaussian technique that renders high-quality RGB & thermal images from new views.\nAlso they introduce a new real-world dataset named RGBT-Scenes, to address the issue of existing problems.\nAuthors propose multimodal regularization to prevent overfitting and show that ThermalGaussian outperforms previous methods by enhancing rendering quality, reducing storage requirements, and improving efficiency.", "questions": "- Can the authors clarify the choice of hyperparameters for multimodal regularization? Are there general guidelines for adjusting them?\n\n- How does the model handle extreme cases, such as highly reflective surfaces or objects at temperatures that differ significantly from ambient conditions?\n\n- Could the RGBT-Scenes dataset be expanded to include data for real-time applications, like moving objects or dynamic scenes?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The proposed approach is novel, especially in integrating thermal and RGB images in 3D reconstruction via Gaussian splatting. And the experimental results are clear, and include diverse evalutaion metrics and ablation studies to validate performance of this methods.\n\nAlso, introducing the RGBT-Scenes dataset is a valuable contribution that can serve as a benchmark for future work in thermal 3D reconstruction.", "weaknesses": "The process of multimodal initialization and the specific configurations for the thermal Gaussian construction are intricate, making the approach challenging to implement without in-depth knowledge. And, although the RGBT-Scenes dataset is comprehensive, details regarding environmental diversity and lighting conditions could strengthen its applicability across broader contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g7p9a1SscN", "reviewer": "Reviewer_teJM", "summary": "This paper proposes a method based on 3DGS that jointly learns multi-modal scene representations of RGB and thermal images. In fact, the authors present and evaluate three different strategies to incorporate thermal images into 3DGS: MFTG (based on fine-tuning), MSMG (leverages multi-task learning), and OMMG—the latter yields a single Gaussian, which is achieved by extending 3DGS with spherical harmonics to represent thermal data. Moreover, a regularization term is proposed that accounts for the smooth nature of thermal images, and a new weight scheduling strategy prevents the proposed MSMG from overfitting to a single modality. Based on a newly collected RGB+thermal dataset consisting of ten scenes, the authors could successfully show that their method outperforms 3DGS baselines and a recent NeRF-based method in terms of RGB and thermal rendering quality while requiring less memory.", "questions": "- What does „Mix50“ in Figure 2 stand for? Is Eq. (5) applied with $\\beta=0.5$? If so, please leave a note in the caption of Figure 2.\n\n- How sensitive is $\\lambda_{smooth}$? Would it be possible to provide a small ablation study?\n\n- How was $\\lambda$ in Eq. (6) and Eq. (10) set? Also, are these two lambdas the same?\n\n- As far as I understand, MSMG uses the cost function described in Eq. (7), i.e., without multimodal regularization. So, whenever \"Ours_MSMG\" is used in Tables 2 and 3, it refers to MSMG method trained without multimodal regularization. This is in accordance with Table 4, where authors add a dedicated row for \"Ours_MSMG+MR\" which would not be necessary if MSMG uses MR per default (in this case, I would rather expect a row with \"Ours_MSMG w\\o MR\" in Table 4). As this would mean that MR has never been used except in the ablation study, I would guess that per default, Eq. (12) has been used to train MSMG. Please make this more clear (e.g., by replacing Eq. (7) with Eq. (12) and referring to Section 3.4. on how to choose $\\gamma$).\n\n- What cost function is used to train OMMG? Is it Eq. (7)? If so, please move it to the respective paragraph (starting L296).\n\n- In Figures 5 and 6, which strategy does \"Ours\" refer to?\n\n- To be honest, it is quite surprising to me that a conventional chessboard pattern printed on normal paper produces such highly contrasting thermal images. There are numerous works (see, e.g., [6] and references therein) that try to design special calibration objects for thermal cameras. I wonder why all these papers exist when a simple chessboard pattern printed on paper will also work. What are the physical principles behind why a simple chessboard pattern printed on paper works, and how does it compare to more complex calibration objects in terms of accuracy and ease of use?\n\nReferences:\n\n[6] Issac N. Swamidoss et al. Systematic approach for thermal imaging camera calibration for machine vision applications. Optik 247, 2021.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The topic is interesting and of importance to the community, as demonstrated by the large body of recent work [1-5].\n\n- The paper is well-written and easy to follow. Experiments are well described and cared out thoroughly. The ablation study nicely supports all design decisions. All in all,  I don’t see any obvious flaws.\n\n- This is (to the best of my knowledge) the first paper to use some kind of scheduling to dynamically adjust the weight that balances the RGB and thermal loss in Eq. (12). The proposed strategy seems plausible to me, and, according to the experimental evaluation, significantly reduces the memory footprint while only slightly decreasing image quality. Due to the reduced number of Gaussians, this strategy also increases rendering speed.\n\n- Compared to prior work [1,3] (but similar to [5]), the method presented in this work is actually capable of *improving* the RGB rendering quality, especially in low-light conditions.\n\n- The paper proposes and evaluates not only a single method but three different strategies to incorporate thermal images into 3DGS, thus can be seen as the 3DGS-based pendant to [1].\n\n- I appreciate the newly collected dataset. Although small-scaled (like all other publicly available RGB+thermal datasets), it is diverse, contains forward-facing and 360-degree scenes, and encompasses indoor and outdoor environments. This is very helpful to the community.\n\nReferences:\n\n[1] Mert Özer et al. Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging. ECCVW, 2024.\n\n[2] Yvette Y. Lin et al. ThermalNeRF: Thermal Radiance Fields. ICCP, 2024.\n\n[3] Miriam Hassan et al. ThermoNeRF: Multimodal Neural Radiance Fields for Thermal Novel View Synthesis. arXiv, 2024.\n\n[4] Tianxiang Ye et al. Thermal-NeRF: Neural Radiance Fields from an Infrared Camera. arXiv, 2024.\n\n[5] Jiacong Xu et al. Leveraging Thermal Modality to Enhance Reconstruction in Low-Light Conditions. ECCV, 2024.", "weaknesses": "- I am missing citations (and discussions) of recent related works [1,2,5] in the introduction and related work sections.\n\n- I would love to see empirical evidence for the statement in L337: \"we observe that the weight coefficients of a modality align linearly with the Gaussian it ultimately generates\". This is a quite interesting finding that would be much better supported with data.\n\n- Authors claim multiple times (for example in L028, L104, and L536) that they are the first to use 3DGS to model RGB and thermal images. This statement is actually not true and should be revised. The authors of [5] already presented a 3DGS-based method that incorporates RGB and thermal data (see Section 4.5 of the main paper and Section 4 of supplementary).\n\n- In Table 1, the following open-source multi-view RGB+thermal datasets are missing: ThermalMix proposed in [1], the datasets presented in [2], and MVTV proposed in [5]. Why is the proposed dataset better than these datasets? Also, it would be good to provide specific criteria for how multi-view consistency and content richness is evaluated across datasets. This would help clarify the comparison and justify any claims about the proposed dataset's advantages.\n\n- With the high number of recently published RGB+thermal datasets, I think it might make sense to evaluate the introduced method on at least one of these datasets, also to mitigate potential dataset bias. \n\n- In addition, the proposed method could be compared to some more of the recently published NeRF-based methods. There is code available for [2].\n\n- In L097 authors mention that ThermoNeRF [3] reduces RGB rendering quality, while the proposed method improves it (according to Table 3, an average increase of almost 5% in PSNR). I think that is a huge plus for the proposed method; however, this argument would be much stronger if authors would add comparisons with NeRF-based methods to Table 3.\n\n- I would love to see a side-by-side comparison of thermal renderings with and without the smoothness term.\n\n- Although the proposed weight scheduling used in MSMG yields lower storage requirements, the average rendering quality for RGB and thermal data is worse than for OMMG (see Tables 2 and 3; difference higher for thermal renderings, drop of about 1% in PSNR). So, if one wants to maximize rendering quality and therefore uses OMMG, the benefit of having lower memory requirements is gone (see Table 4; OMMG requires about 60% of the storage space of 3DGS in contrast to 8% for MSMG).\n\n- In Section 3.2, three different strategies to obtain camera poses for thermal images are mentioned. What strategy is actually used for the experiments? Also, what strategy performs best? A quantitative comparison would be really helpful, also because MSX seems to perform best (just by looking at Figure 2). The problem I see is that MSX is patented by FLIR, as such it is not available to other cameras. Hence, the practical use of the third strategy is highly limited, and it would be good to know which strategy could be used instead. Do the other strategies in any form make use of MSX images as well?\n\nReferences:\n\nSee above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lFyNqJCeOd", "reviewer": "Reviewer_RJXA", "summary": "This paper proposes an enhanced method that extends the 3D Gaussian Splatting (3DGS) technique through the ThermalGaussian model, allowing for joint learning and rendering of RGB and thermal data. By learning thermal images together with RGB data in a multimodal framework, the approach aims to improve 3D scene reconstruction capabilities across fields where thermal imaging is valuable, such as in military, industrial, and medical applications. The paper presents initialization strategies for aligning and merging RGB and thermal data, a loss function tailored to the characteristics of thermal imaging, and multimodal regularization techniques to prevent overfitting and optimize storage. Additionally, a new dataset, RGBT-Scenes, is introduced, providing research data that includes both RGB and thermal images.", "questions": "1. Applicability to a Broader Range of Hyperspectral Imaging\n\nWhile the RGBT images presented involve the addition of a single channel, they can broadly be considered a form of hyperspectral imaging, as they incorporate the infrared spectrum. Could the authors discuss the potential for their multimodal approach to be applied to a broader range of hyperspectral imaging? Additionally, I am interested in the authors' views on whether this approach could be extended to studies involving multi-wavelength, multichannel data, and if it could serve as a basis for hyperspectral data analysis.\n\n2. Requirement for Well-Aligned RGB and Thermal Data and Practical Field Applicability \n\nThe proposed method relies on the availability of well-aligned RGB and thermal data, which may limit practical field applications. In cases where precise data alignment is challenging or unfeasible in real-world scenarios, are there any methods the authors could suggest to mitigate this limitation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Effective Multimodal Learning and Loss Function Design\n\nThe paper introduces an initialization strategy for aligning RGB and thermal data, along with a loss function tailored to thermal characteristics, allowing these modalities to learn complementarily. This approach enables each modality to mitigate the limitations of the other, achieving optimal reconstruction performance​.\n\n2. Model Efficiency and Storage Optimization\n\nBy applying multimodal regularization to prevent overfitting and reduce redundant Gaussians, the model reduces storage requirements by approximately 90% compared to conventional methods. This enables high-quality 3D reconstruction while also enhancing storage efficiency and rendering speed​.\n\n2. Contribution of the RGBT-Scenes Dataset to the Research Community\n\nThe RGBT-Scenes dataset introduced in this paper provides paired RGB and thermal images across diverse indoor and outdoor scenes, offering a valuable resource for thermal-based 3D reconstruction research. This dataset provides a foundation for researchers to test and expand upon the proposed model, thereby promoting advancements in the field​.", "weaknesses": "1. Lack of Thermal Data Description\n\nWhile the provision of a dataset is one of the significant contributions of this paper, it lacks sufficient information for readers to understand the characteristics of thermal data. Including details about general characteristics of infrared imaging, as well as optical or photographic settings used during data collection (e.g., imaging wavelength range, whether exposure values were fixed), would help readers better understand the dataset's properties and enhance reproducibility.\n\n2. Lack of Discussion on Limitations\n\nThe paper focuses on the model's performance but lacks an explicit discussion of its limitations. For example, it would benefit from exploring conditions under which the proposed multimodal regularization and initialization strategies may not perform well. Including such a discussion would allow for a more realistic assessment of the model's applicability and provide important insights for researchers who may wish to build upon this work.\n\n3. Lack of Discussion on Applicability and Future Work\nAlthough the multimodal learning approach for RGBT data proposed in this paper is academically valuable, the discussion on future work is somewhat general. The paper mentions super-resolution and dynamic scene reconstruction as directions for further research, but these are relatively standard research topics rather than areas specifically tailored to advancements in thermal 3D reconstruction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JNrhWsbZZx", "reviewer": "Reviewer_5amv", "summary": "ThermalGaussian is the first work with thermal 3D reconstruction with 3DGS representation.\n\nThe paper achieves thermal and RGB 3D reconstruction simultaneously, and improves the rendering results both of color and thermal images.\n\nFinally, the paper reduces the model storage cost by 90%.", "questions": "In the paper, the authors say that thermal imaging is practical task and 3D reconstruction gives significant applications which 2D images cannot provide. The reviewer wonders what is the specific case of that thermal 3D reconstruction is practical than 2D thermal image? There is a logical jump in the connection between the two. This context would be added at the related work 2.1 section.\n\nIn L. 314, the paper denotes thermal equilibrium, and it would be nice to explain what this means.\n\nIn the related work section, both of ThermoNeRF and ThermalNeRF are denoted but why only ThermoNeRF is compared to your method?\n\nThe reviewer suggests that comparison to existing thermal 3D reconstructions and to datasets used in previous studies, such as ThermoNeRF and Thermal-NeRF, would further emphasize the strength of the method itself.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper presents new thermal-RGB datasets which will be helpful in future research.\n\nThe method shows quality improvement not only for thermal reconstruction but also for RGB reconstruction.\n\nThe method that adds thermal reconstruction on 3DGs is effective and straightforward.", "weaknesses": "The paper proposes three multimodal training methods, which are MFTG, MSMG, and OMMG. The reviewer is confused about the pros and cons of each model, and users are confused about which model to choose. In the quantitative evaluation, the best performing models are distributed across each scene, with no model performing overwhelmingly well.\n\nIn the three multimodal training methods, the insight about why these three design are needed is lacked. In L. 283, the authors say “thermal Gaussian training in the second phase may not fully leverage the information from the color modality”, and there is no evidence about this.\n\nThere is no discussion about the limitations.\n\nIn L.448, there is a type. “Ours” → “ours”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J4ZUSfMYRo", "reviewer": "Reviewer_4Hay", "summary": "This paper studies the problem of scene reconstruction with both RGB and thermal image inputs. Specifically, 3D Gaussian Splatting is adopted as the baseline method as a replacement to NeRF. To equip 3D Gaussian Splatting with the ability to also learn 3D thermal representations, this paper proposes a series of strategies for multimodal Gaussian reconstruction, including multimodal initialization, three different thermal Gaussians, constraints specific to thermal modalities, and multimodal regularization. The paper also introduces a new dataset designed for thermal 3D reconstruction. \n\nExperiments demonstrate the advantage of the proposed method in the quality of reconstruction, in both RGB and thermal modalities.", "questions": "Please respond to the points made in the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper investigates an interesting problem of incorporating thermal information into 3D reconstruction by improving on the prevalent 3D Gaussian Splatting method. The presentation is clear and easy to follow. The results are also interesting in that both RGB and thermal reconstruction have improved.", "weaknesses": "I do not hold critical concerns to this paper. However, there are some issues that I'd be happy to see clarified. \n\nIn the Multiple Single-Modal Gaussians strategy, it is stated that \"Each Gaussian model is influenced\nnot only by its corresponding input modality but also by others.\" But how so? RGB modality and thermal modality seem to be uncorrelated to me since 3D Gaussians are trained separately for each. \n\nRegarding experiments, is there a specific reason that Thermal-NeRF has not been included in the comparison?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)