ThermalGaussian: Thermal 3D Gaussian Splatting

## Abstract

Abstract Thermography is especially valuable for the military and other users of surveillance cameras. Some recent methods based on Neural Radiance Fields (NeRF) are proposed to reconstruct the thermal scenes in 3D from a set of thermal and RGB images. However, unlike NeRF, 3D Gaussian splatting (3DGS) prevails due to its rapid training and real-time rendering. In this work, we propose ThermalGaussian, the first thermal 3DGS approach capable of rendering high-quality images in RGB and thermal modalities. We first calibrate the RGB camera and the thermal camera to ensure that both modalities are accurately aligned. Subsequently, we use the registered images to learn the multimodal 3D Gaussians. To prevent the overfitting of any single modality, we introduce several multimodal regularization constraints. We also develop smoothing constraints tailored to the physical characteristics of the thermal modality.
Besides, we contribute a real-world dataset named RGBT-Scenes, captured by a hand-hold thermal-infrared camera, facilitating future research on thermal scene reconstruction. We conduct comprehensive experiments to show that ThermalGaussian achieves photorealistic rendering of thermal images and improves the rendering quality of RGB images. With the proposed multimodal regularization constraints, we also reduced the model’s storage cost by 90%. The code and dataset will be released.

## 1 Introduction

Thermal imaging is widely used in fields such as military (He et al., ), healthcare (Lahiri et al., ), industry (Glowacz, ), agriculture (Zhou et al., ), building inspection (El Masri & Rakha, ), and search and rescue (Yeom, ) because it converts temperature information—an important physical modality not visible to the human eye—into interpretable images.
3D reconstruction technology, which involves lifting multi-view 2D images into 3D scenes, is foundational for key technologies such as the metaverse, digital twins, autonomous driving, and robotics. Any image with valuable 2D applications can be lifted into 3D to view the captured scene from a new view and in greater detail. Thermal images are no exception (Abreu de Souza et al., ; Liu et al., ).

Previous 3D thermal scene reconstruction (Rangel et al., ; Zhao et al., ; Müller, ; Li et al., ) typically involves a two-stage process. In the first stage, RGB images and traditional multi-view geometry methods (Newcombe et al., ) are used to achieve a 3D geometric reconstruction of the scene. In the second stage, thermal images are mapped onto the reconstructed 3D scene. However, these methods not only fail to fully exploit thermal information but are also constrained by the limitations of traditional 3D reconstruction techniques, which impede their ability to render high-quality images from a new view. This significantly hinders the practical application of thermal reconstruction.

Neural Radiance Fields (NeRF) (Mildenhall et al., ) can render photorealistic images from a new view, thus revolutionizing novel-view synthesis and 3D reconstruction.
Recently, several NeRF-based methods (Hassan et al., ; Ye et al., ) have been proposed to reconstruct thermal scenes in 3D using thermal images. However, NeRF’s slow rendering speed and implicit scene representation limit its practical applications.
In contrast, 3D Gaussian Splitting (3DGS) (Kerbl et al., ) not only maintains photorealistic rendering quality from new views but also significantly improves rendering speed. Additionally, the explicit 3D scene representation of 3DGS, akin to point clouds, facilitates integration with downstream tasks, establishing it as a leading approach in research.
Building on 3DGS, we propose ThermalGaussian, a multimodal Gaussian technique that renders high-quality RGB and thermal images from new views.

The absence of open-source datasets dedicated to thermal scene reconstruction significantly impedes progress in this domain. Several researchers (Hassan et al., ; Ye et al., ) have recognized this issue and have released some datasets. However, these datasets suffer from problems such as lack of color images registered with thermal images, inconsistencies in thermal information from different views, and watermarked images. To address these issues, we contribute a real-world dataset named RGBT-Scenes.

Unlike RGB images, thermal images possess unique low-texture and ghosting characteristics (Bao et al., ) that hinder accurate camera pose estimation using Structure-from-Motion (SfM) (Schönberger & Frahm, ), as illustrated in Fig. 2b. Consequently, thermal images cannot directly replace RGB images for running 3DGS. To address this issue, we first register the RGB and thermal images and then fuse them (Fig. 2c), or use Multi-Spectral Dynamic Imaging (MSX) (Abdullah, ) (Fig. 2d) to localize the thermal image camera. Additionally, We design a thermal loss to adapt to the unique characteristics of thermal images.

Figure: (a) RGB
Refer to caption: extracted/5842471/image/colmap/ICLR_colmap_RGB.png

Introducing a new modality, such as thermal imaging, into 3D reconstruction should enable the model to understand the scene from a more comprehensive perspective. However, ThermoNeRF (Hassan et al., ) reduces the RGB rendering quality after implementing thermal reconstruction. In contrast, our method not only improves thermal rendering quality but also enhances RGB rendering quality by 1 dB. Furthermore, to prevent overfitting of any single modality during multimodal Gaussian training, we introduce a multimodal regularization coefficient. This approach significantly reduces model storage requirements and accelerates rendering speed.
In summary, the main contributions as follows:

(1)We propose ThermalGaussian, the first multimodal 3DGS capable of simultaneously rendering photorealistic thermal and RGB images of a scene.

(2)We propose a series of strategies for multimodal Gaussian reconstruction, including multimodal initialization, three different thermal Gaussians training strategies, constraints specific to thermal modalities, and multimodal regularization.

(3)We introduce RGBT-Scenes, a new dataset designed for thermal 3D reconstruction and novel-view synthesis. The dataset consists of paired RGB and thermal images captured from multiple viewpoints across 10 different scenes.

(4)Finally, experimental results show that our multimodal method not only improves the rendering quality of both thermal and RGB images but also reduces storage space by 90% compared to training each modality separately, while also improving rendering speed.

## 2 Related Work

### 2.1 Thermal imaging and 3D reconstruction

All objects with temperatures above absolute zero emit energy in the form of electromagnetic waves, a phenomenon known as thermal radiation. Through Planck’s law, thermal imaging converts the wavelength and intensity of electromagnetic waves radiated from an object’s surface into thermal information, which is then used to create images. Originally, thermal imaging was developed for military purposes to enable visualization under extreme lighting conditions, such as nighttime or smoke. As costs have decreased in recent years, it has also been widely applied in fields of healthcare, industry, agriculture, building inspection, and so on.

3D reconstruction has significant applications in visualizing, surveying, and analyzing large, inaccessible, and important objects, in ways that 2D images cannot provide. The advent of the groundbreaking KinectFusion (Newcombe et al., ) marked the beginning of an era of high-precision, dense 3D reconstruction. Subsequent developments (Nießner et al., ; Kähler et al., ; Dai et al., ; Gong et al., ; Zhang et al., ) have optimized 3D reconstruction in terms of accuracy, efficiency, and unconstrained camera movement.
Some works (Rangel et al., ; Zhao et al., ; Müller, ; Li et al., ) have been made to integrate thermal imaging with the aforementioned 3D reconstruction methods, leading to the development of thermal 3D reconstruction. However, these traditional multi-view geometry-based methods do not perform as well in rendering new views as the more recent deep learning-based approaches.

NeRF (Mildenhall et al., ) has emerged as a significant milestone in the field of 3D reconstruction due to its impressive ability to render highly realistic images from a new view.
Recently, ThermoNeRF (Hassan et al., ) and Thermal-NeRF (Ye et al., ) have been proposed to reconstruct thermal scenes by combining a set of thermal images with NeRF.
Although these approaches successfully generate images from new perspectives, they are limited by the slow rendering speed and implicit scene representation of NeRF, which hampers their practical application.

### 2.2 3DGS and Multimodality

3DGS (Kerbl et al., ) represent a revolutionary technology in the fields of 3D reconstruction. Distinct from methods like NeRF, 3DGS employs millions of explicit Gaussians, fundamentally altering its approach. This technology merges the advantages of neural network-based optimization with structured data representation, enabling photorealistic rendering from new views, significantly enhancing real-time rendering capabilities, and introducing the ability to manipulate and edit 3D scenes. These features make 3DGS highly compatible with a broad range of downstream applications, establishing it as the baseline for next-generation 3D reconstruction technologies (Chen & Wang, ).
Although 3DGS is constrained and trained using only RGB modality images, it ultimately generates millions of Gaussians, resembling a point cloud. This characteristic makes 3DGS particularly suitable for multimodal fusion with other devices that directly capture scene point clouds, such as depth cameras and LiDAR.
Studies (Matsuki et al., ; Yan et al., ; Keetha et al., ) have effectively integrated depth cameras to implement 3DGS-based simultaneous localization and mapping.
Studies (Li et al., ; Chung et al., ) have effectively combined depth images (Yan et al., ) estimated from a pre-trained Monocular Depth estimation model (Bhat et al., ) with the RGB modality, resulting in improved rendering quality and more accurate geometric structures.
3D scenes contain not only RGB and geometric modalities but also other important modalities relevant to various applications, such as thermal, material, and pressure. We integrate 3DGS with thermal modality.

## 3 Method

Figure: Figure 3: ThermalGaussian Overview. We simultaneously construct Gaussians for RGB and thermal modalities using the point cloud obtained from multimodal initialization. Each modality’s Gaussians are used to render images in their respective modality. However, the losses from different modalities are combined to jointly constrain the optimization of both sets of Gaussians. Additionally, we establish a multimodal regularization based on the number of Gaussians in each modality, which dynamically adjusts the training coefficients for both modalities.
Refer to caption: extracted/5842471/image/new_pipline.png

Fig. 3 shows the overview of the proposed ThermalGaussian, which is based on the 3DGS (Kerbl et al., ), aiming to extend its capability to simultaneously render images of color and temperature.
In this section, we first briefly introduce the background of the 3DGS.
Then, we provide a detailed description of our method’s specific implementation details, including multimodal initialization, three types of multimodal thermal Gaussians, thermal loss, and multimodal regularization.

### 3.1 Preliminary: 3D Gaussian Splatting

3DGS (Kerbl et al., ) represents a 3D reconstruction scene using a large number of anisotropic 3D Gaussians. This representation not only provides differentiability, which offers advantages in learning-based methods, but also enables explicit spatial expression, enhancing the editability and controllability of 3D scenes. Furthermore, it allows for rapid and efficient rasterization rendering through splatting.
Initially, a set of unordered images of objects to be reconstructed is processed using SfM to obtain the camera poses and sparse point clouds.
3DGS then initializes these sparse point clouds as the position $\mu$ of a 3D Gaussian:

$$ $G(x)=e^{-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)}$ (1) $$

where $\Sigma$ represents the covariance matrix of the 3D Gaussian, and $x$ denotes any point in the 3D scene. $\Sigma$ is defined using a scaling matrix $\bm{S}$ and a rotation matrix
$\bm{R}$:

$$ $\Sigma=\bm{R}\bm{S}\bm{S}^{T}\bm{R}^{T}$ (2) $$

The 3D Gaussian $G(x)$ is projected onto the imaging plane using the camera’s intrinsic parameters, transforming it into a 2D Gaussian. Subsequently, the image is rendered through alpha-blending:

$$ $C\left(x^{\prime}\right)=\sum_{k\in N}c_{k}\alpha_{k}\prod_{j=1}^{k-1}\left(1- \alpha_{j}\right)$ (3) $$

where $x^{\prime}$ represents the queried pixel position, $N$ denotes the number of 2D Gaussians corresponding to this pixel, $\alpha$ denotes the opacity of each Gaussian and the color $c$ on each Gaussian is modeled spherical harmonics.
All attributes of the 3D Gaussians are learnable and optimized directly in an end-to-end manner during training.

### 3.2 Multimodal Initialization

Previously, methods for calibration RGB and thermal images (Zhang et al., ) often involve designing specialized, non-standard metallic calibration boards with uniformly sized circular holes, as shown in Fig. 4a. The calibration relies on the temperature difference between the board and the background to compute thermal features, enabling calibration. However, the high complexity and stringent requirements for producing these calibration boards make them difficult to obtain and lack a universal standard.
We find that a standard chessboard pattern, as shown in Fig 4b, commonly used for RGB camera calibration, can effectively be used for calibrating both thermal and color cameras, with a mean reprojection error of less than 0.5 pixels. Initially, we heat the calibration board using devices like an infrared heater; black regions, absorbing heat faster due to their material properties, exhibit relatively higher temperatures. We capture simultaneous color and thermal images before a thermal equilibrium with surroundings is achieved. Subsequently, conventional camera calibration (Zhang, ) is performed.

Using the calibrated intrinsic parameters
$\bm{K}_{\text{RGB}}$ for the color camera, $\bm{K}_{\text{Th}}$ for the thermal camera, and the rotation $\bm{R}$ and translation $\bm{t}$ from the temperature camera to the color camera, we computed the corresponding positions $(u_{\text{Th}},v_{\text{Th}})$ on the thermal image mapped to the registered positions on the color image:

$$ $\left[\begin{array}[]{c}u_{\text{RGB}}\\ v_{\text{RGB}}\\ 1\end{array}\right]=\bm{K}_{\text{RGB}}\left(\bm{R}\cdot\bm{K}_{\text{Th}}^{-1 }\left[\begin{array}[]{c}u_{\text{Th}}\\ v_{\text{Th}}\\ \end{array}\right]+\bm{t}\right)$ (4) $$

Figure: (a) Special board
Refer to caption: extracted/5842471/image/registration/special_board.png

As shown in Fig.2(b), directly using thermal images, which exhibit low texture and ghosting characteristics, makes it difficult to successfully run SfM (Schonberger & Frahm, ). Therefore, to obtain the thermal camera poses, we tested three different multimodal SfM strategies.
The first utilizes registered high-texture RGB images directly for camera pose estimation. These poses serve simultaneously for both the RGB and thermal cameras. However, practical scenarios that require thermal scene reconstruction often occur under dim lighting conditions or in scenes lacking distinct color features. Therefore, relying solely on color images may impede the precise camera pose estimation necessary for thermal scene reconstruction.
The second approach, illustrated in Fig.2(c), involves blending registered color and thermal images using the following formula:

$$ $I_{\text{mix}}=\beta I_{\text{Th}}+(1-\beta)I_{\text{RGB}}$ (5) $$

where in the above equation, $\beta$ represents the opacity of the thermal image.
This method produces blended images containing both rich color and thermal information, catering to various practical applications of thermal scene reconstruction.
The third strategy, depicted in Fig.2(d), maps high-frequency color variations from the color images onto the thermal images. This approach mitigates the lack of feature points caused by thermal images’ low texture and ghosting characteristics.

### 3.3 Thermal Gaussian

We utilize three different multimodal training strategies to construct the thermal Gaussian.

Multimodal Fine-Tuning Gaussians (MFTG):
Inspired by the fine-tuning approach used in large-scale models, our first multimodal training strategy is training a basic Gaussian with RGB images and then refining this Gaussian with thermal images to generate thermal Gaussian.
This is a two-stage process.
In the first stage, similar to 3DGS, we utilize multimodal camera poses and initial point clouds obtained from multimodal initialization as inputs. The training is supervised using RGB images, with $\mathcal{L}_{1}$ combined with a D-SSIM term:

$$ $\mathcal{L}_{\text{RGB}}=(1-\lambda)\mathcal{L}_{1}+\lambda\mathcal{L}_{\text{ D-SSIM}}$ (6) $$

This stage enables us to render high-quality RGB modality images from a new view and establish a basic 3D Gaussian with preliminary geometry. In the second stage, we fine-tune this basic Gaussian model with thermal images and multimodal camera poses obtained from initialization. Since the first stage constraints were based on texture-rich color images rather than thermal images, which results in a better geometry. Therefore, training on this geometry yields better results than training thermal Gaussian directly from the initial point cloud derived from multimodal initialization.

Multiple Single-Modal Gaussians (MSMG):
The training of MFTG initially utilizes RGB modal information followed by thermal modal information. Although both modalities are employed, they are not used simultaneously. This sequential approach during the thermal Gaussian training in the second phase may not fully leverage the information from the color modality. Therefore, in MSMG (as shown in Fig. 3), we constrain the training with information from both color and thermal modalities simultaneously. We train two single-modal Gaussians initialized by point clouds from multimodal initialization. The thermal Gaussian renders thermal images, while the RGB Gaussian renders RGB images. Subsequently, these rendered images of both modalities are compared separately with the ground truth of their respective inputs using loss functions:

$$ $\mathcal{L}=\mathcal{L}_{\text{RGB}}+\mathcal{L}_{\text{thermal}}$ (7) $$

The details of $\mathcal{L}_{thermal}$ constraint will be elaborated below. Each Gaussian model is influenced not only by its corresponding input modality but also by others. Experimental results indicate that joint constraints across multiple modalities enhance the training outcomes for both color and thermal modalities. Moreover, since these modalities jointly optimize the entire scene from different perspectives, redundant points are pruned to some extent, reducing the number of points in the point cloud and lowering the model’s storage requirements.

One Multi-Modal Gaussian(OMMG):
OMMG extends MSMG by not only employing dual-modal loss constraints but also integrating multiple modalities onto a single Gaussian. This integration ensures that information from diverse modalities is unified within a single geometric structure. Specifically, we construct a multimodal Gaussian comprising positional coordinates $x$, scaling matrix $\mathbf{S}$, rotation matrix $\mathbf{R}$, opacity $\alpha$, spherical harmonics $c$ for RGB representation, and spherical harmonics $t$ for thermal representation. RGB rendering is achieved using Equation 3, while thermal rendering follows the equation below:

$$ $T\left(x^{\prime}\right)=\sum_{k\in N}t_{k}\alpha_{k}\prod_{j=1}^{k-1}\left(1- \alpha_{j}\right)$ (8) $$

### 3.4 Thermal Loss and Multimodal Regularization

The loss function for RGB modality images is directly given by Equation 6. The same loss function can also be applied to thermal modality images. However, because thermal images exhibit unique low-texture and ghosting characteristics, we design a specific thermal loss function to better accommodate these features.

The RGB modality may exhibit abrupt changes. However, because all objects above absolute zero continuously engage in heat transfer and thermal radiation, eventually reaching thermal equilibrium with their surroundings, significant abrupt changes are typically not observed in thermal images. Additionally, most regions of objects in thermal equilibrium have similar temperatures, resulting in smoother thermal images. Therefore, we introduce a smoothness term for regularization:

$$ $\mathcal{L}_{\text{smooth }}=\frac{1}{4M}\sum_{i,j}\left(\left|T_{i\pm 1,j}-T_ {i,j}\right|+\left|T_{i,j\pm 1}-T_{i,j}\right|\right)$ (9) $$

where $T_{i,j}$ represents rendered thermal values at pixel position $(i,j)$. $M$ denotes the number of rendering pixels.
Similarly to the color modality, we also incorporate $\mathcal{L}_{1}$ and $\mathcal{L}_{\text{D-SSIM}}$ Thus, our final temperature loss is:

$$ $\mathcal{L}_{\text{thermal}}=(1-\lambda)\mathcal{L}_{1}+\lambda\mathcal{L}_{ \text{SSIM}}+\lambda_{\text{smooth}}\mathcal{L}_{\text{smooth}}$ (10) $$

where $\lambda_{\text{smooth}}$ is the coefficient of $\mathcal{L}_{\text{smooth}}$.

**Table 1: Comparison of our collected dataset with others.**
| Dataset | Mode | Bimodal Calibration | Multiview Consistency | Content Richness |
| --- | --- | --- | --- | --- |
| T / RGB |  |  |  |  |
| Thermal-NeRF | ✓  /  $\times$ | $\times$ | - | Simple |
| ThermoNeRF | ✓  /  ✓ | $\times$ | $\times$ | Moderate |
| Ours | ✓  /  ✓ | ✓ | ✓ | Rich |

When training multiple modalities of the same object simultaneously, it is often undesirable for one modality to dominate at the expense of others. Therefore, a regularization strategy is needed to dynamically adjust the weight of each modality’s loss during training. In the training of MSMG, we observe that the weight coefficients of a modality align linearly with the number of generated Gaussian. A higher weight for a modality results in more Gaussian generated by that modality, while fewer Gaussian are generated by another modality. Hence, we design multi-modal regularization coefficients $\gamma$ based on the number of Gaussian generated by each modality during training.

$$ $\gamma=\frac{N_{thermal}}{N_{thermal}+N_{RGB}}$ (11) $$

where $N_{thermal}$ represents the number of Gaussian for the thermal modality during training. When the number of one modality’s Gaussian increases, we increase the training weight of the other modality. This dynamic balancing of weights ultimately prevents overfitting to any single modality.
The final design of this loss is:

$$ $\mathcal{L}=\gamma\mathcal{L}_{\text{RGB}}+(1-\gamma)\mathcal{L}_{\text{ thermal}}$ (12) $$

## 4 Self-Collected Theraml Dataset

We introduce a new dataset, named RGBT-Scenes, which consists of aligned collections of thermal and RGB images captured from various viewpoints of a scene. The images are collected using the commercial-grade handheld thermal-infrared camera FLIR E6 PRO (Teledyne FLIR, ), which can simultaneously capture RGB and thermal images. The basic specifications of this camera include a resolution of 240×180, a field of view of 33°×25°, a temperature range from -20°C to 550°C, and a temperature accuracy of ±2$\%$ of the reading. Our dataset includes over 1,000 RGB and thermal images from 10 different scenes. These scenes encompass both indoor and outdoor environments, various object sizes (from large structures to everyday items), different temperature variations (ranging from a 300°C difference to a 4°C difference), and include both 360-degree and forward-facing scenarios.
We provide the raw images captured by the thermal camera, as well as the RGB images, thermal images, MSX images, and camera pose data. In Table 1, we compare our dataset with those from concurrent works, Thermal-NeRF (Ye et al., ) and ThermoNeRF (Hassan et al., ). Our dataset includes both RGB and thermal images and applies multimodal calibration methods to align these images. The images used for calibration will also be made available. Compared to ThermoNeRF, our dataset features improved multiview consistency of thermal information and encompasses a richer variety of scenes. Detailed descriptions of each scene are provided in the supplementary materials.

**Table 2: Quantitative evaluation of thermal image using our method compared to previous work from test views. ”$\times$” indicates a failure to localize using only thermal images in the scene, making it impossible to success with 3DGS. 3DGS+MI represents the results obtained by directly training 3DGS after Multimodal Initialization.**
| Metric | Method | Dimsum | Daily Stuff | Ebike | Road Block | Truck | Rotary Kiln | Building | Iron Ingot | Parterre | Land Scape | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | 3DGS | 25.38 | $\times$ | $\times$ | $\times$ | 20.97 | 23.79 | 23.75 | $\times$ | $\times$ | $\times$ | $\times$ |
| ThermoNeRF | 24.27 | 17.34 | 19.70 | 17.17 | 23.53 | 26.40 | 23.31 | 22.97 | 17.88 | 18.79 | 21.13 |  |
| 3DGS+MI | 26.35 | 18.77 | 20.89 | 26.75 | 26.17 | 26.59 | 25.76 | 29.57 | 22.09 | 20.17 | 24.31 |  |
| OursMFTG | 26.94 | 20.52 | 22.51 | 24.96 | 25.02 | 26.91 | 26.11 | 30.41 | 23.55 | 20.03 | 24.70 |  |
| OursMSMG | 26.73 | 21.35 | 23.23 | 26.52 | 26.27 | 27.15 | 26.83 | 30.06 | 25.01 | 20.61 | 25.38 |  |
|  | OursOMMG | 26.46 | 22.28 | 23.31 | 27.17 | 25.88 | 26.33 | 26.72 | 29.86 | 26.16 | 22.27 | 25.64 |
| SSIM $\uparrow$ | 3DGS | 0.860 | $\times$ | $\times$ | $\times$ | 0.717 | 0.872 | 0.810 | $\times$ | $\times$ | $\times$ | $\times$ |
| ThermoNeRF | 0.747 | 0.759 | 0.694 | 0.781 | 0.750 | 0.916 | 0.804 | 0.717 | 0.709 | 0.774 | 0.765 |  |
| 3DGS+MI | 0.889 | 0.789 | 0.806 | 0.917 | 0.872 | 0.922 | 0.872 | 0.887 | 0.843 | 0.794 | 0.859 |  |
| OursMFTG | 0.890 | 0.798 | 0.845 | 0.906 | 0.880 | 0.920 | 0.886 | 0.895 | 0.859 | 0.808 | 0.869 |  |
| OursMSMG | 0.891 | 0.829 | 0.857 | 0.909 | 0.879 | 0.926 | 0.897 | 0.898 | 0.860 | 0.832 | 0.878 |  |
|  | OursOMMG | 0.886 | 0.835 | 0.862 | 0.919 | 0.874 | 0.922 | 0.888 | 0.896 | 0.883 | 0.850 | 0.882 |
| LPIPS $\downarrow$ | 3DGS | 0.157 | $\times$ | $\times$ | $\times$ | 0.281 | 0.193 | 0.299 | $\times$ | $\times$ | $\times$ | $\times$ |
| ThermoNeRF | 0.312 | 0.494 | 0.290 | 0.293 | 0.291 | 0.170 | 0.234 | 0.152 | 0.309 | 0.264 | 0.280 |  |
| 3DGS+MI | 0.124 | 0.274 | 0.313 | 0.204 | 0.139 | 0.125 | 0.211 | 0.093 | 0.252 | 0.328 | 0.206 |  |
| OursMFTG | 0.121 | 0.258 | 0.235 | 0.210 | 0.133 | 0.129 | 0.199 | 0.091 | 0.232 | 0.317 | 0.192 |  |
| OursMSMG | 0.124 | 0.208 | 0.220 | 0.213 | 0.133 | 0.130 | 0.189 | 0.086 | 0.227 | 0.293 | 0.182 |  |
|  | OursOMMG | 0.129 | 0.210 | 0.203 | 0.198 | 0.136 | 0.124 | 0.177 | 0.091 | 0.181 | 0.248 | 0.170 |

## 5 Experiments

### 5.1 Implementation Details

Our method is an improvement upon the 3DGS framework, with all experimental settings consistent with the reference 3DGS. The specific hyperparameter $\lambda_{smooth}$ is set to 0.6. Each comparative experiment was trained for 30K iterations. All experiments are conducted on a single NVIDIA 3090 GPU. The resolution of the rendered RGB images and thermal images is 640×480.

### 5.2 Thermal View Synthesis

Figure: (a) RGB GT
Refer to caption: extracted/5842471/image/exp_image/mini_new_RGB_GT.png

Similar to 3DGS, we employ image quality assessment metrics including Peak Signal-to-Noise Ratio (PSNR) (Hore & Ziou, ), Structural Similarity Metric (SSIM) (Wang et al., ), and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., ) to evaluate the quality of reconstructed thermal and RGB images from new views.

As shown in Table 2, even in scenes with pronounced thermal variations, specifically targeting low-texture thermal characteristics, direct application of thermal data proves challenging for 3DGS.
In very few successful cases, inadequate precision in thermal camera positioning has compromised the quality of thermal reconstructions.
3DGS+MI denotes training the original 3DGS using thermal images instead of RGB images after obtaining accurate thermal poses through our multimodal initialization.
Compared to 3DGS, 3DGS+MI adapts to a wider range of scenarios and achieves higher reconstruction quality.
Given the higher reconstruction quality of 3DGS (Kerbl et al., ) compared to NerfStudio (Tancik et al., ), 3DGS+MI and our method naturally outperforms ThermoNeRF.
Our three thermal Gaussian methods outperform 3DGS+MI across all scenes in PSNR, SSIM, and LPIPS. Among them, OursOMMG shows an average PSNR improvement of 1.3 dB.
As shown in Fig. 5, Our method’s qualitative rendering of thermal images is clearly superior.
Additionally, as depicted in Fig. 5f, we enhance thermal image readability by training with MSX images using thermal Gaussian. This hierarchical and easily recognizable thermal Gaussian further promotes the application of thermal scene reconstruction. We provide more qualitative comparison results in the supplementary materials.

### 5.3 RGB View Synthesis

**Table 3: Quantitative evaluation of RGB image using our method compared to 3DGS.**
| Metric | Method | Dimsum | Daily Stuff | Ebike | Road Block | Truck | Rotary Kiln | Building | Iron Ingot | Parterre | Land Scape | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | 3DGS | 23.91 | 20.43 | 26.77 | 27.80 | 22.30 | 20.79 | 20.95 | 23.96 | 24.91 | 20.20 | 23.20 |
| OursMSMG | 24.42 | 21.71 | 27.34 | 28.22 | 23.57 | 22.23 | 23.08 | 25.69 | 25.57 | 20.91 | 24.27 |  |
| OursOMMG | 24.38 | 21.76 | 26.85 | 28.12 | 24.17 | 23.14 | 24.19 | 24.55 | 25.48 | 21.71 | 24.34 |  |
| SSIM $\uparrow$ | 3DGS | 0.847 | 0.748 | 0.901 | 0.910 | 0.807 | 0.772 | 0.791 | 0.872 | 0.859 | 0.696 | 0.820 |
| OursMSMG | 0.858 | 0.793 | 0.917 | 0.916 | 0.833 | 0.811 | 0.844 | 0.891 | 0.874 | 0.715 | 0.845 |  |
| OursOMMG | 0.858 | 0.797 | 0.905 | 0.920 | 0.840 | 0.822 | 0.849 | 0.884 | 0.855 | 0.739 | 0.847 |  |
| LPIPS $\downarrow$ | 3DGS | 0.194 | 0.299 | 0.171 | 0.201 | 0.232 | 0.217 | 0.228 | 0.188 | 0.183 | 0.280 | 0.219 |
| OursMSMG | 0.194 | 0.262 | 0.156 | 0.221 | 0.217 | 0.190 | 0.168 | 0.172 | 0.184 | 0.275 | 0.204 |  |
| OursOMMG | 0.194 | 0.253 | 0.169 | 0.199 | 0.211 | 0.184 | 0.170 | 0.186 | 0.195 | 0.268 | 0.203 |  |

Figure: (a) 3DGS(RGB)
Refer to caption: extracted/5842471/image/exp_image/ima_3DGS.png

Our method not only achieves high-quality thermal image rendering but also significantly enhances RGB image rendering quality.
As shown quantitatively in Table 4, our multimodal constraints improve RGB rendering quality in nearly all scenarios, with an average PSNR improvement of 1.1 dB compared to the original 3DGS (Kerbl et al., ).
This improvement is particularly evident in scenarios where the RGB modality struggles to identify the environment, while the thermal modality can recognize it clearly. As shown in the top part of Fig. 6, where distinguishing between foreground and background is challenging in the RGB modality but straightforward in the thermal modality due to temperature differences, constraints from the thermal modality aid in the accurate learning of the RGB modality. Additionally, as depicted in the bottom part of Fig. 6, the assistance from thermal images enables accurate color rendering in low-light scenes for the RGB modality.
Our results demonstrate that, under multimodal constraints, when one modality fails, our approach leverages accurate information from the other modality to enhance the model’s understanding of the scene, thus facilitating the correct learning of the failing modality. This enables our method to advance 3D reconstruction in low-light scenes and enhances the robustness of 3D reconstruction techniques to some extent.

**Table 4: Ablation Study. We conducted ablation experiments by gradually adding each component to the baseline 3DGS model. We then performed a comprehensive comparison across various dimensions, including rendering capability, the quality of rendered color and thermal images, training time, model memory usage, and the number of Gaussians. ”-” indicates that the model lacks the specified capability or metric, ”$\times$” denotes a reconstruction failure. Since multimodal regularization relies on the Gaussians from multiple modalities, it only applies to OursMSMG.**
| Methods | Mode | Thermal | RGB | Train | FPS | Mem. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| T / RGB | PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS |  |  |  |  |
| 3DGS | ✓  /  - | $\times$ | $\times$ | $\times$ | - | - | - | $\times$ | $\times$ | $\times$ |
| -   / ✓ | - | - | - | 23.20 | 0.820 | 0.219 | 507s | 231 | 159MB |  |
| +MI | ✓ /  - | 24.31 | 0.859 | 0.206 | - | - | - | 367s | 277 | 65MB |
| + $\mathcal{L}_{\text{smooth }}$ | ✓ /  - | 24.65 | 0.867 | 0.198 | - | - | - | 603s | 292 | 61MB |
| OursMFTG | ✓  /  - | 24.70 | 0.871 | 0.191 | - | - | - | 491s+575s | 316 | 51MB |
| OursMSMG | ✓  / ✓ | 25.38 | 0.883 | 0.180 | 24.27 | 0.845 | 0.204 | 873s | 330 / 298 | 18MB+66MB |
| OursOMMG | ✓  / ✓ | 25.64 | 0.883 | 0.169 | 24.34 | 0.800 | 0.203 | 838s | 271 / 242 | 136MB |
| OursMSMG+MR | ✓  / ✓ | 25.09 | 0.880 | 0.189 | 24.21 | 0.840 | 0.235 | 760s | 390 / 420 | 9MB+9MB |

### 5.4 Ablation Study

We separate different contributions and algorithm choices to test their effectiveness. As shown in Table 2 and Table 4, after incorporating multimodal initialization, allows 3DGS to achieve thermal reconstruction across various environments.
Our designed multimodal thermal Gaussian models, MSMG and OMMG, not only render both thermal and RGB images simultaneously but also improve rendering quality for both modalities in all scenes, with an average increase of over 1.2 dB.
We also observed that multimodal constraints mitigate the generation of excessive redundant Gaussians in certain modalities. Later, we introduced a regularization term to dynamically adjust the coefficients of both modalities.
As shown in Table 4,
directly training RGB modality Gaussians with 3DGS results in an average storage requirement of 159 MB. On the other hand, directly training thermal Gaussians with MI requires an average of 65 MB. The RGB Gaussians for OursMSMG+MR average only 9 MB in storage, with thermal Gaussians averaging the same. Our method requires only 8% $(\frac{9+9}{159+65}=0.08)$ of the storage space compared to directly using 3DGS. Due to the reduction in the number of Gaussians, the rendering speed has also significantly increased. Additionally, the rendering quality for both modalities has also improved.
In Fig. 7a, we compare our multimodal regularization $\gamma$ with manually adjusting the thermal constraint coefficients in a truck scene. The comparison shows that our multimodal regularization approach reduces storage space for both RGB and thermal modalities while maintaining high image quality.
In Fig.7b, we visually present the Gaussian distributions of the original 3DGS method and our method with multimodal regularization.

Figure: (a) MR ($\gamma$) vs. Fixed coefficient
Refer to caption: x1.png

## 6 Conclusion and Future Work

We are the first to implement thermal reconstruction based on 3DGS. We not only achieve simultaneous rendering of thermal and RGB images but also significantly improve the rendering quality of both color and thermal images. Additionally, we greatly reduce the model’s storage requirements.
In the future, we plan to incorporate super-resolution for more detailed thermal reconstruction and to explore the reconstruction of dynamic thermal processes.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: - Can the authors clarify the choice of hyperparameters for multimodal regularization? Are there general guidelines for adjusting them?

- How does the model handle extreme cases, such as highly reflective surfaces or objects at temperatures that differ significantly from ambient conditions?

- Could the RGBT-Scenes dataset be expanded to include data for real-time applications, like moving objects or dynamic scenes?
soundness: 3
strengths: The proposed approach is novel, especially in integrating thermal and RGB images in 3D reconstruction via Gaussian splatting. And the experimental results are clear, and include diverse evalutaion metrics and ablation studies to validate performance of this methods.

Also, introducing the RGBT-Scenes dataset is a valuable contribution that can serve as a benchmark for future work in thermal 3D reconstruction.
summary: This paper introduce ThermalGaussian, a multimodal Gaussian technique that renders high-quality RGB & thermal images from new views.
Also they introduce a new real-world dataset named RGBT-Scenes, to address the issue of existing problems.
Authors propose multimodal regularization to prevent overfitting and show that ThermalGaussian outperforms previous methods by enhancing rendering quality, reducing storage requirements, and improving efficiency.
weaknesses: The process of multimodal initialization and the specific configurations for the thermal Gaussian construction are intricate, making the approach challenging to implement without in-depth knowledge. And, although the RGBT-Scenes dataset is comprehensive, details regarding environmental diversity and lighting conditions could strengthen its applicability across broader contexts.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - What does „Mix50“ in Figure 2 stand for? Is Eq. (5) applied with $\beta=0.5$? If so, please leave a note in the caption of Figure 2.

- How sensitive is $\lambda_{smooth}$? Would it be possible to provide a small ablation study?

- How was $\lambda$ in Eq. (6) and Eq. (10) set? Also, are these two lambdas the same?

- As far as I understand, MSMG uses the cost function described in Eq. (7), i.e., without multimodal regularization. So, whenever "Ours_MSMG" is used in Tables 2 and 3, it refers to MSMG method trained without multimodal regularization. This is in accordance with Table 4, where authors add a dedicated row for "Ours_MSMG+MR" which would not be necessary if MSMG uses MR per default (in this case, I would rather expect a row with "Ours_MSMG w\o MR" in Table 4). As this would mean that MR has never been used except in the ablation study, I would guess that per default, Eq. (12) has been used to train MSMG. Please make this more clear (e.g., by replacing Eq. (7) with Eq. (12) and referring to Section 3.4. on how to choose $\gamma$).

- What cost function is used to train OMMG? Is it Eq. (7)? If so, please move it to the respective paragraph (starting L296).

- In Figures 5 and 6, which strategy does "Ours" refer to?

- To be honest, it is quite surprising to me that a conventional chessboard pattern printed on normal paper produces such highly contrasting thermal images. There are numerous works (see, e.g., [6] and references therein) that try to design special calibration objects for thermal cameras. I wonder why all these papers exist when a simple chessboard pattern printed on paper will also work. What are the physical principles behind why a simple chessboard pattern printed on paper works, and how does it compare to more complex calibration objects in terms of accuracy and ease of use?

References:

[6] Issac N. Swamidoss et al. Systematic approach for thermal imaging camera calibration for machine vision applications. Optik 247, 2021.
soundness: 3
strengths: - The topic is interesting and of importance to the community, as demonstrated by the large body of recent work [1-5].

- The paper is well-written and easy to follow. Experiments are well described and cared out thoroughly. The ablation study nicely supports all design decisions. All in all,  I don’t see any obvious flaws.

- This is (to the best of my knowledge) the first paper to use some kind of scheduling to dynamically adjust the weight that balances the RGB and thermal loss in Eq. (12). The proposed strategy seems plausible to me, and, according to the experimental evaluation, significantly reduces the memory footprint while only slightly decreasing image quality. Due to the reduced number of Gaussians, this strategy also increases rendering speed.

- Compared to prior work [1,3] (but similar to [5]), the method presented in this work is actually capable of *improving* the RGB rendering quality, especially in low-light conditions.

- The paper proposes and evaluates not only a single method but three different strategies to incorporate thermal images into 3DGS, thus can be seen as the 3DGS-based pendant to [1].

- I appreciate the newly collected dataset. Although small-scaled (like all other publicly available RGB+thermal datasets), it is diverse, contains forward-facing and 360-degree scenes, and encompasses indoor and outdoor environments. This is very helpful to the community.

References:

[1] Mert Özer et al. Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging. ECCVW, 2024.

[2] Yvette Y. Lin et al. ThermalNeRF: Thermal Radiance Fields. ICCP, 2024.

[3] Miriam Hassan et al. ThermoNeRF: Multimodal Neural Radiance Fields for Thermal Novel View Synthesis. arXiv, 2024.

[4] Tianxiang Ye et al. Thermal-NeRF: Neural Radiance Fields from an Infrared Camera. arXiv, 2024.

[5] Jiacong Xu et al. Leveraging Thermal Modality to Enhance Reconstruction in Low-Light Conditions. ECCV, 2024.
summary: This paper proposes a method based on 3DGS that jointly learns multi-modal scene representations of RGB and thermal images. In fact, the authors present and evaluate three different strategies to incorporate thermal images into 3DGS: MFTG (based on fine-tuning), MSMG (leverages multi-task learning), and OMMG—the latter yields a single Gaussian, which is achieved by extending 3DGS with spherical harmonics to represent thermal data. Moreover, a regularization term is proposed that accounts for the smooth nature of thermal images, and a new weight scheduling strategy prevents the proposed MSMG from overfitting to a single modality. Based on a newly collected RGB+thermal dataset consisting of ten scenes, the authors could successfully show that their method outperforms 3DGS baselines and a recent NeRF-based method in terms of RGB and thermal rendering quality while requiring less memory.
weaknesses: - I am missing citations (and discussions) of recent related works [1,2,5] in the introduction and related work sections.

- I would love to see empirical evidence for the statement in L337: "we observe that the weight coefficients of a modality align linearly with the Gaussian it ultimately generates". This is a quite interesting finding that would be much better supported with data.

- Authors claim multiple times (for example in L028, L104, and L536) that they are the first to use 3DGS to model RGB and thermal images. This statement is actually not true and should be revised. The authors of [5] already presented a 3DGS-based method that incorporates RGB and thermal data (see Section 4.5 of the main paper and Section 4 of supplementary).

- In Table 1, the following open-source multi-view RGB+thermal datasets are missing: ThermalMix proposed in [1], the datasets presented in [2], and MVTV proposed in [5]. Why is the proposed dataset better than these datasets? Also, it would be good to provide specific criteria for how multi-view consistency and content richness is evaluated across datasets. This would help clarify the comparison and justify any claims about the proposed dataset's advantages.

- With the high number of recently published RGB+thermal datasets, I think it might make sense to evaluate the introduced method on at least one of these datasets, also to mitigate potential dataset bias. 

- In addition, the proposed method could be compared to some more of the recently published NeRF-based methods. There is code available for [2].

- In L097 authors mention that ThermoNeRF [3] reduces RGB rendering quality, while the proposed method improves it (according to Table 3, an average increase of almost 5% in PSNR). I think that is a huge plus for the proposed method; however, this argument would be much stronger if authors would add comparisons with NeRF-based methods to Table 3.

- I would love to see a side-by-side comparison of thermal renderings with and without the smoothness term.

- Although the proposed weight scheduling used in MSMG yields lower storage requirements, the average rendering quality for RGB and thermal data is worse than for OMMG (see Tables 2 and 3; difference higher for thermal renderings, drop of about 1% in PSNR). So, if one wants to maximize rendering quality and therefore uses OMMG, the benefit of having lower memory requirements is gone (see Table 4; OMMG requires about 60% of the storage space of 3DGS in contrast to 8% for MSMG).

- In Section 3.2, three different strategies to obtain camera poses for thermal images are mentioned. What strategy is actually used for the experiments? Also, what strategy performs best? A quantitative comparison would be really helpful, also because MSX seems to perform best (just by looking at Figure 2). The problem I see is that MSX is patented by FLIR, as such it is not available to other cameras. Hence, the practical use of the third strategy is highly limited, and it would be good to know which strategy could be used instead. Do the other strategies in any form make use of MSX images as well?

References:

See above.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Applicability to a Broader Range of Hyperspectral Imaging

While the RGBT images presented involve the addition of a single channel, they can broadly be considered a form of hyperspectral imaging, as they incorporate the infrared spectrum. Could the authors discuss the potential for their multimodal approach to be applied to a broader range of hyperspectral imaging? Additionally, I am interested in the authors' views on whether this approach could be extended to studies involving multi-wavelength, multichannel data, and if it could serve as a basis for hyperspectral data analysis.

2. Requirement for Well-Aligned RGB and Thermal Data and Practical Field Applicability 

The proposed method relies on the availability of well-aligned RGB and thermal data, which may limit practical field applications. In cases where precise data alignment is challenging or unfeasible in real-world scenarios, are there any methods the authors could suggest to mitigate this limitation?
soundness: 3
strengths: 1. Effective Multimodal Learning and Loss Function Design

The paper introduces an initialization strategy for aligning RGB and thermal data, along with a loss function tailored to thermal characteristics, allowing these modalities to learn complementarily. This approach enables each modality to mitigate the limitations of the other, achieving optimal reconstruction performance​.

2. Model Efficiency and Storage Optimization

By applying multimodal regularization to prevent overfitting and reduce redundant Gaussians, the model reduces storage requirements by approximately 90% compared to conventional methods. This enables high-quality 3D reconstruction while also enhancing storage efficiency and rendering speed​.

2. Contribution of the RGBT-Scenes Dataset to the Research Community

The RGBT-Scenes dataset introduced in this paper provides paired RGB and thermal images across diverse indoor and outdoor scenes, offering a valuable resource for thermal-based 3D reconstruction research. This dataset provides a foundation for researchers to test and expand upon the proposed model, thereby promoting advancements in the field​.
summary: This paper proposes an enhanced method that extends the 3D Gaussian Splatting (3DGS) technique through the ThermalGaussian model, allowing for joint learning and rendering of RGB and thermal data. By learning thermal images together with RGB data in a multimodal framework, the approach aims to improve 3D scene reconstruction capabilities across fields where thermal imaging is valuable, such as in military, industrial, and medical applications. The paper presents initialization strategies for aligning and merging RGB and thermal data, a loss function tailored to the characteristics of thermal imaging, and multimodal regularization techniques to prevent overfitting and optimize storage. Additionally, a new dataset, RGBT-Scenes, is introduced, providing research data that includes both RGB and thermal images.
weaknesses: 1. Lack of Thermal Data Description

While the provision of a dataset is one of the significant contributions of this paper, it lacks sufficient information for readers to understand the characteristics of thermal data. Including details about general characteristics of infrared imaging, as well as optical or photographic settings used during data collection (e.g., imaging wavelength range, whether exposure values were fixed), would help readers better understand the dataset's properties and enhance reproducibility.

2. Lack of Discussion on Limitations

The paper focuses on the model's performance but lacks an explicit discussion of its limitations. For example, it would benefit from exploring conditions under which the proposed multimodal regularization and initialization strategies may not perform well. Including such a discussion would allow for a more realistic assessment of the model's applicability and provide important insights for researchers who may wish to build upon this work.

3. Lack of Discussion on Applicability and Future Work
Although the multimodal learning approach for RGBT data proposed in this paper is academically valuable, the discussion on future work is somewhat general. The paper mentions super-resolution and dynamic scene reconstruction as directions for further research, but these are relatively standard research topics rather than areas specifically tailored to advancements in thermal 3D reconstruction.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: In the paper, the authors say that thermal imaging is practical task and 3D reconstruction gives significant applications which 2D images cannot provide. The reviewer wonders what is the specific case of that thermal 3D reconstruction is practical than 2D thermal image? There is a logical jump in the connection between the two. This context would be added at the related work 2.1 section.

In L. 314, the paper denotes thermal equilibrium, and it would be nice to explain what this means.

In the related work section, both of ThermoNeRF and ThermalNeRF are denoted but why only ThermoNeRF is compared to your method?

The reviewer suggests that comparison to existing thermal 3D reconstructions and to datasets used in previous studies, such as ThermoNeRF and Thermal-NeRF, would further emphasize the strength of the method itself.
soundness: 3
strengths: The paper presents new thermal-RGB datasets which will be helpful in future research.

The method shows quality improvement not only for thermal reconstruction but also for RGB reconstruction.

The method that adds thermal reconstruction on 3DGs is effective and straightforward.
summary: ThermalGaussian is the first work with thermal 3D reconstruction with 3DGS representation.

The paper achieves thermal and RGB 3D reconstruction simultaneously, and improves the rendering results both of color and thermal images.

Finally, the paper reduces the model storage cost by 90%.
weaknesses: The paper proposes three multimodal training methods, which are MFTG, MSMG, and OMMG. The reviewer is confused about the pros and cons of each model, and users are confused about which model to choose. In the quantitative evaluation, the best performing models are distributed across each scene, with no model performing overwhelmingly well.

In the three multimodal training methods, the insight about why these three design are needed is lacked. In L. 283, the authors say “thermal Gaussian training in the second phase may not fully leverage the information from the color modality”, and there is no evidence about this.

There is no discussion about the limitations.

In L.448, there is a type. “Ours” → “ours”

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please respond to the points made in the Weaknesses section.
soundness: 3
strengths: The paper investigates an interesting problem of incorporating thermal information into 3D reconstruction by improving on the prevalent 3D Gaussian Splatting method. The presentation is clear and easy to follow. The results are also interesting in that both RGB and thermal reconstruction have improved.
summary: This paper studies the problem of scene reconstruction with both RGB and thermal image inputs. Specifically, 3D Gaussian Splatting is adopted as the baseline method as a replacement to NeRF. To equip 3D Gaussian Splatting with the ability to also learn 3D thermal representations, this paper proposes a series of strategies for multimodal Gaussian reconstruction, including multimodal initialization, three different thermal Gaussians, constraints specific to thermal modalities, and multimodal regularization. The paper also introduces a new dataset designed for thermal 3D reconstruction. 

Experiments demonstrate the advantage of the proposed method in the quality of reconstruction, in both RGB and thermal modalities.
weaknesses: I do not hold critical concerns to this paper. However, there are some issues that I'd be happy to see clarified. 

In the Multiple Single-Modal Gaussians strategy, it is stated that "Each Gaussian model is influenced
not only by its corresponding input modality but also by others." But how so? RGB modality and thermal modality seem to be uncorrelated to me since 3D Gaussians are trained separately for each. 

Regarding experiments, is there a specific reason that Thermal-NeRF has not been included in the comparison?

Meta-review: The paper introduces ThermalGaussian, a method that extends 3D Gaussian Splatting (3DGS) for joint RGB and thermal 3D scene reconstruction. It proposes three multimodal training strategies, thermal-specific regularization techniques, and a new dataset (RGBT scenes) to achieve improved RGB and thermal rendering quality while reducing storage costs by ~90%. 

The strengths of the work are its extension of 3D Gaussian Splatting for joint RGB and thermal 3D reconstruction, introducing multimodal training strategies and thermal-specific regularization to improve rendering quality and efficiency. It significantly reduces storage costs while improving RGB and thermal image performance, especially in challenging scenarios such as low light. The newly introduced RGBT scenes dataset is a valuable resource for the community.

Reviewers found that the comparisons with recent methods like Thermal-NeRF and datasets such as ThermalMix is needed. Also the paper does not sufficiently address the practical limitations of the proposed method, such as challenges in real-world data alignment, and provides limited discussion of the dataset characteristics and applicability.

Decision: Accept (Poster)