High-Dynamic Radar Sequence Prediction for Weather Nowcasting Using Spatiotemporal Coherent Gaussian Representation
Authors: Ziye Wang
Yiran Qin
Lin Zeng
Ruimao Zhang
Venue: ICLR 2025 Oral (2025)
Keywords: 3D Gaussian
Dynamic Reconstruction
Radar Prediction
Weather Nowcasting
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Weather nowcasting is an essential task that involves predicting future radar echo sequences based on current observations, offering significant benefits for disaster management, transportation, and urban planning. Current prediction methods are limited by training and storage efficiency, mainly focusing on 2D spatial predictions at specific altitudes. Meanwhile, 3D volumetric predictions at each timestamp remain largely unexplored. To address such a challenge, we introduce a comprehensive framework for 3D radar sequence prediction in weather nowcasting, using the newly proposed SpatioTemporal Coherent Gaussian Splatting (STC-GS) for dynamic radar representation and GauMamba for efficient and accurate forecasting. Specifically, rather than relying on a 4D Gaussian for dynamic scene reconstruction, STC-GS optimizes 3D scenes at each frame by employing a group of Gaussians while effectively capturing their movements across consecutive frames. It ensures consistent tracking of each Gaussian over time, making it particularly effective for prediction tasks. With the temporally correlated Gaussian groups established, we utilize them to train GauMamba, which integrates a memory mechanism into the Mamba framework. This allows the model to learn the temporal evolution of Gaussian groups while efficiently handling a large volume of Gaussian tokens. As a result, it achieves both efficiency and accuracy in forecasting a wide range of dynamic meteorological radar signals. The experimental results demonstrate that our STC-GS can efficiently represent 3D radar sequences with over $16\times$ higher spatial resolution compared with the existing 3D representation methods, while GauMamba outperforms state-of-the-art methods in forecasting a broad spectrum of high-dynamic weather conditions.

Published as a conference paper at ICLR 2025
HIGH-DYNAMIC RADAR SEQUENCE PREDICTION FOR WEATHER NOWCASTING USING SPATIOTEMPORAL COHERENT GAUSSIAN REPRESENTATION
Ziye Wang Yiran Qin", Lin Zeng", Ruimao Zhang-!
1 Sun Yat-sen University The Chinese University of Hong Kong, Shenzhen
3 Guangzhou Meteorological Observatory
ABSTRACT
Weather nowcasting is essential task that involves predicting future radar echo sequences based on current observations, offering significant benefits for disas- ter management, transportation, and urban planning. Current prediction meth- ods are limited by training and storage efficiency, mainly focusing on 2D spatial predictions specific altitudes. Meanwhile, 3D volumetric predictions each timestamp remain largely unexplored. To address such challenge, we introduce a comprehensive framework for 3D radar sequence prediction in weather now- casting, using the newly proposed Spatio Temtoral Coherent Gaussian Splatting (STC-GS) for dynamic radar representation and GauMamba for efficient and ac- curate forecasting. Specifically, rather than relying on 4D Gaussian for dynamic scene reconstruction, STC-GS optimizes 3D scenes at each frame by employing group of Gaussians while effectively capturing their movements across consec- utive frames. It ensures consistent tracking of each Gaussian over time, making it particularly effective for prediction tasks. With the temporally correlated Gaus- sian groups established, we utilize them to train GauMamba, which integrates a memory mechanism into the Mamba framework. This allows the model to learn the temporal evolution of Gaussian groups while efficiently handling a large vol- ume of Gaussian tokens. As a result, achieves both efficiency and accuracy forecasting a wide range of dynamic meteorological radar signals. The experi- mental results demonstrate that our STC-GS can efficiently represent 3D radar sequences with over 16x higher spatial resolution compared with the existing 3D representation methods, while GauMamba outperforms state-of-the-art methods in forecasting broad spectrum of high-dynamic weather conditions. Code and dataset are available on our project page
INTRODUCTION
Weather nowcasting is a critical component of metcorological forecasting that focuses on predicting short-term weather conditions based on real-time observations. It supports many metcorological applications. such as precipitation forecasting Shh et al.] (2017): : Yu et al. (2024), extreme weather warnings /Zhang et al. (023), and hurricane prediction Li al. (2022), and plays vital role in various applications, including disaster management, transportation safety, and urban planning. Ac- curate nowcasting relies on timely and precise predictions of rapidly changing weather patterns. One of the key pathways for weather nowcasting is radar sequence prediction, which focuses on forecasting future radar echo frames from current observations.
Recent advancements have primarily concentrated on predicting 2D radar sequences at specific alti- tudes Wang al. (2017); Gao et al. (2022b): Wang et al. (2023): al. (2023); Yu et al. (2024). However, the atmospheric system develops in three spatial dimensions, where the intensity of echoes from low-altitude clouds is closely linked to surface convective activity, while higher-altitude clouds
Work completed by Ziye Wang as a Research Assistant under the supervision Prof. Ruiman Zhang. z ziye. . wangl 9 998@cus cuuutuuuai look. com
Corresponding author is Prof. Ruimao Zhang. ru mao. zhang@ eeg.org

---

Published as a conference paper at ICLR 2025
The
235
2D Predectio Model
Vatkal Disannanr
Darrd
To x b.x.35 x 1201
Estead 2D Opesno
(Tee x x x. 120 120)
Sipur (Mt)))
94
3D Prediction Mode!
Rodtiii
36x 512x
X49152 16]
GuiManba
Tonaer HID Totar Ceuriiz
(b)
Figure 1: Comparisons between 2D and 3D radar sequence prediction frameworks. (a) Current 2D methods mainly focus on 2D spatial predictions at specific altitudes. Limited by training and storage efficiency, it is challenging size of the storage occupied by a single frame input is calculated according to the resolution of two datasets. The resolution of MOSAIC is 36 384 512, 2D input is 384 X 512, 3D input is
extend these methods 3D architectures directly. (b) The
36 x 384 x 512, and the size of 3D Gaussians is 49, 152 x 11. The resolution of NEXRAD is 36 512 x 512 with 6 channels, the inputs are x 512 512, 6 x 36 512 x 512 and 49, 152 16.
indicate potential severe weather. Relying solely on 2D radar echoes at specific altitudes overlooks the dependencies between different heights and fails to capture the complete structure of the system. As shown in Fig. [ a straightforward way is to extend the 2D predictive model to 3D. However, this transition requires the model to receive higher-dimensional features, leading to increased memory usage and computational complexity, which further restricts high-resolution scaling.
To address such issue, present comprehensive framework for effective 3D radar sequence prediction. This framework utilizes the newly proposed SpatioTemporal Coherent Gaussian Splat- ting (STC-GS) for dynamic radar representation and incorporates GauMamba for accurate and effi- cient forecasting. Specifically, our STC-GS differs from traditional methods that rely on 4D Gaus- sians for daily dynamic scene reconstruction, where only the main subjects are dynamic and most objects are treated as locally static. In contrast, radar observations of clouds are continuously in motion and changing, without adhering any rigid body or morphological constraints. STC-GS begins by reconstructing the initial frame utilizing group of 3D Gaussians (Kerbi et al. 2023), which are then employed as anchors. By monitoring and aggregating changes to these anchors as subsequent frames are pre-reconstructed, STC-GS effectively captures the underlying motion trends of the radar sequence. The initial Gaussians from the first frame, combined with the series of mo- tion trends extracted from subsequent frames, establish clear temporal continuity. Such scheme ensures the consistent tracking of each 3D Gaussian over time, thereby enhancing its efficacy in develop bidirectional reconstruction pipeline for STC-GS that
predictive tasks. In practice, accurately models the growth and dissipation of radar echoes. This pipeline incorporates dual-scale constraints to preserve the temporal coherence of Gaussian movements across both global trends and local details in the data.
Once we establish temporally correlated Gaussian groups across all of the frames, we adopt them to train the proposed GauMamba. It integrates a memory mechanism into the Mamba framework, allowing model the temporal evolution of Gaussian groups while efficiently managing large volume of Gaussian tokens. This capability is crucial for effectively forecasting wide range of dynamic meteorological radar signals. Overall, our framework not only enhances the accuracy of 3D radar sequence predictions but also improves computational efficiency, making it robust solution for weather nowcasting applications. By leveraging the strengths of STC-GS and GauMamba, we aim to advance the state-of-the- art in meteorological forecasting, providing timely and accurate information for various practical applications.
The main contributions of this work can be summarized as follows. (1) To our best knowledge, this is the first work for 3D-based weather nowcasting by predicting high-dynamic radar sequences. Our framework introduces a novel 3D Gaussian representation termed STC-GS that adeptly captures radar data dynamics, paired with a memory-augmentee network, GauMamba, which learns tempo- ral evolution from these representations forecast radar changes. (2) We propose bidirectional
2

---

Published as a conference paper at ICLR 2025
Gaussian reconstruction pipeline, which is designed to precisely track the motion trajectory of 3D Gaussians along the sequential frames. It incorporates dual-scale constraints to ensure coherence at both global trends and local details, effectively preserving the temporal consistency of Gaus- sian movements. (3) In addition, we collect and organize a novel high-dynamic 3D radar sequence dataset named MOSAIC and reorganize a dataset named NEXRAD. Our experiments demonstrate that GauMamba effectively predicts future 3D radar data with 16x resolution, achieving a 19.7% and 50% reduction in Mean Absolute Error (MAB), and a notable improvement in predicting regions with significant radar signals.
RELATED WORK
2.1 3D GAUSSIAN REPRESENTATION
Recently, Gaussian Splatting- based representations offer real-time rendering with high training effi- ciency and garner considerable research interest:[ang et al. (2023); Diolatzis et al. (2024); [Mallck et al. (2024); Zhou et al. (2024). Motivated by the success of 3D Gaussian Splatting, numerous stud- ies have extended it to real-time dynamic scene reconstruction and rendering. Incremental transla- tion methods Luiten et al. (2023); [Snn et al. (2024) tackle this challenge by initializing each frame based on the preceding one, leveraging motion constraints to enforce temporal coherence. Other approaches extend Gaussian representations to 4D space-time Yang al. (2024a) or model global scene deformations with neural networks or polynomial Wu et al. (2024): Yang et al. (2024b); Li eet al. ) enabling efficient reconstructions. Despite these advancements, such methods face limitations in accurately capturing complex or discontinuous scene dynamics. Moreover, they often rely on fixed color and opacity settings ensure consistency.
Recent studies Shen et al. (2024); Yi et al. (2024); Ziwen et al. (024); Zhang et al. (2025) have explored integrating Mamba Gu Dao (003); Dao Gu (2024) Transformer architectures with 3D Gaussian representations, focusing on reconstructing 3D Gaussians from single or multi-view images. However, these approaches are limited their ability to model spatiotemporal dynamics, as the lack of memory mechanisms. Our method leverages sequences of 3D Gaussians to represent the temporal evolution of 3D radar echo data and employs a Gemnory-Aggmette GauMamba model to effectively integrate information from preceding frames.
2.2 SPATIO-TEMPORAL PREDICTION
Spatio-temporal prediction is crucial in meteorological forecasting, requiring models to capture both spatial patterns and temporal dynamics. U-Net architectures using 2D 31D CNNs have been ap- plied to tasks like precipitation nowcasting, Arctic Sea ice prediction, and ENSO forecasting Veil- lette et al. (2020); Andersson al. (202T); Ham et al. (2019), though they struggle with tempo- ral dependencies. To improve this, methods such as ConvLSTM [Shi et al. (015)). ConvGRU Shi al.) (017)). and PredRNN Wang al.) (2017) 2023) integrate memory mechanisms
better han- dle spatio-temporal correlations. E3D-LSTM Wang et al. (2019) combines 3D CNNs with LSTM for long-term forecasting, while PNy Guen Thome (020) embeds physical constraints into models. Sim Gao et al. (2022a) simplifies prediction using convolutional encoders and decoders, while transformer-baat rangerange dependencies. However, deterministic models often struggle with prediction blur and fail to capture the stochastic nature weather systems. To address this, diffusion based models Gao et al. (2023); Yu al. (2024) have been introduced to estimate spatio-temporal uncertainty.
models Pathak et al. (2022): ; Bai et al. (2022); Gao al. (2022b) capture
3 METHODOLOGY
3.1 PRELIMINARY
3D Gaussian Splatting (3DGS). Gaussian Splatting Kerbi et al. (2023) uses a collection of 3D Gaussians to represent 3D objects or scenes. Each Gaussian is characterized by a position E 3R3 scaling factor 8 E 3 and a rotation quaternion g E R For rendering RGB images, opacity value R and color feature R also included, with optional spherical harmonies for
3

---

Published as a conference paper at ICLR 2025
Frame 0
Frumct
Frame H+1
Raw 3D Raler Dutu
kk(
Me proor
ID STC-Gs
GuuManba
Sr
Seer
Profiction on STC
GnuMamba
Stucil
Tripkne Flow
(1)
Figure 2: Overview of our 3D prediction framework based on STC-GS and GauMamba. (a) STC-GS can effectively compress the size of 3D data while fully representing it. GauMamba is memory-augmented predictive model that leverages STC-GS for effective and accurate predictions. The STC-GS at Frame is imput into the GauMamba to predict set of DiffGaussians, AG representing the differences between G++1 and Gn- This process is applied iteratively from Frame 0 Frame Tin Tout 1. (b) In the process of radar reconstruction, dual-scale constraints implemented to capture both the global trends and the local details present in the Gaussian motions.
modeling view-dependent effects. In this way, a group of Gaussians can be represented as θ, where θ {p;, S; qs, Qi c;) denotes the parameters of the i-th Gaussian.
The rasterization stage involves calculating the Gaussian's 's contribution to individual pixels. Initially, each Gaussian is projected into the camera's coordinate. Following this, the renderer divides the screen into tiles and culls the Gaussians that fall outside the view frustum. Lastly, the renderer conducts alpha blending in depth-sorted order within the view space for each pixel.
3.2 THE OVERALL FRAMEWORK
As illustrated in Fig. we introduce a comprehensive framework for the prediction of 3D radar echo sequences in weather nowcasting. The overall framework includes two main components. The first part is the newly proposed SpatioTemporal Coherent Gaussian Splatting (STC-GS) for Mamba
dynamic radar data representation. The second is the GauMamba, memory-augment network, coupled with STC-GS representations, for efficient and precise forecasting. In practice, unlike 3DGS, which is designed for rendering real-world RGB scenes, STC-GS is specifically tailored for radar observations, whose definition is provided Sec. 3.3.1
Given the high-resolution 3D radar sequence, first require to re-represent each frame F, CC into a group of 3D Gaussians, where H and W donate the vertical and horizontal spa- tial resolution, and C denotes the number of measurement channels. Unlike the traditional 3DGS technology used for scene reconstruction, our goal is to obtain a new representation of 3D radar data based on reconstruction further accomplish downstream prediction tasks. Therefore, the proposed STC-GS should maintain the spatiotemporal coherence based on reconstruction. In this way, we in- troduce bi bi-directional reconstruction scheme to achieve it, where a backward pre-reconstruction first conducted to preliminarily retain the growth and dissipation of the Gaussians along the temporal dimension, and the forward reconstruction is carried out by using the pre-reconstruction results as the reference to enhance the accuracy. Please refer to Sec. 3.3.2 for more details.
Next, the prediction based on 3D radar can be transformed to predict the variation of STC-Gaussians along frames. We coupled the memory mechanism with Mamba achieve linear time training and inference based on Gaussian Groups and the capacity to model the temporal evolution Gaussian groups. Our proposed GauMamba takes parameters of 1-th frame Gaussians G, inputs and predicts the difference between the G.+1 and Go. see 3.4 for more details.
3.3 SPATIOTEMPORAL COHERENT GAUSSIAN REPRESENTATION
The goal of reconstructing 3D radar data into a set of 3D Gaussians is to obtain a feature-dense repre- sentation that nearly losslessly represent the original 3D radar data. To achieve this, propose
4

---

Published as a conference paper at ICLR 2025
differentiable reconstruction pipeline motivated by 3D Gaussians Splatting (3DGS) method Kerbl et al. (2023). Instead of relying on Structure from Motion (SfM) in 3DGS, our STC-GS randomly samples points from the original 3D radar data to initialize the positions and radar features of the 3D Gaussians. We then design a differentiable radar profile renderer capable of rendering radar cross-sections from any angle, including horizontal, vertical, and oblique views. Readers can refer to Appx. A for more details.
3.3.1 DEFINITION OF 3D RADAR GAUSSIAN IN STC-GS
Given the specific location a E I of the radar frame, we use the slightly perturbated coordinates indices of the corresponding pixels the initial positions of the 3D Gaussians, avoid falling into saddle points in the optimization space. Unlike using RGB values to reconstruct real-world scenes, reconstructing radar observation data requires dimensional features f E IR which indicates radar echo intensity, horizontal reflectivity factor, spectrum width, etc. The number of radar features N depends on the specific radar dataset. Additionally, defining a 3D Gaussian requires an optimizable full 3D covariance matrix Σ. It can be decomposed into rotation matrix R and scaling matrix S for by:
E = RSS RD
In practice, the rotation matrix R and the scaling matrix S are represented as a rotation quaternion R' and scaling factor E 3 respectively. In summary, for the i-th Gaussian, the optimizable attributes are given by θ { fi, 8i, q:). We then optimize each Gaussian by minimizing the error between the rendered results and the radar profiles from the origin 3D radar data.
3.3.2 BIDIRECTION RECONSTRUCTION SCHEME.
Adding or removing Gaussians during reconstruction disrupts spatiotemporal consistency, making impossible track Gaussians throughout the sequence. However, simply disabling adaptive density control significantly degrades reconstruction quality, impairing the capture of cloud dynamics in radar sequences. To address these issues, we propose bidirectional reconstruction strategy that preserves spatiotemporal consistency without altering the Gaussian set. This approach effectively captures frame-by- by-fframe motion trends from raw data while accurately modeling the growth and dissipation of clouds in highly dynamic radar sequences.
The bidirectional reconstruction strategy consists of stages: backward reconstruction and for- ward reconstruction. In the backward reconstruction stage, we pre-reconstruct from timestamp to timestamp O. Only the positions of the Gaussians are optimized using the local detail and global trend constraints. Components that persist or gradually dissipate over time are preserved, and each Gaussian's 's position is updated according to cloud motion at each timestamp. By the time backward reconstruction reaches timestamp 0, it effectively incorporates information from future frames, bringing this information back to the initial frame. In the forward reconstruction stage, iteratively reconstruct each frame from timestamp 0 to timestamp T. We uniformly sample from the Gaussians obtained during the backward reconstruction and the original 3D data at timestamp 0 to initialize using the local detail and global constraints, then optimize all Gaussian parameters by additional reconstruction loss.
set of Gaussians. In the coarse reconstruction stage, we optimize the positions
Achieving a spatiotemporally coherent representation requires each Gaussian to precisely track the motion of the region it reconstructs. Besides, radar sequences often exhibit complex and diverse dy- namics, including motion, deformation, growth, and dissipation. Reconstructing such high-dynamic sequences necessitates simultaneous adjustments to all attributes of the Gaussians. Identifying which parameters to modify for optimal error minimization becomes a significant challenge. Thus, the pre- diction based on 3D Gaussians introduces new challenges for the optimization process during the reconstruction phase. To address this, we propose local detail constraint (3D flow constraint) and a global trend constraint (global energy constraint), which introduce 3D motion priors and optimal position distribution priors estimated from the original 3D radar data.
Local Detail Constraint. To introduce 3D motion priors from original 3D radar data and achieve spatiotemporally coherent for each Gaussian, introduce a 3D optical flow constraint, shown in Fig. 2(b). First, we utilize a pre- trairrd 2D optical flow model, RAFT & Deng (2020), to estimate the motion of radar observations across the xoy, xoz, and yoz planes. These 2D flows
5

---

Published as a conference paper at ICLR 2025
11-1
ManbaCRRU
MannbuCiUU,
ManiboGiUU,
24
ManloCCRUU,
MamboGGRU1
c
26
Unver
Linverz
Shekoond Marka BRnck
ManiuCikU,
ManiboGiRU,
Maule
4i-a
c?
MambaGRR
MambuCilU)
ManibGGRU
Linsun
The overall architecture of the GauMamba. (a) Main architecture of GauMamba which consists of multiple stacked MambaGRU Block. (b) Detailed architecture design of MambaGRU.
Figure 3:
then fused into a pseudo 3D flow within an xyz grid to approximate motion in 3D space. We take the 3D flow grid closest each Gaussian as the reference flow, and constrain the distance between the displacement of the Gaussians A E and the reference 3D flow A to ensure that the Gaussians move tandem with the corresponding cloud formations:
Liocat = A A 11
However, since the 3D optical flow is estimated and fused from 2D images, it may not always be accurate. Moreover, employing an optical flow model pre-trained on real-world image sequences to estimate the flow of radar sequences could introduce inductive bias, which may accumulate over iter- ations. As a result, while the 3D flow constraint helps ensure that the Gaussians generally align with the direction of actual cloud motion, it cannot guarantee that each Gaussian is optimally positioned in every iteration.
Global Trend Constraint. From the original sparse 3D radar observation data, we can infer an optimal distribution of group of Gaussians that accurately reconstructs the 3D data. Intuitively, the probability of Gaussians existing should be high in non-null regions of the 3D radar data, gradually decreasing zero from the boundary areas between non-null and null regions to fully null regions, as shown in Fig. b). We approximate the unnormalized optimal distribution of Gaussians using smoothed function with threshold suppression. Specifically, we apply Gaussian kernel-based smooth convolution to the original radar data and clamp values that exceed predefined threshold т, the estimated unnormalized optimal distribution P( P(F) can be expressed mathematically as follows:
P(F) min IC()),))
Here, F is the original radar data, denotes the convolution operator and KC(-) is the Gaussian ker- nel. For the current group of 3D Gaussians, we can derive their corresponding position probability densities by reusing the differentiable renderer. Specifically, each 3D Gaussian treated as a normal distribution centered at its current location, and the renderer is used to obtain the position probabil- ity densities of the current Gaussians. Then, utilizing the energy function in energy-basee learning LeCun et al.] (2006), we can define a simple yet effective method to measure the similarity between two distributions:
Lgtobel RR(()) ?(
Here, Rp denotes the reused renderer for distribution estimation and Rr(g) is the estimated distri- bution based on current Gaussians G. By minimizing this energy function, we can ensure that each Gaussian in the current set positioned at more optimal initial location.
3.4 MEMORY AUGMENTED MAMBA PREDICTIVE MODEL.
We now introduce proposed Mamba-based prediction model termed GauMamba, incorporated with the spatiotemporal coherent Gaussian representation for high-dynamic 3D radar sequence pre- diction. After reconstructing the raw radar data into sequence of Gaussian groups, we aim train the predictive model based on the 3D Gaussian representations.
Thanks to STC-GS, the prediction of 3D radar sequence can be transformed into the prediction of variation of Gaussian group along the temporal dimension. We leverage the standard Mamba ar- chitecture Gu & Dao 2023): Dao & Gu (2024) and incorporate the GRU memory mechanism to

---

Published as a conference paper at ICLR 2025
predict parameters of future Gaussians. The former effectively handle large number of Gaus- sian tokens as input, while the latter can effectively learn the spatiotemporal relationships between adjacent Gaussian Groups. As shown in Figure a), the proposed GauMamba consists of multiple stacked MambaGRU blocks. The I-th layer MamhaGRU block at timestamp & takes two inputs: the output embedding from the previous layer C. and the hidden state used for memory from the previous timestamp H1 The block then outputs the updated Gaussian embeddings C₁, along with the updated memory hidden state Hi to be passed the next timestamp.
Training objective. In our 3D Gaussians prediction training, we parameterize our model fe to predict DifiGaussian AG, = Gt Go for 1,2 .in + Touts using:
Tint1
Cpred
1, 2 +
Σ ( H,-1) 1 T=Ti+2
t=1
Here, Tin is the length of given observation sequence, Tout is the length of predicted sequence, and G- Go @ A Notably, since we only have access to true states of the first Tin Gaussian Groups, there no true value for timestamps beyond Tin-1 I to be used as input for prediction. Thus, for S T. + 2. --1 is estimated based on the predicted result - from the previous timestamp and the initial state Go- Specifically,
tokenize 9o to obtain its embedding, and then combine with the embedding of A9,-1. which produced by the final MambaGRU block from the previous time step. This summation results in the estimated embedding for timestamp ( 1.
4 EXPERIMENT
More details of datasets, implementation, metrics, and experimental setup are in Appendix C
4.1 DATASET
Prame 0
Prome
Phime 20
Foamr
Derai
The datasets used in this study include NEXRAD and MOSAIC. NEXRAD] com- prises radar observations the U.S., with 3D reflectivity data sampled at 5-minute intervals. Seven radar features, such reflectivity, azimuthal shear, differential re- flectivity, and so on, are included. MOSAIC records radar observations of storms in Guang- dong, China, with 6-minute intervals, focusing solely on intensity data of radar echoes. Both datasets are preprocessed to ensure consistent vertical spacing and are divided into training, validation, and test sets. The prediction task in- volves forecasting up to future frames based on 5 observed frames. For further information, please refer to the supplementary material C.I
30G8mmam
severe storms in
4DGS
Defiem-ADOS
Oun
Figure 4: Qualitative results of reconstruction.
4.2 EXPERIMENT RESULTS
Resconstruction. First, we verify the effectiveness of our proposed re-represent method on the high- dynamic 3D radar dataset and compare it with other otate-of-the-art methods designed for dynamic scene reconstruction. Table [ presents the quantitative evaluation result on reconstructing hight- dynamic 3D radar sequences. The competing methods adhere their original configurations, with certain parameters fixed during optimization. In contrast, our approach allows all parameters to be freely optimized, posing a significantly more challenging task. Despite this, our method achieves superior performance across all metrics. As illustrated in Figure E our proposed method preserves more details and maintains consistent accuracy throughout entire sequence.

---

Published as a conference paper at ICLR 2025
Table I: Comparision of reconstruction in NEXRAD
Model
MAEx x
PSNR(dB)
SSIM
LPIPS*
LPIPS Radar
3DGStream Sun et al. (2024)
0.019
38.133
0.954
0.091
0.902
4DGS Wu ci al. 20244
0.028
35.731
0.933
0.135
0.623
Deform-3DGS Yang et al. 2022bb
0.029
35.027
0.931
0.14 1.141
0.578
Ours
0.014
40.262
0.970
0.057
0.123
Prediction. The large and cumbersome 3D tensors present a significant challenge in extending radar echo predictions from 2D to 3D, thereby limiting the seal-
GPU Memorry Cossumed (GB))
Estimuted
ability of models for high-resolution predictions. Fig.
5 illustrates the GPU memory requirements for feature extraction and prediction using methods based on 3D data, including ConvGRU, PhyDNet, SimVP, and Dif- fCast, alongside proposed feature-dense 3D Gaus- sian prediction approach. Scaling these radar echo pre- diction methods, which use raw 3D data input, high-resolution training and deployment is both difficult and impractical, the time and space complexity scales quadratically. i.e., O(N²), with respect to the horizontal resolution of proposed Gaussian-based method is independent of the input or output resolution, scaling linearly only with the number of Gaussian primitives. As shown in Fig. 5 the memory usage our Gau- Mamba method is evaluated with fixed number of Gaussian primitives tailored for high-resolution images, balancing computational efficiency and reconstruction precision. When predicted at lower resolutions, e.g., N 256, fewer Gaussian primitives required, leading to reduced memory consumption.
Harizontal Resolution
ConvGRU
PhyySSet
SimVP
Difti
Ours
Figure 5:
Memory usage of different methods with various input resolutions.
contrast, the memory usage of our
With the available computational resources of A100 80G GPUs, methods that use 3D data as input cannot be effectively trained at horizontal resolution of 512 x 512. To facilitate a fair comparison with our proposed GauMamba, at horizontal resolution of 128 x 128, and trained GauMamba at 512 x 512, while maintaining consistent vertical resolution. Then, we upsampled the predictions from ConvGRU, PhyDNet, Sim VP, and DiffCast to 512 x 512 for a fair evaluation.
trained ConvGRU, PhyDNet, Sim and DiffCast
The quantitative performance metrics of 3D radar datasets are presented Table and 3 re- spectively. In general, our proposed GauMamba with STC-GS as input outperforms all existing models based on raw 3D radar data, even if predicting an actual higher resolution. The Gau- Mamba achieves 12. 1%, 69.0% and 4.8%, 101.1% improvements in CSI-20 and CSI-30 with 4 x 4 pooling MOSAIC and NEXRAD dataset, respectively. We also present the qualitative results in Fig. 6 The prediction results of our proposed method contain more details and the high values of the significant regions are preserved. This is because the re-represented Gaussian with signifi- cant features is prioritized to be tracked and predicted in order minimize our optimization objective. In addition, the results of Mamba and our GauMamba demonstrate that the approaches and experiments developed within our proposed framework, which redefine the 3D prediction task by first re-representing the sequences with 3D Gaussians and then predicting their future evolution, significantly outperform traditional methods, highlighting the effectiveness and robustness of our reformulation.
4.3
ABLATION STUDY
4.3.1 RECONSTRUCTION
Reconstruction w/o local details. The flow constraint introduces pseudo-3D motion flow that guides the reconstruction process. Its absence leads to dynamic Gaussians that struggle to accurately track the movement of the reconstructed elements. This deficiency results in 32.6% increase in
8

---

Published as a conference paper at ICLR 2025
Forme 10
Franc 15
Foanc 20
Frame 25
Desii
Frame 10
Frume 15
Frane 20
Frame
Denil
C
PlyDNei
SinVP
DiffCast
GzuMambo
Figure 6: Qualitative results of 3D radar prediction. (a) The prediction results in MOSAIC dataset. (b) The prediction results in NEXRAD dataset. Detail comparisons are plotted in the last column, respectively. Our proposed framework can more effectively predict local changes in detail, which may determine specific weather change trends.
Table 2: Experiment results in MOSAIC
Model
MAE
SSIM
LPIPS
LPIPS
Radar Radar
CSI-20pool4
CSI-30 poo14
CSI-40 Pool4
ConvGRU
1.728
0.621
0.303
4.837
PhyDNet
0.910
0.810
0.244
1,451
0.294
0.108
0.002
SimVP
0.890
0.835
0,270
3,516
0.264
0.075
DiffCast
1.878
0.355
0.433
2.216
0.305
0.126
0.006
Mamba
0.750
0.894
0.164
0.777
0.293
0.166
0.055
GauMamba
0.714
0.897
0.157
0.741
0.342
0.213
0.062
the MAE score. Reconstruction w/o global trends. The energy-based constraint plays a crucial role during the initial stages by ensuring that the set of Gaussians achieves a more appropriate coarse-level spatial distribution. This constraint also helps mitigate error accumulation that can arise from inaccurate flow estimations. After removing the energy-based iteration phase, we keep the total number of iterations constant. Reconstruction w/o bidirectional reconstruction. Before the forward reconstruction phase, a backward reconstruction conducted to optimize the spatial distribution of the Gaussians. Omitting this reverse pre-reconstruction stage results in the loss of the ability to propagate information about areas that emerge in the future back to the first frame. Consequently, the initialization of Gaussian positions becomes inadequate, leading significant decrease accuracy.
4.3.2 PREDICTION
GauMamba w/o Mermory. Removing the memory mechanism results in a vanilla Mamba, which not suitable for the current prediction task. To evaluate the effectiveness of the memory mech- anism, we incorporate DLinear Zeng et al. (2023), simple yet effective prediction method that does not rely on memory. Specifically, we split a vanilla Mamba into two symmetric parts: an en- coder and decoder. The encoder generates embeddings from historical observations, which are subsequently fed into DLinear to predict future embeddings. The decoder then produces the pre- dicted parameters of Gaussian. However, Mamba with DLinear does not surpass method with a 20.2% increment of MAE score. Our memory mechanism can be updated temporally while con- sidering global embeddings. In contrast, DL inneear infers future embeddings based solely local

---

Published as a conference paper at ICLR 2025
Table 3: Experiment results in NEXRAD
Model
MAE+
SSIM
LPIPS
LPIPS
CSI-20
CSI-30
CSI-40 Poot4
Radar
ConvGRU
0.006
0.819
0.205
1.621
0.306
PhyDNet
0.017
0.373
0.320
2.058
0.311
0.089
0.002
SimVP
0.066
0.379
0.481
2.925
0.085
0.088
0.018
DiffCast
0.157
0.004
0.932
4.057
0.049
0.021
0.021
Mamba
0.004
0.899
0.129
0.699
0,309
0.165
0.074
GauMamba
0.003
0.900
0.126
0.665
0.326
0.179
0.078
Table 4: Ablation Study in Reconstruction
Method
MAE x 100
PSNR(dB)
SSIM
LPIPS*
Reconstruct w/o flow (local details)
1.947
37.521
0.953
0.089
Reconstruct w/o energy (global trends)
1.497
40.049
0,969
0.059
Reconstruct w/o bi-direction
3.245
32,746
0.921
0.152
Full Model
1.468
40.262
0.970
0.057
Table 5: Ablation Study in Prediction
GauMamba
ME
MAR
SSIM
LPIPS
CSI-20
CSI-30
CSI-40
w/o Memory
-0.445
0.858
0.883
0.170
0.224
0.122
0.035
w/o GRU
-0.374
0.743
0.895
0.161
0.289
0.165
0.052
w/o Sort
-0.532
0.880
0.883
0.183
0.179
0.085
0.025
Full Model
-0.103
0.714
0.897
0.157
0.342
0.213
0.062
embeddings. GauMamba w/o GRU. We removed the reset gate and update gate from the GRU, while retaining the memory mechanism. The hidden states, which store historical information, are directly concatenated with the current Gaussian embeddings. In this configuration, the MAE score increases 4.1%. which is lower than that of GauMamba w/o Mermory. highlighting the effective- ness the memory mechanism. However, removing GRU reduces the flexibility in updating the memory. GauMamba w/o Sort. Morton sorting aggregates spatially adjacent points in the 1D quence. Without Morton sorting, the Gaussians are input into GauMamba their original unsorted order, as determined during the reconstruction stage. Although each Gaussian maintains its position consistently across frames, significant performance degradation is observed. This suggests that spatially unstructured sequences negatively impact the feature extraction capabilities.
CONCLUSION
In this work, we introduced novel framework for 3D radar sequence prediction in weather fore- casting, addressing the limitations of current 2D spatial prediction methods. Our proposed Spa- tioTemporal Coherent Gaussian Splatting (STC-GS) and GauMamba effectively capture dynamic radar signals while maintaining high efficiency and accuracy. Experimental results demonstrate that STC-GS achieves superior reconstruction accuracy compared to existing 3D representations, while GauMamba outperforms state-of-the-arr models in predicting dynamic weather conditions. In general, we provide general solution framework for 3D prediction, paving the way for future advancements in this domain. However, the current scope of our approach is specialized, focusing primarily radar-based weather nowcasting, with limited exploration of broader applications. We are actively working to enhance its generalization and simplify the pipeline for broader applicability to dynamic 3D scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is partially supported by the Natural Science Foundation of Guangdong Province (2022A1510011 1346). the Young Scientists Fund of the National Natural Science Foundation of China (62 106154), Shenzhen Science and Technology Program (JCYJ20220818100) 8103001002), the Guangdong Key Laboratory of Big Data Analysis and Processing. Sun Yat-sen University, China, and the reghpperformance Computing Public Platform (Shenzhen Campus) of Sun Yat-sen Univer- sity. We thank to Huaiyu Li from Guangzhou Meteorological Observatory and Haowen from Quzhou Meteorological Bureau for their valuable support in the MOSAIC dataset.
REFERENCES
Tom R Andersson, Scott Hosking, María Pérez-Ortiz, Brooks Paige, Andrew Elliott, Chris Russell, Stephen Law, Daniel C. Jones, Jeremy Wilkinson, Tony Phillips, et al. Seasonal arctic sea ice forecasting with probabilistic deep learning. Nature communications, 12(1):5124, 2021.
Cong Bai, Feng Sun, Jinglin Zhang, Yi Song, and Shengyong Chen. Rainformer: Features extrac- tion balanced network for radar-based precipitation noweasting. IEEE Geoscience and Remote Sensing Letters, 19:1-5, 2022.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learn- ing. 2024.
Department of Atmospheric Sciences, Texas A&M University and School of Meteorology, Univer- sity of Oklahoma. Gridrad three-dimeesioonal gridded nexrad wsr- reflectivity and velocity spectrum width, 2021. URL httpt:tttt://ot ucar
Stavros Diolatzis, Tobias Zirr, Alexander Kuznetsov, Georgios Kopanas, and Anton Kaplanyan. N-dimensional gaussians for fitting of high dimensional functions. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z. Li. Simvp: Simpler yet better video predic- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3170-3180, 2022a.
Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Yuyang (Bernie) Wang, Mu Li, and Yan Yeung. Earthformer: Exploring space- time transformers for earth system forecasting. In NeurlPS 2022, 2022b.
Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang, Xiaoyong Jin, Danielle C. Maddix, Yi Zhu, Mu Li, and Yuyang Wang. Prediff: Precipitation nowcasting with latent diffusion models. Ad- vances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00722 2023.
Vincent Le Guen and Nicolas Thome Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
Yoo-Geun Ham, Jeong-Hwan Kim, and Jing-Jia Luo. Deep learning for multi-year enso forecasts. Nature, 573(7775):5685722 2019.
Cameron R Homeyer and Kenneth P Bowman. Algorithm description document for version 4.2 of the three-dimensional gridded nexrad wsr-88d radar (gridrad) dataset. University of Oklahoma- Texas A&M University Tech. Rep, 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., -3:4-:139 14, 2023.
11

---

Published as a conference paper at ICLR 2025
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, al. A tutorial on energy- based learning. Predicting structured data, 100). 2006.
Xin Li, Zhaoxia Pu, Jun A. Zhang, and George David Emmitt. Combined assimilation of doppler wind lidar and tail doppler radar data over a hurricane inner core for improved hurricane prediction with the NCEP regional HWRF system. Remote. Sens., 14(10))2367, 2022.
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
2024.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians:
Tracking by persistent dynamic view synthesis, 2023.
Saswat Subhajyoti Mallick, Rahul Goel, Berhaard Kerbl, Francisco Vicente Carrasco, Markus Stein- berger, and Fernando De La Torre Taming 3dgs: High-quality radiance fields with limited re- sources. arXiv preprint arXiv:2406. 15643, 2024.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data- driven high-resolution weather model using adaptive fourier neural operators, 2022.
Suman V. Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Rémi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, Rachel Prudden, Amol Mand- hane, Aidan Clark, Andrew Brock, Karen Simonyan, Raia Hadsell, Niall H. Robinson, Ellen Clancy, Alberto Arribas, and Shakir Mohamed. Skilful precipitation nowcasting using deep gen- erative models of radar. Nature, 597(7878):672- 2021.
Qiuhong Shen, Zike Wu, Xuanyu Yi, Pan Zhou, Hanwang Zhang, Shuicheng Yan, and Xinchao Wang. Gamba: Marry gaussian splatting with mamba for single view 3d reconstruction. arXiv preprint arXiv:2403. Stt.::2033.179555 2024.
Xingjian Shi, Zhourong Chen, Hao Wang, Yan Yeung, Wai-kin Wong, and Wang-chun Woo. Convolutional Istm network: machine learning approach for precipitation nowcasting. In Pro- ceedings of the 28th International Conference on Neural Information Processing Systems Vol- ume 1, pp. 802-810, 2015.
Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang- chun WOO. Deep learning for precipitation nowcasting: A benchmark and new model. In Advances in Neural Information Processing Systems, volume 30, 2017.
Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the- fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoin videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20675-20685, 2024.
Jaaxian Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.1660 2023.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision ECCV 2020, pp. 402-419, 2020.
Mark Veillette, Siddharth Samsi, and Chris Mattioli. Sevir: A storm event imagery dataset for deep learning applications radar and satellite meteorology. In Advances in Neural Information 2020.
Processing Systems, volume 33, pp.
Yunbo Wang, Mingsheng Long, Jianmin Wang. Zhifeng Gao, and Philip S Yu. Predmn: Recur- rent neural networks for predictive learning using spatiotemporal Istms. In Advances in Neural Information Processing Systems, volume 30. 2017.
Yunbo Wang. Lu Jiang, Ming Kingn Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d LSTM: model for video prediction and beyond. International Conference on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2025
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S. Yu, and Mingsheng Long. Predmn: A recurrent neural network for spatiotemporal predictive learning. IEEE Trans- (1D::2208-22223 2023.
actions Pattern Analysis and Machine Intelligence,
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600~ 612. 2004.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 20310- 20320, 2024.
Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene repre- sentation and rendering with 4d gaussian splatting, 2024a.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20331-200 20331-20341, 2024b.
Xuanyu Yi, Zike Wu, Qiuhong Shen, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, Shuicheng Yan, Xinchao Wang, and Hanwang Zhang. Mvgamba: Unify 3d content generation state space sequence modeling. arXiv preprint arXiv:2406.06330 2024.
Demin Yu, Xutao Li, Yunming Ye, Baoquan Zhang, Chuyao Luo, Kuai Dai, Rui Wang, and Xunlai Chen. Diffcast: A unified framework via residual diffusion for precipitation nowcasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 27758 27767, 2024.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 11121-11128, 2023.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference on Computer Vision, pp. 1-19. Springer, 2025.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 586-595, 2018
Yuchen Zhang, Mingsheng Long, Kaiyuan Chen, Lanxiang Xing, Ronghua Jin, Michael I. Jordan, and Jianmin Wang. Skilful nowcasting of extreme precipitation with nowcastnet. Nature, 619 (7970):526-532. 2023.
Shijie Zhou, Haoran Chang. Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 21676-21685, 2024.
Chen Ziwen, Hao Tan, Kai Zhang, Sai Bi, Fujun Luan, Yicong Hong, Li Fuxin, and Zexiang Xu. Long-Imm: Long-sequence large reconstruction model for wide-coverage gaussian splats. arXiv preprint arXiv:2410. 12781, 2024.
13

---

Published as a conference paper at ICLR 2025
A 3D RADAR
RENDERING
3D Radar Cross-Section Rendering aims to project 3D Gaussians from the world coordinate system onto the virtual camera imaging plane render specific cross-section results. First, apply trans- formation matrix to convert the 3D Gaussians from the world coordinates to the camera coordinates. Then, we discard any Gaussians that exceed a predefined threshold distance from the virtual imaging plane. Finally, the remaining Gaussians are rendered in order of their distances to the imaging plane, from nearest farthest. The 3D Gaussian's 's contribution to the pixel can be calculated by:
F= Σ fm may " (x))
IEN,
where Np is the number of Gaussians overlapping with the given pixel, is the radar features of i-th Gaussian, By and Fg; are the centered positions of the pixel and i-th gaussian respectively, Σ, is the 3D covariance matrix of i-th Gaussian. The rasterization strategy, similar to the origin 3DGS, fully differentiable, enabling to take full use of the GPU acceleration framework optimize parameters of Gaussians using stochastic gradient descent.
B MEMORY MECHANISM.
The core of the Gated Recurrent Unit (GRU) is based on two primary gates: the reset gate and the update gate, which control the flow of information. Unlike LSTM, GRU combines the forget and input gates into a single update gate, simplifying the architecture. When new input C. is received time step E, the reset gate r, controls how much of the previous hidden state H4-1 should be ignored when computing the candidate hidden state H4. A smaller reset gate value causes the model to "forget" parts of the previous hidden state, effectively resetting the memory. The update gate 2n determines how much of the previous hidden state H,-1 should be retained, controlling the balance between preserving past information and incorporating final hidden state H4, which is a combination of the previous hidden state H4-1 and the candidate hidden state H4. The ad advantages of GRU's gating mechanism are that it allows efficient control over information flow and helps mitigate the vanishing gradient problem by maintaining long-term dependencies more effectively.
data. the update gate 2 controls the
As illustrated Fig. 3] incorporating Mamba and GRU preserves the linear-time complexity of Mamba and introduces a long-range memory mechanism that allows mamba- based models to predict the next Gaussians based on the previous Gaussians. The key equations are shown below:
Tt a C + W Wr b,)
2: = o.W.CC + Whz H,,HH11-0 1 ba)
H4 = WanC. + Wnn H--1) + bn.
é, =BiMamba
H4 = 2. He-1 + (1 -) C.
C EXPERIMENT SETTING
C.I DATASET
NEXRAD: The 3D gridded radar reflectivity data used in this study were collected by the U.S. NEXRAD WSR-88D radar network. These data are soureed from the ds841.6 dataset product avail- able through the National Center for Atmospheric Research (NCAR) Research Data Archive De- partment of Atmospheric Sciences, Texas A&M University & School of Metcorology, University of Oklahoma (2021). Due constraints on storage and GPU resources, radar observations of severe storm events in 2022 with a longitude and latitude grid size ranging from 512 to 1024 are selected from different geographical coverage.
The selected storm events are observed at 5-minute intervals with a horizontal resolution of ap- proximately 0.021 degrees. A total of 6255 3D radar observations considered in this study. To
14

---

Published as a conference paper at ICLR 2025
preprocess the raw radar data, a quality control procedure is applied, following the methodology outlined in Homeyer & Bowman (222). Subsequently, a central 512 x 512 longitude/latiude grid is cropped from each event observation. In the vertical dimension, there are 28 levels, spanning from 0.5 km to 7 km with 0.5 km intervals, and from 7 km to 22 km with km intervals. To ensure consistent vertical spacing of features,
interpolate the original data obtain 44 vertical layers with 0.5 km intervals spanning from 0.5 km to 22 km. The 3D radar data from these storm events include seven channels: (1) ZH. the horizontal reflectivity factor, which indicates the intensity of radar returns from precipitation; (2) SW, the spectrum width, representing the variability of Doppler velocities within the radar pulse volume; (3) AzShr, the azimuthal shear, measure of wind shear in the horizontal plane often used to detect rotation within storm systems; (4) Div, the divergence, which reflects the horizontal divergence or convergence of wind fields; (5) Zpn, the differential reflectivity, used to differentiate precipitation types by comparing horizontal and vertical polariza- tions: (6) KDp, the specific differential phase, providing information on phase shifts of the radar signal, useful for estimating rainfall rates; and (7) TиV. the correlation coefficient between horizon- tal and vertical polarizations, which assesses the uniformity of precipitation particles and identifies non-meteorological targets.
Given the objective of predicting up to 20 future frames (100 minutes) based on observed frames (25 minutes), we sample 25 continuous frames with a stride of 10 from each event. These sequences are then divided into training, validation, and test sets in a 9:0.5:0.5 ratio.
MOSAIC The MOSAIC dataset records radar echoes over several years within Guangdong, China, with a time interval of 6 minutes between consecutive frames. To ensure efficient training while considering GPU and storage limitations, we excluded radar data that did not capture meteorological events, focusing on severe storm events in 2022. This resulted in a total of 24,542 radar observations. The raw data have a horizontal resolution of 880 X 1050 pixels, with 21 vertical layers ranging from 0.5 km to 6 km with 0.5 km intervals, from 6 km to 10 km with I km intervals, and additional layers at 12 km, 14 km, 15.5 km, km, and 19 km. To achieve consistent vertical spacing of features, we interpolate the original data to obtain 38 vertical layers with 0.5 km intervals spanning from 0.5 km to 19 km. The central 768 X 1024 region is cropped and downsampled to 384 x 512, resulting in radar observation data of dimensions 38 x 384 x 512. Unlike the NEXRAD dataset, MOSAIC includes only the intensity data of radar echoes. For the task of predicting up to 20 future frames (120 minutes) based on 5 observed frames (30 minutes), we sample 25 continuous frames with stride of 20 from each event. These sequences are then divided into training, validation, and test sets in 9:0.5:0.5 ratio.
C.2 IMPLEMENTATION DETAILS
We define the prediction task as predicting 20 frames future frames based on 5 initial frames, fol- lowing Yu et al. (2024). For reconstruction, we random sample 10% points from non-null regions to initialize the Gaussians. To maintain a constant total number of Gaussians, additional points are randomly sampled from the null regions, bringing the final count to 49, 19,152 Gaussians. During the reverse pre- rre-reconttructoon stage, we perform 5,000 iterations to adjust the positions of the Gaussians, applying 3D flow and energy constraints. In the forward reconstruction stage, we first optimize the positions of the Gaussians in the same manner for the first 5,000 iterations. Afterward, all Gaussian parameters optimized, incorporating the 3D flow, energy constraints, and reconstruction loss. The initial learning rate is set to 0.002, which is gradually reduced to 0.0002 by the end of train- ing. For the prediction task, we train our framework for 50 epochs using the Adam optimizer with a learning rate of 0.0005. For baseline methods, their 2D operators are extended to 3D to handle 3D radar prediction tasks, with their configurations adjusted accordingly for different datasets. All experiments are conducted under the same settings using 4 A100 GPUs.
C.3 METRICS
To evaluate the accuracy of predictions, we calculate the Mean Error (ME) and Mean Absolute Er- ror (MAE) to assess the overall numerical discrepancy between the predicted results and the ground truth. An ME value closer to zero indicates better accuracy; an ME greater than zero signifies that the predictions generally exceed the ground truth, while an ME less than zero indicates the opposite. To evaluate the visual quality of the predictions, we employ the Structural Similarity In-
15

---

Published as a conference paper at ICLR 2025
Table 6: Extended Experiment results in NEXRAD
Model
MAE+
SSIM
LPIPS
LPIPS Radar
CSI-22 Pool4
CSI-30 Pools
CSI-40 Pool4
ConvGRU
0.006
0.836
0.194
1.632
0.326
PhyDNet
0.017
0.366
0.323
2.114
0.348
0.097
0.002
SimVP
0.008
0.817
0.176
1,483
0.227
0.002
0.000
DiffCast
0.152
0.005
0.925
4.005
0.051
0.023
0.044
Mamba
0.004
0.902
0.125
0.625
0.304
0.158
0.075
GauMamba
0.003
0.907
0.122
0.600
0.361
0.205
0.089
dex (SSIM) Wang et al. (2004) Peak Signal-to-Noise Ratio (PSNR), and Learned Perceptual Image Patch Similarity (LPIPS) Zhang et al. (2018). Here, we use the pretrained AlexNet the evaluator of LPIPS. Besides, we pretrained BiGAN model on radar data in self-supervised manner. The LPIPS score calculated with the encoder of BiGAN is marked as LPIPS The results in Table 2 and 3 demonstrate that LPIPSRadur is well-aligned with the original LPIPS results. More impor- tantly, it highlights perceptual differences that were undetected by the original LPIPS and robust to noise. In Table the scores for ConvGRU and DiffCast show discrepancies between LPIPS and LPIPS Referring to Figure 6 left, it can be clearly observed that ConvGRU fails to predict the next few frames accurately, instead providing smoothed average result. In contrast, DiffCast produces results that are closer to the ground truth but with some noise. The higher LPIPS score for DiffCast indicates that LPIPS lacks robustness to noise in radar data, while LPIPS more accurately reflects the perceptual differences between the two methods.
In addition, the Critical Success Index (CSI) used quantify the degree of pixel-wise agreement Hits
between the prediction and the ground truth. CSI is defined as Hits Misses False Alarms? where Hits (truth=1, pred=1), Misses (truth=1, pred=0), and False Alarms (truth=0, pred=l) are counted after binarizing the continuous values of predictions and ground truth into 0/1 values at thresholds [20, 30, 40] dBz, Following Ravuri et al. (202T); ): Gao et al. (2023); Yu al. al. (2024), we report the CSI at pooling scale 4 x 4, which relaxes the pixel-wise matching to evaluate the accuracy on neighborhood aggregations.
C.4 PREDICTION
The existing prediction models used for comparison are originally designed for 2D sequence pre- diction. Therefore, adjustments must be made adapt them for 3D radar sequence prediction. Two feasible modifications can be considered. The first involves treating the newly introduced third di- mension, which represents the vertical axis, as channel dimension. However, this approach would limit the model's ability to predict changes along the vertical axis as effectively as it does in the hor- izontal plane. Instead, we opted extend these models from 2D to 3D, enabling them to natively handle 3D prediction. Specifically, we expanded 2D CNNs to 3D CNNs and similarly extended modules such as ConvGRU and ConvLSTM included in PhyDNet and DiffCast from 2D to 3D. Additionally, the 2D PDE constraint in PhyDNet is also adapted for 3D prediction.
D EXTENDED EXPERIMENTS
We conduct extended experiments using the NEXRAD dataset spanning three years (2020-2022) under the same training epochs, aiming to investigate the impact increased dataset diversity and size model performance. As shown in Table 6 most metrics of these models exhibit slight improvements when trained on the extended dataset. This performance gain can be attributed to the enhanced data diversity and the increased number of iteration steps enabled by the larger dataset. Notably, our model continues to outperform others, demonstrating its robustness and effectiveness under extended experimental conditions.
16

---

Published as a conference paper at ICLR 2025
E
QUALITATIVE RESULTS
Reconstruction. The full reconstructed results are shown in Fig.
g and 9
10
4DGS
Defomm33DGS
Ours
GT
Figure 7: Qualitative results of 3D radar reconstruction from frame frame 10.
Frame II
Frame 12
Frame 13
Frame 14
Frame 15
Frame 16
Frame 17
Frame 18
Frame 19
Frame 20
3DGStream
4DGS
Deform-3IXS
GT
Figure 8: Qualitative results of 3D radar reconstruction from frame 11 to frame 20.
Prediction. The full predicted results are shown in Fig.
T0
Fig.
Fig.
and Fig. 13
F DISCUSSION
Training objective of GauMamba. It is possible to directly predict G, based on G.-1- However, we found that this approach often leads to poor convergence. The difficulty arises because directly predicting G, requires the model to estimate both the overall trend of G and its exact position. In contrast, predicting AG, simplifics the problem by focusing only on the changes from the initial timestamp to timestamp thus reducing the solution space and making it more compact. Another viable alternative is to predict the difference between consecutive timestamps, i.c.. A&, G G1-1- However, this method suffers from the drawback error accumulation during inference, where the iterative predictions lead to increasingly compounded errors time.
17

---

Published as a conference paper at ICLR 2025
Frame 21
Frame 22
Frame 23
Frame 24
Frame 25
Detail
3DGStream
4DGS
Deform-DDGS
Figure 9:
Qualitative results of 3D radar reconstruction from frame 21 to frame 25 and details of
frame 25.
Frame 6
Frame
Frame 8
Frame
Frame 10
Frme II
Frame
Frame 13
Frame
Frame 15
PlyDNet
SinVP
GnoMambn
Figure 10: Qualitative results of 3D radar prediction from frame 6 to frame 15 in MOSAIC dataset.
Frame
Frame
Trame 18
Frume
Frame 20
Frame 21
Frimi
Frame 23
Frame
Frame 25
CanvERU
SimVP
DefCCast
GunMambo
GT
Figure 11: Qualitative results of 3D radar prediction from frame 16 frame 25 MOSAIC dataset.
18

---

Published as a conference paper at ICLR 2025
PbyDNet
SimVP
DifiCast
GauMamba
GT
Figure 12: Qualitative results of 3D radar prediction from frame 6 to frame 15 in NEXRAD dataset.
Frame 17
Frame
Frame
Frame 20
Frame 21
Frame
Frame 23
Fiame
ConyGRU
PhytNNet
SimVP
DiffCast
GauMamba
Figure 13: dataset.
Qualitative results of 3D radar prediction from frame 16 frame 25 in NEXRAD
Discussion of convergence issue in reconstruction. To accurately reconstruct highly dynamic radar sequences, all parameters of the 3D Gaussians need to be optimized. Following this setting the exist- ing dynamic Gaussian reconstruction methods face significant challenges convergence. The fixed parameters serve efficient anchors, allowing each Gaussian to lock onto a corresponding region of the object being reconstructed, thereby consistently tracking changes in the object. When all pa- rameters become adjustable, this crucial prior constraint is lost, leading to incoherent movement of the Gaussians with the reconstructed parts, and resulting in a vague and suboptimal optimization ob- jective. In extreme case, 3D Gaussians exhibiting random Brownian motion still reconstruct dynamic scenes and achieve a suboptimal outcome, as long as the deformation function or neural network used for reconstruction has sufficient capacity and enough iterations. Furthermore, each
19

---

Published as a conference paper at ICLR 2025
Gaussian influences only very small region of the space, which theoretically limits the effective area for gradient descent. small perturbation introduced by randomly initialized neural networks polynomial functions can cause the Gaussian to jump out of its original reconstruction region, hindering the gradient flow from the corresponding ground truth area from properly affecting the disturbed Gaussian and trapping it in unrelated region, leading significant convergence issues.
20


Reviews:
[{"review_id": "nkwKKUtSCT", "reviewer": "Reviewer_pwg9", "summary": "This paper presents a novel 3D weather nowcasting approach using high-dynamic radar sequences. The method introduces a SpatioTemporal Coherent Gaussian Splatting technique to efficiently represent the dynamic radar data, which is then processed by the GauMamba network to generate weather forecasts. Additionally, the paper proposes MOSAIC, a new high-resolution 3D radar sequence dataset, containing more than 24K radar observations. Experimental results on both NEXRAD and MOSAIC show that the proposed approach outperforms baseline methods with significant margins.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed representation and processing pipeline are well-motivated and supported by the experiments, resulting in performance improvements over baseline methods.\n- The memory usage of the method remains constant w.r.t. horizontal resolution, in contrast to other baselines with linear memory growth.\n- The MOSAIC dataset offers a large dataset of radar echoes that capture meteorological events across multiple years.", "weaknesses": "- A short discussion of the proposed dataset, MOSAIC, in the main paper could be beneficial to the readers.\n- Could the authors clarify the plan for the dataset? Will it be made publicly available, and if so, how to ensure that the dataset can be accessed by the public continuously?\n- For the LPIPS evaluation, how is the evaluator model trained? If the model is pretrained with a general-purpose dataset, would it work well with radar data?\n- Minor visual artifacts can be seen in Figure 2 (center bottom).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwWE1o9Drs", "reviewer": "Reviewer_yyVm", "summary": "The paper proposed a framework utilizing SpatioTemporal Coherent Gaussian Splatting (STC-GS) for dynamic radar representation and GauMamba for forecasting dynamic meteorological radar signals. The STC-GS established 4D dynamic Gaussians, and the GauMamba utilizes the Mamba framework to predict 3D radar sequences. The experiments on NEXRAD and MOSAIC present superior performance against previous 4D reconstruction methods.", "questions": "- The paper mentions employing a group of Gaussian primitives for radar sequence prediction. Is there a fixed number of Gaussian primitives used across all frames, and if so, how is this number determined and maintained throughout the model's operations?\n\n- It is claimed that the GauMamba model is efficient; however, from the comparisons in Figure 4, it appears that at resolutions lower than 256, baseline methods are more memory efficient. Please discuss why there is this discrepancy in memory efficiency at different resolutions and how the baselines' memory scales with increasing resolution.\n\n- Please also discuss the limitations of the proposed method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing is comprehensive and easy to follow\n\n- The ablation study is thorough and provides insights into the model design and the selection of hyperparameters.\n\n- By combining Gaussian and Mamba methodologies, the GauMamba model is likely designed to enhance forecasting accuracy, especially in scenarios involving temporal and spatial data complexities which are typical in meteorological datasets.", "weaknesses": "- Previous research has introduced combinations such as Gamba that combine Gaussian and Mamba model. However, this paper does not sufficiently review these previous works. It would be beneficial to highlight how the proposed GauMamba method differs from earlier studies. Additionally, including comparisons with Mambda + Gaussian baseline methods in the experimental section is crucial for a sufficient comparisons.\n\n- The paper describes experiments conducted on two weather forecasting datasets. These datasets appear to be quite limited, containing only two scenes—if this is incorrect, please advise. Such a small dataset may not adequately demonstrate how the method compares with established baselines. Conducting experiments on larger, more diverse datasets could provide a more thorough evaluation.\n\nReferences:\n- Shen, Qiuhong, et al. \"Gamba: Marry gaussian splatting with mamba for single view 3d reconstruction.\" arXiv preprint arXiv:2403.18795 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QT9AVnHZwI", "reviewer": "Reviewer_U22W", "summary": "This paper presents a comprehensive framework for 3D radar sequence prediction in weather nowcasting that combines a novel SpatioTemporal Coherent Gaussian Splatting (STC-GS) representation with a memory-augmented predictive network (GauMamba). The approach addresses the key limitations of current 2D prediction methods by enabling efficient and accurate 3D sequence prediction while maintaining computational efficiency.\n\nThe main contributions include:\n\n1. A novel 3D radar representation method (STC-GS) that efficiently captures radar data dynamics through bidirectional reconstruction with dual-scale constraints, achieving 16× higher spatial resolution compared to existing 3D representation methods\n\n2. A memory-augmented predictive model (GauMamba) that effectively learns temporal evolution patterns from the STC-GS representations to forecast radar changes, outperforming state-of-the-art methods in prediction accuracy\n\n3. Two new high-dynamic 3D radar sequence datasets:\n- MOSAIC: 24,542 single-channel high-resolution radar observations\n- NEXRAD: 6,255 six-channel radar observations", "questions": "Please see Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Good Originality:\n- The paper presents a novel solution for 3D radar sequence prediction by combining Gaussian Splatting with a memory-augmented network\n- The adaptation of 3D Gaussian Splatting to dynamic radar data representation represents meaningful innovation in both representation and prediction aspects\n- The bidirectional reconstruction pipeline with dual-scale constraints is a creative approach to handle the unique challenges in radar sequence prediction\n\nGood Quality:\n- The technical development is thorough with comprehensive theoretical foundations and implementation details\n- The experimental evaluation is extensive, covering multiple datasets and comparing with various baselines\n- The ablation studies effectively validate the contribution of each component\n\nGood Clarity:\n- The paper is well-structured with clear motivation and problem formulation\n- The methodology is presented in a logical flow with detailed explanations\n- The figures are well-designed and effectively illustrate the key concepts\n- The writing is generally clear and easy to follow\n\nGood Significance:\n- The work addresses an important practical problem in weather nowcasting\n- The proposed framework achieves significant improvements over existing methods:\n  * 16× higher spatial resolution in representation\n  * 19.7% and 50% reduction in MAE on two datasets\n- The introduction of two new high-dynamic 3D radar sequence datasets contributes valuable resources to the research community\n\n> While I support accepting this paper based on its technical merits and clear presentation, I am not very familiar in radar sequence prediction specifically. Therefore, I remain open to adjusting my assessment during the discussion phase based on comments from other reviewers more specialized in this domain.", "weaknesses": "Major Weaknesses: about **Reconstruction Results**\n\n1. Unusual Performance Gap in Reconstruction:\n- The results in Table 1 show dramatically better performance compared to baselines\n- While authors explain this is due to convergence issues in existing methods, this raises concerns:\n  * The performance gap (10× improvement in MAE) seems unusually large\n  * Need to consider whether alternative baselines outside 3DGS family might be more appropriate\n  * Traditional scene reconstruction methods might provide more reasonable comparisons\n \n> Therefore, I would like to suggest the authors to include non-3DGS based methods that might achieve better convergence\n\n2. Missing Qualitative Results:\n- The paper lacks visualization results for the reconstruction stage\n- This omission is particularly concerning given the significant quantitative improvements claimed\n- Visual results would help validate the dramatic performance improvements\n\n> Therefore, I would like to suggest the authors to either:\n>   * Add qualitative reconstruction visualizations\n>   * OR Provide clear justification for why such visualizations are not included\n\n\nMinor Weaknesses: Grammar and Writing Issues\n\n- Several grammatical errors throughout the paper\n    - For example, on Page 7, line 371\n\n> These should be carefully corrected in the final version", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)