Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution
Authors: Jacob Lin
Edward Gryspeerdt
Ronald Clark
Venue: NeurIPS 2025 spotlight ()
Keywords: AI for Science
Computer Vision
Climate Change
3D Reconstruction
Primary area: machine_learning_for_sciences
TL;DR: Use ground-based cameras to retrieve physical properties of clouds at a high spatial and temporal resolution.

Abstract: There has been great progress in improving numerical weather prediction and climate models using machine learning. However, most global models act at a kilometer-scale, making it challenging to model individual clouds and factors such as extreme precipitation, wind gusts, turbulence, and surface irradiance. Therefore, there is a need to move towards higher-resolution models, which in turn require high-resolution real-world observations that current instruments struggle to obtain. We present Cloud4D, the first learning-based framework that reconstructs a physically consistent, four–dimensional cloud state using only synchronized ground‐based cameras. Leveraging a homography-guided 2D‐to‐3D transformer, Cloud4D infers the full 3D distribution of liquid water content at 25 m spatial and 5 s temporal resolution. By tracking the 3D liquid water content retrievals over time, Cloud4D additionally estimates horizontal wind vectors. Across a two-month deployment comprising six skyward cameras, our system delivers an order-of-magnitude improvement in space-time resolution relative to state-of-the-art satellite measurements, while retaining single-digit relative error ($<10\\%$) against collocated radar measurements.

Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution
Jacob Lin Department of Computer Science University of Oxford jacob. lin@cs. ac uk
Edward Gryspeerdt Department of Physics
Imperial College London
grysseerrchhimpriilll ac. uk
Ronald Clark
Department of Computer Science
University of Oxford
ronald. clarkêcs ox ac. uk
Abstract
There has been great progress in improving numerical weather prediction and climate models using machine learning. However, most global models act at a kilometer-scale, making it challenging to model individual clouds and factors such as extreme precipitation, wind gusts, turbulence, and surface irradiance. Therefore, there is a need to move towards higher-resolution models, which turn require high-resolution real-world observations that current instruments struggle to obtain. We present Cloud4D, the first learning- framework that reconstructs physically consistent, four- dour-dimensional cloud state using only synchronized ground-based cameras. Leveraging a homography- Cloud4D infers the full 3D distribution of liquid water content at 25 m spatial and s temporal resolution. By tracking the 3D liquid water content retrievals over time, Cloud4D additionally estimates horizontal wind vectors. Across two-month deployment comprising six skyward cameras, our system delivers an order-of -orgnntt- improvement in space-time resolution relative state-of-the- art satellite measurements, while retaining single-digit relative error (< 10%) against collocated radar measurements. Code and data available our project page htttp:://///tt //coudd4/ facpodddjjcobbooo.o pacobblin.co
2D-to-3D transformer,
Introduction
Accurately estimating state of Earth's atmosphere is essential to all aspects society, from protecting food supplies to routing flights and optimizing renewable power generation. Therefore, there has been much interest in creating machine learning weather and climate prediction systems able to accurately model the evolution of the atmosphere. However, many processes in the atmosphere are too detailed for current models to simulate directly. For example, shallow cumulus clouds generally span less than kilometer and cover much 40% of the Earth's surface (Tselioudis) et al. (202)) and they play a central role in controlling the Earth's 's temperature. As cumulus clouds are small and short-lived, current weather and climate simulators can not resolve them directly. Instead, hand-crafted "parameterizations" are used to approximate the aggregate effect of cloud properties on variables such as temperature, moisture, and winds much coarser grid. These approximations are a major source of error in both day-to-day forecasts and long-range climate projections. Modern machine-learning systems such as GraphCast (Lam et 2023 Pangu- Weather (Bi al., 2023).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and Aardvark (Allen et al. 2025) inherit this problem: they train on "reanalysis" data that comes from the same physics-based models and therefore bake the same biases.
Improved models are needed, but this is hampered by the difficulty of observing detailed atmospheric phenomena such shallow cumulus clouds. To develop better models and assess their impact on aggregate environmental variables, we need detailed observations of cloud properties such as the size, spacing, and water content of individual clouds over their whole lifetime (Geerts al. 2018). However, detailed cloud properties are not easily measured by current observing systems. High-resolution satellite imagery has revisit time of several days, and while scanning radar and in situ aircraft measurements provide data of high detail, they are typically of small part of the cloud, lacking the larger context.
In this paper, we introduce Cloud4D, the first learning-based system that reconstructs a physically consistent, four-dimensional cloud state solely from synchronized ground-based cameras. Our method is based on a homography-guine -uided 2DD-t--3D transformer architecture that estimates the full spatial distribution of liquid water content and horizontal wind vectors 25 m resolution and s rate. A two-month real-world deployment with six cameras demonstrates that Cloud4D achieves an mnder-of-maanituuee improvement in space-time resolution over state-of-the-art satellite products while maintaining < 10% relative error against collocated radar retrievals.
Our contributions are threefold:
1. We propose the first method jointly estimate cloud physical properties (liquid water content, height, thickness) at a high spatial and temporal resolution from multí-view images.
2. To accomplish this, we propose a homography- guided 2D-3D transformer model that ingests images and accurately predicts 3D cloud properties on high-resolution grid.
3. We demonstrate, on a two-mme two-month deployment with six cameras, that our method delivers an order-of- magnitude improvement in temporal resolution over space-borne products while achieving < 10% relative error against collocated radar retrievals.
Problem Background
State-of-the-ar of-tt-- weather prediction systems, ranging from traditional dynamical models recent neural-network ap- proaches such as GraphCast (Lam et al., 2023), evolve core set of variables on a dimensionll grid. Typical variables include air temperature, pressure, the horizontal wind components, humidity, and average droplet amounts such cloud water, rain, and snow. However, the models resolve clouds only implicitly and at much coarser reso- lutions than the scales which individual cloud elements form and evolve. Our work focuses on estimating these quantities at much higher resolution using ground-based cameras that can be used as a substitute for radar to help validate and improve weather models.
Height (km
Layer Base Height
Dewpoint
Temperature
Temperature (*C)
Figure 1: As result of the cloud forma- tion process, clouds are spatially struc- tured into layers.
The main physical cloud property that we are interested in is the amount of water they contain, it is key driver affecting weather and climate. For cumulus clouds, this is characterized by the liquid water content (LWC) (kgm-3) distribution, and is common variable in weather and climate models. As the vertical structure of water in the atmosphere especially important, the summation of the LWC along height column is often explicitly predicted by weather and climate models. This is referred to the liquid water path (LWP) (kgmm - -). Other important properties are the cloud base height (CBH) and the cloud top height (CTH), as they impact weather and climate while also capturing the small-scale physics of a cloud.
Our model takes advantage of the fact that cloud fields are spatially structured into cloud layers at specific altitudes (see Figure D). These cloud layers exhibit different properties and are categorized into various cloud types. In this work, we focus estimating the physical properties of shallow cumulus clouds, which generally form altitudes below 2000 m and are the first cloud layer from the ground. These clouds are very difficult to measure using radar and satellite images because their

---

images Layer 3.1)
DINOV2 LoRUp
20 CNN
Cloud Layer Homography
Heights
b) 3D Refinement ($ 3.2)
c) Inference: Wind Retrieval 3.3)
Eeit
sparsity
Sparse Transformer
2.5D
DINOv2 Grld
Predicted Cloud LWC p
Prysics-
Time-Series
Figure 2: Model overview. Our model estimates the liquid water content of clouds using sparse set of ground-basee cameras. a) Cloud Layer Model: We leverage an inductive bias on the spatial structure of clouds by defining homography that maps images to cloud layers. The cloud homography is used to predict key 2.5D cloud properties, giving an initial estimate of the 3D cloud layer. b) 3D Refinement: A sparse transformer then refines the initial 3D field and estimates the final cloud liquid water content. e) Inference: Wind Retrieval: By tracking the motion of cloud reconstructions over time, our method retrieves height and time-varying horizontal wind profile.
life cycle is short, and each individual cloud is small. Section 3.1 we show how we can exploit the layer structure of clouds through a homography-guuie reconstruction model.
3 Method
In this section, we present our approach to estimating a high-resolutie 3D grid of cloud properties from ground-based images. We formulate the problem as follows, given N ground-base
{4) together with their corresponding camera poses (P. [R. t. J i= and intrinsics {K L.V. h=1 the goal is to produce an estimate of the cloud liquid water content at each grid cell X E R.N. XN,
Figure 2 shows overview of our method, Cloud4D. We first leverage the spatial structure of clouds and use homography to map images to cloud layers (Section 3.1). Taking advantage of the vertically thin nature of cloud layers, Cloud4D uses the homography and formulates the cloud layer estimation as an easier 2D-to-2D task. A sparse 3D transformer then refines the predictions using a full learned 3D prior attending over the whole volume (Section 3.2). Lastly, at inference time, estimate the liquid water content at multiple time steps and exiract horizontal wind profiles by tracking our predictions over time (Section 3.3
3.1 Cloud Layer Model
To estimate the properties of a cloud layer, we first consider homography between an image 7 with homogeneous coordinates (u,, 0s 1)T and a plane height h with world coordinates (I., U, h) as follows:
T y
t,n
RR
K
U;
(1)
where n [0, 0, T and = -h define the plane n X + d= 0. To search for cloud layer at an arbitrary height, we consider a feature volume with multiple such planes H different heights / This is similar to previous approaches that build cost volumes for multi-view stereo Yao al. 2018 Gu et al. (020) with the key difference being that our homography is explicitly spatially aligned with cloud layers.
{hr) H

---

In practice, we leverage the robustness and 3D capabilities (El Banani et al. 2024) of vision foundation
models by first processing the images {1,}== using a DINOv2 (Oquab al. 2023) backbone producing features J For additional image-level detail, we use LoftUp (Huang et al. 2025) to upsample the DINOv2 features back to the original image resolution and then use a learned projection downsample the feature dimension d channels.
The DINOv2 features from each camera are lifted to world space using the cloud homography as defined in I resulting in feature volume V E R x Nr N₂, where the features have been averaged across the N views. Following previous work (Peebles and Xie 2022 Perez et al. (2018). use adaptive layer normalization (adaLN) condition each of the H total feature layers on the height from which it was sampled.
Due to the vertically thin nature of cloud layers relative to their horizontal length, the main variation in cloud structure will be the horizontal plane. Motivated by this, we formulate the cloud layer estimation 2D-to-2D task, where the 3D cloud layer can be represented using 2.5D cloud properties. Specifically, the feature volume is first flattened into a 2D feature with Hd channels, such that it can be processed by 2D CNN, Ω. The 2D CNN predicts key 2D cloud properties, namely the liquid water path LWP, the cloud base heights CBH, and the cloud geometrical thicknesses Ah.
3.2 3D Refinement
In order obtain full 3D grid estimate of cloud properties, lift 2.5D maps into 3D give an initial 3D estimate of the liquid water content at each point in space ø' E Nx xN,xN, To refine this coarse initial estimate, we then use sparse transformer V/ which estimates the LWC ý by learning better 3D distribution of the LWC.
Specifically, we first initialize the 3D LWC ø' from our 2.5D features as the following:
(w Ah(e))
2(2s, -CBI(*)) Ahte.)
if CBH(
< 282 < CBH(2 (2.2) Ah().))
0,
otherwise
where s, is the size of a voxel along the height dimension. The first term evenly distributes the LWP along the cloud height column, while the second term adds a linear increase towards cloud tops to better match the expected LWC distribution of a cloud (Brenguier et al. 2000).
We then discard empty voxels in initial LWC fL, extracting sparse structure of M voxels, where M & N₂ NN, as the cloud layer is vertically thin relative to the horizontal length. This allows us to process the initial estimate using sparse transformer, learning 3D cloud prior without the added computational complexity of attending over full voxel grid.
To add image-level features to the transformer processing, concatenate each sparse voxel of LWC with DINOv2 features {F that are backprojected and averaged across all cameras. As in Section 3.1 the DINOv2 features are spatially upsampled with LoftUp and then downsampled channel-win -wis using another learned projection to a dimensionality of dp. We additionally add sinusoidal positional encoding based on the coordinates of each sparse voxel.
We normalize the output of the sparse transformer along the height dimension using softmax and then scale each height column such that the original LWP (Section 3.1) is preserved. This formulation takes advantage the strong 2.5D features from the cloud layer model while learning a full 3D cloud prior.
3.3 Wind Retrieval
The large-scale motion of clouds is dominated by horizontal winds that primarily change as function of altitude. By tracking the movement of our reconstructed clouds, we retrieve the horizontal wind profile over time. This is similar to previous approaches for the estimation wind through cloud motion using satellites (Horváth and Davies 2001) with a key difference being that tracking is done on full 3D LWC fields rather than satellite lite imagery. This allows us to directly track the motion of clouds along their full vertical width, giving height-varying horizontal wind profile.
To track the motion of our reconstructed clouds, we visualize the LWC of the reconstructed clouds across different 2D height slices Psticed N and use an off-the- shelf learned point tracker,

---

specifically CoTracker3 (Karaev et al., 2024). This approach allows to use temporal cues to track the reconstructed clouds while also processing long sequences efficiently. By tracking how points on the height-sliced I LWC move horizontally over time, the rate of change in pixel space directly gives us horizontal wind retrievals. We provide additional implementation details in the supplemental material.
3.4 Implementation Details
Training We train our model in two separate stages. The cloud layer model is first trained to predict 2.5D cloud properties, which is then followed by training the sparse transformer for full 3D refinmentt For the 2.5D predictions, we optimize for the following objective:
L2D LIWP ACBICCBU + AanCas
where all losses are L1 losses applied to each predicted 2.5D cloud property. ACBI and Aaa are hyperparameters to scale the losses to similar ranges and are both set to 0.1.
In the second stage, we freeze the weights trained in the first stage, and train the sparse transformer with the following objective:
CAD = 1lp Pll1.
(4)
Training is performed for 60k steps the first stage and 30k steps in the second stage. Optimization is done using Adam (Kingma and Ba 2015), and takes three days with 4x H100 80GB GPUs.
Model Configuration Our model reconstructs volumes from heights of 4000 m, within a km x 5 km area. We choose a voxel size of 25 m, giving us dimensions of ý E 20200x2000100 For the cloud homography, we sample heights every 200 m starting from 400 and ending 3800 m. This results in H 18 heights. The DINOv2 features are downsampled to channel size of d dp 16.
Cloud Datasets
4.1 Synthetic Dataset
To train our model, we require ground- images that are paired with 3D grids representing the cloud liquid water content. There are no instruments that can give such data at the resolution and scale we are aiming for, and thus we rely on realistic large eddy simulations (LES) to generate necessary training data.
We use the accelleaee LES software, MicroHH (van Heerwaarden three scenarios of cumulus cloud days (ARM (Brown et al.) 22022). Cabauw (Tijhuis et al. 2023). and RICO (vanZanten et al., 201T)). By converting our liquid water content from the LES output to scattering coeffccients we render images with Monte Carlo path tracing in Blender's 's Cycles engine.
2017) and simulate
For additional data diversity, also render cloud volumes from Terragen, application aimed at creating photorealistic natural scenes. As these are not physically realistic and will not accurately represent real-world cloud liquid water content, we only use this data for pre-training.
For both MicroHH and Terragen volumes, we render images from six different views, with intrinsics and extrinsics matching real-world dataset (see Section 4.2). For each camera, render 1500 images from MicroHH volumes and 1000 images from Terragen volumes, totaling 15000 images.
Real-World Cloud Dataset
To evaluate our method, we collect real-world images from six cameras across two-month period. The six cameras are positioned in an inward-looking array, covering 5 km x km area where Cloud4D does cloud property estimation. Taking an image every five seconds, the ground-based cameras enable Cloud4D to operate a high temporal resolution. Comparatively, other instrument- based retrievals, such satellites, can take up to hours days. Ground-bar based cameras, therefore,
5

---

Ours (5 seconds)
ERAS hour)
Reference (30 seconds)
255
2.5-
Height
2a
10
10
1.-
11:00
11:30
12:00
12:00
11:00
11:30
12:00
12:30
11:00
11:30
12:00
12:30
Time (UTC)
Time (UTC)
Time (UTC)
mm
(a) ID liquid water content along a vertical line aligned with radar scan
Ours (5 seconds)
ERAS (1 hour)
Reference (30 seconds)
2.9
Mean value
Mean value
0.-
3.6
5.6
64
11:30
3.6
11:00
12:30
4.0
11:30
12:00
12:30
Time (UTC)
Time (UTC)
Time (UTC)
(h) Liquid water path
Figure 3: Comparison with radar retrievals. To compare with radar values, we visualize our 3D liquid water content along single ray over time. ERA5 captures coarse properties such as cloud heights and mean LWP. Comparatively, our method predicts high-resolution cloud properties that match up with radar retrievals.
provide the potential for a cheap and scalable means of obtaining high-resolution observations of clouds at state-of-the-art spatial and temporal resolution.
To evaluate the accuracy of our cloud property retrievals, the cameras collocated with instruments that can retrieve liquid water content and horizontal wind profiles. Specifically, a radar retrieves a ID vertical liquid water content profile above the radar with 30 height resolution every 30 seconds. Wind profiles are measured by a radar wind profiler that retrieves the horizontal wind profile every five minutes with height resolution of 70
Across the two-month deployment, we manually identify 12 days with shallow cumulus clouds, resulting in 17 hours of camera data. This forms the basis of the real-world data that we use for evaluation.
5 Experiments
We conduct experiments using real-world ground-based camera dataset, consisting of 17 hours of camera data capturing shallow cumulus clouds across 12 days. Given that there is no prior work which uses ground-based cameras for the estimation of physical cloud properties, instead rely comparisons to methods for satellite- based cloud retrievals (Ronen et al. 2022), radar, wind profiler, and satellite measurements and ERA5 (Hersbach al. 2020 data
Radar: The standard instrument for high-resolution measurements of clouds generally a cloud- profile radar. Therefore, we compare cloud property predictions, which span much larger area
Provided by the European Space Agency under Open Access compliant Creative Commons CC BY-SA 3.0 IGO Licence
Provided by the Copernicus Climate Change Service under the ECMWF Copernicus License.

---

Table 1: Quantitative comparison against existing methods. Cloud occupancy (Occ) is evaluated using an Fl score, where height column is considered occupied if there are non-zero densities in the column, Mean absolute error is shown for retrievals on liquid water content (LWC), liquid water path (LWP), cloud base height (CBH), and cloud top height (CTH).
Occ
LWC (gm
LWP (kgm
CBH (m)
CTH (m)
VIP-CT Ronen et al. (2022)
0.40
0.13
0.39
791.23
1021.49 49
Cloud4D (ours)
0.70
0.03
0.06
189.58
295.77
of 5 km x 5 km, with the single height-column retrievals of radar. To enable this comparison, evaluate our 3D liquid water content along a single height column at the location of the radar.
Satellite: In contrast to cloud-profilie radars, which scan in limited spatial area, satellites provide retrievals across large area but are instead limited temporal resolution. Our method, on the other hand, operates with significantly larger area compared to cloud- houu-profiling radars, but simultaneously preserves the high temporal resolution, estimating cloud properties every five seconds. The combina- tion of high temporal and spatial resolution is unique to ground- based cameras and is key to resolving the small-scale physics of a cumulus cloud across its life cycle.
Wind can be measured through different instruments, such as satellites, weather balloons, and radar wind profilers. Of these, wind profilers are typically the most accurate, and therefore, we evaluate our wind estimates by comparing them to wind profiler situated at the site where we collected the evaluation dataset.
Wind profiler:
5.1 Results
Figure 3 compares our cloud liquid water content and cloud liquid water path predictions with radar values. Our predictions have high spatial resolution (25 m) and temporal resolution (5 seconds) while also covering a significantly larger area than the radar. This enables the estimation of fine details across whole clouds and is key to closing the observational gap where current observations do not capture full microphysics across clouds. From Figure 3 we highlight in particular that our predictions of cloud properties, such cloud base heights, cloud top heights, and liquid water path, closely match the radar retrievals. Global models such as ERAS, instead, operate at kilometer and hourly scales, which do not observe individual clouds but instead capture coarse properties such as the average liquid water path and approximate heights of cloud layers.
Comparing against radar retrievals over a longer period of 12 cumulus days, T quantitative results with our method and also with VIP-CT (Ronen et al. 2022). VIP-CT is a satellite- based cloud retrieval method, which observe is unable to learn cloud property estimation from ground- based cameras. This is not unexpected, as VIP-CT is designed for satellite imagery and relies on unobstructed orthographic views. Additionally, VIP-CT implicitly learns multi-view geometry, which is significantly more difficult for ground- ground-based cameras where views vary more than for satellites. In comparison, Cloud4D retrieves cloud properties with less than 10% relative error I compared to radar values, while doing so in a significantly larger area of 5 km x km.
Figure 4 shows visualizations of our 3D liquid water content retrievals at times coinciding with satellite imagery at the location of the cameras. We highlight that our cloud envelopes qualitatively match up with satellite retrievals, while operating at a significantly higher temporal resolution of seconds instead of days. This enables cloud property estimation across the full life cycle of a cloud rather than single snapshot provided by satellites. We note that our liquid water content retrievals are visualized using Blender with accurate sun positions, enabling coarse qualitative evaluation of cloud heights through the shadow of the clouds.
'This is calculated using the mean LWC of clouds from the radar GT as following: MAE(LWC of clouds) CC of clouds from radar GT) 0.321 gmm-
0.029gm 8.9%
7

---

Ground-based Camera Inputs
Sentinel-2 (5 days)
Ours (5 seconds)
MODIS
(1 day)
(5 seconds)
Ours
Figure 4: Qualitative comparison against satellite imagery. We render our 3D liquid water content predictions from top-down orthographic view using Blender and compare against 2D image retrievals from Sentinel-2 and MODIS. Cloud4D estimates volumetric cloud properties every five seconds, while average, Sentinel-2 takes image every five days and MODIS once per day.
Ours minutes)
hour)
Reference minutes)
Height
1.0
1.
30
11.00
11:00
12te
12:32
11.00
11:0
12400
30:92
33.00
11:91
1250
12.50
Time (UTC)
Time (UTC)
Time (UTC)
Horizontal Wind Speed
Figure 5: Comparison of horizontal wind retrieval. Cloud4D is able estimate height-varying horizontal wind vectors from the motion of our cloud predictions. Our wind profiles are of similar magnitude and direction retrievals from wind profiler. The arrow direction denotes the horizontal wind direction following the convention of a compass bearing.
Figure compares horizontal wind profiles at different heights and times. We observe that our horizontal wind profiles have similar magnitudes and directions when compared the radar wind profiler retrieval.

---

Related Work
Cloud property estimation The retrieval of physical cloud properties is an important task for weather and climate applications. With the advent of deep learning, learned methods for the estimation of cloud properties have been explored using satellite images. Notably, 3DeepCT (Sde-Chen et al.
202T) and VIP-CT (Ronen al. 2022 have shown comparable retrieval results compared to traditional explicit physics-based methods (Levis et al. 2013 2017 2020). However, VIP-CT and 3DeepCT both implicitly learn camera geometries from small -mall-scale synthetic datasets. This is a much more difficult task for ground- basddd-aeddde cameras, which will greatly differ in viewing directions compared to the orthographic satellite views. Our method explicitly models the camera geometry by mapping images to cloud layers in world space using a homography, simplifying the cloud property estimation task. Previous works with ground baseed cameras have focused on estimating geometric cloud properties such as cloud base heights using stereo cameras (Romps and Öktem 2018 Öktem jet al., 2014). but do not recover physical quantities such as the liquid water content. Our method uses a learned approach to estimate both physical and geometrical properties of clouds, doing so at high spatial and temporal resolution.
Weather and climate models Recently, there has been a growing number of works on ML- based weather and climate models (ConvLSTM (SHI et al.] 2015), GraphCast (Lam al.) (2023). NeuralGCM (Kochkov et al., 2024), Pangu-Weather Weather (Bi et al. 2023)), and Aardvark (Allen et al.. 2025)). However, these global models, operating a kilometer scale, and therefore do not capture the physics of individual clouds. There a need for higher resolution models such that phenomena that occur at subgrid scale can be predicted. Our method fills observational gap, providing high-resolution data enabling the evaluation of fine-scale models and the training of data-driven surrogates.
7 Discussion and Limitations
We have trained our model mainly on data of cumulus clouds. We made this choice as cumulus clouds have a large effect on the atmosphere (Chen et al.] 2000). while also being difficult to measure with other instruments due to high requirements on temporal resolution and spatial coverage. However, it worth noting that other cloud types are also significant drivers of the atmosphere (Lee et al., 202T Chen et al. (2000), which makes the extension of Cloud4D to other cloud types an interesting avenue for future work.
Secondly, we have focused on retrieving cloud properties of a single layer. This is well motivated for ground-based cameras, as any higher layers will generally be occluded. Investigating the retrieval of cloud properties from higher cloud layers would increase the robustness and potential applications of our work and a promising direction.
Ground- based cameras provide a cheap and scalable means of obtaining cloud retrievals through Cloud4D. However, we note that they also include some limitations that do not apply to other instruments. In particular, they are susceptible to occlusions from environmental conditions such as rain, fog, and snow.
8 Conclusion
We introduced Cloud4D, the first learning-based framework that can provide cloud measurements at a high spatial and temporal resolution using ground-base cameras. Leveraging a homography-guide 2D-to-3D transformer and a sparse-voxel refinement stage, Cloud4D reconstructs liquid water content and height-resolved horizontal winds on a 25 m 25 m 25 m grid every s. During two-month deployment, the system achieved an order-of -fiio--t--miiuudeee finer temporal resolution than state-of-the-ee satellite estimates while maintaining < 10% relative error against collocated radar and wind profiler measurements. Because it relies only on low-cost, widely available cameras, Cloud4D offers path to scalable, high-frequency observations of clouds worldwide. These estimates can close long-standing observational gap that hampers both physics-based and neural weather and climate models. For future work, extending the framework to multilayer and optically thick cloud regimes, embedding dhiative-transfer constraints, and coupling the retrievals with differentiable simulators could further

---

increase physical fidelity. We plan to release our code, synthetic training data, and 17-hour real-world benchmark dataset.
Acknowledgements
This work was supported by the Natural Environment Research Council (grant nos nos. Nos.N//00183391 WOEST, NE/X012255/I), ARIA SEOP-SEEO-PPOE Next-CAM), an ESPRC scholarship (2922572), and the Royal Society (grant no. URF/RI/191602).
References
Anna Allen, Stratis Markou, Will Tebbutt, James Requeima, Wessel P. Bruinsma, Tom R. Audersson, Michael Herzog, Nicholas Lane, Matthew Chantry, Scott Hosking, and Richard E. Turner. End-to-end data data-driven weather prediction. Nature, 2025.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium-range global weather forecasting with 3d neural networks. Nat., 63366799 2023.
Louis Brenguier. Hanna Pawlowska, Lothar Schülllr, Rene Preusker, Jürgen Fischer, and Yves Fouquart. Radiative properties of boundary layer clouds: Droplet effective radius versus number concentration. Journal of the Amoosphrrcc Sciences, 57())880 821, 2000.
A. R. Brown, R. T. Cederwall, A. Chlond, P. G. Duynkerke, J.-C. Golaz, M. Khairoutdinov, D. C. Lewellen, A. P. Lock, M. K. Mac C.-H. Moeng, R. A. J. Neggers, A. P. Siebesma, and B. Stevens. Large-eddl simulation of the diurnal cycle of shallow cumulus convection over land. Quarterly Journal of the Royal Meteorological Society, 128()82) 1000--103,,2200 2002.
Ting Chen, William B. Rossow, and Yuanchong Zhang. Radiative effects of cloud-type variations. Journal of Climate, 13(1):264 - 286, 2000.
Mohamed EI Banani, Amit Raj. Kevis-Kokite Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D Awareness of Visual Foundation Models. In CVPR, 2024.
Bart Geerts, David J. Raymond. Vanda Grubišić, Christopher A. Davis, Mary C. Barth, Andrew Detwiler, Petra M. Klein, Wen-Chau Lee, Paul M. Markowski, Gretchen L. Mullendore, and James A. Moore. Recommendations in situ and remote sensing capabilities atmospheric convection and turbulence. Bulletin of the American Meteorological Society, 99(12):2463 2470, 2018.
Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2495-2504. 2020.
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabated Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bidlot, Massimo Bonavita, Giovanna De Chiara, Per Dahigren, Dick Dee, Michail Diamantakis, Rossana Dragani, Johannes Flemming, Richard Forbes, Manuel Fuentes, Alan Geer, Leo Haimberger, Sean Hcaly, Robin J. Hogan, Elías Hólm, Marta Janisková, Sarah Keeley, Patrick Laloyaux, Philippe Lopez, Cristina Lupu, Gabor Radnoti, Patricia de Rosnay, Iryna Rozum, Freja Vamborg, Sebastien Villaume. and Jean-Noël Thépaut. The eraS global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730):199)) 1999-2049, 2020.
Ákos Horváth and Roger Davies. Simultancous retrieval cloud motion and height from polar-orbiter multiangle measurements. Geophysical Research Letters, 28( (5):2915-2991) 2001.
Haiwen Huang, Anpei Chen, Volodymyr Havrylov, Andreas Geiger, and Dan Zhang. Loftup: Learning a coordinate-based feature upsampler for vision foundation models, 2025.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht CoTracker: It is better to track together. 2023.
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. In Proc. arXiv:2410.118311 2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
10

---

Dmitri Kochkov, Janni Yuval, Ian Langmore, Peter Norgaard, Jamie Smith, Griffin Mooers, Milan Kloewer. James Lottes, Stephan Rasp. Peter Dueben, Samuel Hatfield, Peter Battaglia, Alvaro Sanchez, Matthew Willson, Michael Brenner, and Stephan Hoyer. Neural general circulation models for weather and climate. Nature, 132:1060-1000 1066, 2024.
Remi Lam, Alvaro
Matthew Willson, Peter Wimsserger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer. George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel. Shakir Mohamed, and Peter Battaglia. Learning skillful medium-range global weather forecasting. Science, 382(6677): (11777))::11114443 1421, 2023.
D.S. Lee, D. W. Fahey, A. Skowron, M.R. Allen, U. Burkhardt, Q. Chen, S.J. Doherty, S. Freeman, P.M. Forster, J. Fuglestvedt, A. Gettelman, R.R. De León, I.I., Lim, M.T. Lund, R.J. Millar, B. Owen, J.F. Penner, G. Pitari, M.J. Prather, R. Sausen, and I.J. Wileox. The contribution of global aviation to anthropogenic climate forcing for 2000 2018. Atmospheric Environment, 244: 17834, 2021.
Aviad Levis. Yoav Y Schechner, Amit Aides, and Anthony B Davis. Airborne three-dimensional cloud tomography. In Proceedings of the IEEF International Conference on Computer Vision, pages 3379-3387, 2015.
Aviad Levis, Yoav Y Schechner, and Anthony B Davis. Multpple-cattering microphysies tomography. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6740-672 6749, 2017.
A Levis, YY Schechner, AB Davis, and Loveridge. Multi-view polarimetric scattering cloud tomography and retrieval of droplet size, remote sens., 12, 2831, 2020.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa. Alaaeldin El-Nouby, Russell Howes, Po-Yao Yao Huang. Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnacve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers, arXiv preprint arXiv:2212.09744 2022.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, 2018.
David M Romps and Rusen Öktem. Observing clouds in 4d with multiview stereophotogrammeetry Bulletin of the American Meteorological Sociery, 99(12), 2018.
Roi Ronen, Vadim Holodovsky, and Yoav Y Schechner. Variable imaging projection cloud scattering tomography. IEEE Trunsactions on Pattern Analysis and Muchine Intelligence, 2022.
Yael Sde-Chen, Yoav Y. Schechner, Vadim Holodovsky, and Eshkol Eytan. 3deepet: Learning volumetric scattering tomography of clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Xingjian SHI, Zhourong Chen, Hao Wang, Din Yeung, Wai-kin Wong, and Wang-chun WOO. Convolutional Istm network: A machine learning approach for precipitation nowcasting, In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2015.
Mirjam Tijhuis, Bart J. H. van Stratum, Menno A. Veerman, and Chiel C. van Heerwaarden. An efficient parameterization for surface shortwave 3d radiative effects in large-eddy simulations of shallow cumulus clouds. Journal of Advances in Modeling Earth Systems, 15()xe20222S000322, 2023. e2022MS003262 2022MS003262.
George Tselioudis, William B. Rossow, Christian Jakob, Jasmine Remillard, Derek Tropf, and Yuanchong Zhang. Evaluation of clouds, radiation, and precipitation in cmip6 models using global weather states derived from iscep-h cloud property data. Journal of Climate, 34(17):3311 - 7324, 2021.
C. C. van Heerwaarden, B. J. H. van Stratum, T. Heus, J. A. Gibbs, E. Fedorovich, and J. P. Mellado. Microhh 1.0: a computational fluid dynamics code for direct numerical simulation and large-eddy simulation of atmospheric boundary layer flows. Geoscientific Model Development, (18):3145-3165, 2017.
Margreet C vanZanten, Bjom Stevens, Louise Nuijens, Pier Siebesma, A. S. Ackerman, F. Burnet, A. Cheng, F. Couvreux, H. Jiang. Khairoutdinov, Y. Kogan, D. C. Lewellen, D. Mechem, K. Nakamura, A. Noda, B. J. Shipway, J. Slawinska. S. Wang, and A. Wyszogrodzki. Controls on precipitation and cloudiness in simulations of wind cumulus as observed during rico. Journal of Advances in Modeling Earth Systems. 3(2), 2011.
11

---

Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstructured multi-view stereo. European Conference Computer Vision (ECCV), 2018.
Rusen Öktem, Prabhat, James Lee, Aaron Thomas, Paquita Zuidema, and David M. Romps. Stereophologram- metry of Oceanic Clouds. J. Atmos. Ocean. Tech., 3177))1118)2211150000 1501, 2014.
12

---

Appendix Table of Contents
Training Details
14
Model Details
14
B. Architecture Details
14
B.2 Cloud Homography Visualization
14
B.3 Wind Retrieval Additional Implementation Details
14
Dataset Details
16
LES Settings
16
Rendering
16
Real-World Dataset
16
Additional Experiment: LES Data
17
Additional Results
17
Ablations
17
Performance Across Different Cloud Properties
18
Additional Real-Worle Dataset Radar Examples
18
13

---

A Training Details
Table Overview of our multi-stage training pipeline. We first pre-train our synthetic data generated using Terragen, which is then followed by training on our LES data (MicroHH).
Dataset
Trainable parameters
Learning rate
Schedule
Steps
Terragen
2D CNN
1 10-4
Cosine
20000
MicroHH
2D CNN Sparse Transformer
1 x 10-5
Constant
40000
MicroHH
1 x 10-5
Constant
30000
Training configurations
The multi-stage training pipeline is outlined Table 2 We use the following hyperparameters for all stages of training:
Optimizer: Adam with (81, B2) (0.9, 0.999)
Batch Size:
Gradient Clipping:
Weight Decay: 0
Augmentations
Saturation: 0.75 1.25
Hue: 3.14
-0.05 to 3.11
0.05
Brightness: The synthetic cloud images are high-dynamic range (HDR), which we leverage for an augmentation the brightness. Specifically, map a-th percentile brightness value to a ,3-th percentile brightness value after tonemapping, where the values are randomly sampled Uniform 0.95) and Uniform(0.7, 0.9).
Random Dropping of Cameras: During training of the cloud layer 2D CNN model, we uniformly drop 0 to N-1 input views, where N is the number of camera views.
B
Model Details
B.1
Architecture Details
Our 2D CNN implementation is based on the architecture EDM Karras et al. (2022), with only input and output channel sizes being changed to fit our task. For the sparse transformer, we follow the architecture of TRELLIS (Xiang et al. 2024). where we do attention after two sparse 3D CNN downsampling blocks, and use channel sizes of 128, 256, and 384. We use 12 sparse transformer blocks, each which has 12 heads.
B.2 Cloud Homography Visualization
Figure 6 gives additional intuition into our cloud homography by visualizing the homography different altitudes. For the visualization, we use RGB images instead of DINOv2 features, where the shown RGB value is averaged across the different views. We that when the height of the considered plane in the homography matches the cloud layer height, there are minimal artifacts in the feature plane. In comparison, when the height is incorrect, the plane features do match up across the views, resulting in artifacts when averaged. By considering multiple heights using the cloud homography, our 2D CNN processes the different planes and is able estimate the correct cloud height.
B.3
Wind Retrieval Additional Implementation Details
As described in the main paper, Cloud4D retrieves horizontal wind vectors by using CoTracker3 (Karaev et al. (223)) to track our 3D cloud volumes. Here, we give additional details on how this is implemented.
Pre-processing 2D slices of our liquid water content predictions are tracked at varying altitudes to retrieve a height-varying wind profile. We find that individual slices can be noisy, and we instead consider the sum of H = 5 consecutive slices:
(2.0..)
Σ
((....).))
Pelice
==h-2
14

---

6 -500
250 m
6 0 m
250 m
500
GT LWP
Figure 6: Cloud homography visualization. We visualize the cloud homography by mapping synthetic RGB input images to different heights. denotes the height difference from the sampled height and the mean height of the cloud.
where h is the index of the height which we are considering. For each height of the summed liquid water content, we track points across T frames, where T is chosen such that the sequence long enough for noticeable cloud movement but short enough so that the tracked points are stable. In practice, we choose 20 frames with 15 seconds between each frame, resulting in tracked points over five minutes.
Tracking For each sequence of liquid water content, we scale the values to standard grayscale image range of [o, 1]. and then initialize points on the first frame for CoTracker3 track. The points are initialized by uniformly sampling 25 random pixels from the 50th percentile of the highest intensity pixels. This process is repeated for all frames, with all predicted tracks being aggregated into five-minute buckets. This improves the robustness of the wind retrieval but decreases the temporal resolution to five minutes. We extract the wind speed from each track and then take the median wind within each bucket as the final retrieved wind profile. Specifically, for each predicted track that starts (x1, V1) and ends at (I2, 82), we estimate the horizontal wind as:
8(32 xI)
s(y2 D1)
u
V
dt
where is the pixel size in meters (25 m) and di is the duration of the point tracking (five minutes).
Track Filtering We find that there are significant number of tracks that drift and track empty space. We observe that these failed tracks generally move slowly across the frames. To filter out failed tracked points, we first discard all points that CoTracker3 predicts be occluded. We then additionally only keep the tracks that have the highest 95% magnitudes in pixel displacement. This removes the failed point tracks as they have lower pixel displacements, and results in a smaller subset of high-quality tracks, which we then use for wind estimation.
Table LES Configurations. Voxel sizes and grid extents for all LES cases used for training.
Case
Voxel sizes (m)
Grid Extent (m)
Cabauw Tijhhuis et al. 2023
10 10 x 25
5120 x 5120 4000
Rico (vanZanten et al.
201T
20 20 x 20
12800 x 12800 x 4000
ARM Brown et al. 2002
10 10 x 10
6400 x 6400 4400
15

---

Dataset Details
C.1 LES Settings
The configurations for all cases used for LES training data given in Table 3 Rendering is done with the native voxel sizes and grid extents shown in the table. To emulate far-away clouds that are outside the simulated grid, the volumes rendering. For supervision, all voxel sizes are resized to 25 x 25 x 25 using trilinear interpolation, and all grid extents cropped to the center 5 km 5 km 4 km.
continuously repeated along the horizontal plane during
C.2 Rendering
The output of the simulation is grid of liquid water mixing ratio which needs to be converted to scattering coefficients for rendering. First, the mixing ratio converted droplet concentration Na by assuming a typical droplet size for cumulus clouds of 20 (Durbin 1959):
Q
N
4T p
where Pwr the density of water. The scattering cross-section scat and scattering coefficient are then calculated as:
Oscat = Qseat Tr² Na ascat
where Qecat is set to 2 for Mie scattering. The scattering probability along path A is then given by:
Pscat 1 - exp(-3 Az).
The scattering probability then used for Monte Carlo path tracing in Blender's Cycles engine.
C.3 Real-World Dataset
Our camera setup is visualized in Figure 7 and consists of three stereo camera pairs positioned in an inwards-looking triangle. The baselines vary from 190 m to 350 m, while the distances between the pairs are between 5000 to 8000 m. We note that Cloud4D does not use any stereo vision techniques and thus works on any camera array.
The cameras are synchronized by GPS, such that an image is taken every five seconds. We calibrate the cameras using real-time kinematic positioning, giving a position accurate within a centimeter. Night-time time images of stars are then used to optimize for the rotation and focal length.
Stereo Inputs
A
B
C
B
10 km
C
Figure 7: Camera configuration. The camera configuration of the real-world dataset is shown with accurate camera poses. For visual clarity, we only illustrate the left camera of each stereo pair. Cloud4D retrieves the cloud liquid water content in the middle km x km area.
16

---

Predicted
Ground Truth
3.5
Height (km)
3.0
3.0
2.5
2.5
2,0
1.0
0.5
0.5
0.0
(km)
Y (km)
0.0
0.2
0.4
0.6
0.8
1.0
kgm -2
(a) LWC summed along the x-axis
Predicted
Ground Truth
4.0
3.5
Height (km)
3.0
2.5
2.5
2.0
2.0
1.5
0.5
0.0
0.0
X (km)
X (km)
0.0
0.2
0.4
0.6
0.0
1.0
kgm -
(b) LWC summed along the y-axis
Figure 8: Synthetic data evaluation. We visualize our liquid water content predictions by summing along (a) the x-axis and (b) the y-axis. Our predictions capture the cloud envelope, including the cloud base height, cloud top height, and the plume shape. We note that the evaluated data from an unseen LES configuration.
D Additional Experiment: LES Data
The evaluation in the main paper has relied on comparisons to radar and satellite data. However, the radar retrievals only evaluate narrow vertical beam, and satellite retrievals generally focus on top-down photometric evaluation. To evaluate the side profiles of our predictions, additional -en-rreeedt volume using a configuration unseen during training (BOMEX (Siehesma) et al. 2003)). Figure shows the liquid water content summed along the x-axis and the y-axis. Our predictions retrieve accurate cloud base heights and cloud top heights, while also capturing correct plume shapes.
generate an
E
Additional Results
F.1
Ablations
In Table 4 we show an ablation study on the sparse transformer and the number of input views evaluated using radar retrievals on our real-world datasets.
Sparse transformer The addition of the sparse transformer improves predictions on all metrics (liquid water content, cloud base heights, and cloud top heights). We note that the sparse transformer
17

---

Table Ablation study on the sparse transformer and the number of input views. We note that the liquid water path and cloud occupancy remain unchanged with the sparse transformer, as it preserves the original liquid water path. For the ablation on input views, we uniformly drop camera pairs from the input.
Sparse Transformer
Model
Occ
LWC (ym=))
LWP (kgmn-3
CBH (m)
CTH (m)
2D CNN Only
0.70
0.030
0.063
202.04
299.04
Sparse Transformer
0.70
0.029
0.063
189.58
295.77
Input Views
Dropped Views
Occ
LWC (gm-
LWP (kgm 33
CBH (m)
CTH (m)
0.70
0.029
0.063
189.58
295.77
2
0.76
0.038
0.070
316.28
371.71
0.68
0.037
0.072
325.03
383.16
Table 5: Real-world performance variation across different cloud properties. We use the GT radar retrievals to classify the cloud propertics in our real-world dataset. Note that the cloud thickness refers to the vertical geometric thickness of the cloud.
Cloud Coverage
Occ
LWC (gm=3)
LWP (kgm
CBH (m)
CTH (m)
0.20 0.45
0.65
0.034
0.077
149.36
287.12
0.75 0.90
0.75
0.023
0.048
230.92
304.67
Mean CBH (m)
Occ
LWC (gm
LWP (kgm-
CBH (m)
CTH (m)
800 1350 1350 1650
0.74
0.027
0.055
198.63
281.29
0.63
0.033
0.077
173.36
321.73
Mean Cloud Thickness (m)
Occ
LWC (gm 33)
LWP (kgm-)
CBH (m)
CTH (m)
150 275
0.69
0.025
0.050
194.84 184.09
269.58
275 500
0.71
0.033
0.076
323.12
preserves the liquid water path from the 2D CNN prediction and therefore will not affect the evaluation of the liquid water path and the cloud occupancy.
Number of input views The estimation of CBH and CTH worsens as the number of views decreases, but the LWC and the LWP are less affected. The decreasing CBH and CTH accuracy with dropped views matches established results in 3D vision, where increasing the number of views improves accuracy in 3D shape estimation.
E.2 Performance Across Different Cloud Properties
[sslos results across different cloud conditions where the radar GT has been used to classify the cloud coverage, mean cloud base height, and mean cloud geometric thickness, over one-hour segments in our real-world dataset. From the results, we note slightly worse performance for higher cloud base heights and also geometrically thicker clouds, but not to significant extent.
E.3 Additional
Dataset Radar Examples
In Figures to 13 we show additional comparisons to radar retrievals.
18

---

Ours (5 seconds)
Reference (30 seconds)
3.5
55
2.5
25
Helght
20
8
15
1.9
10
0:3
0.
11:00
10:00
10:30
11:00
Time (UTC)
Time (UTC)
10-3
10
10-a
10
kg m
Ours (5 seconds)
Reference (30 seconds)
m-
0.0
0.4
0.4
0.2
10:00
10:30
11:00
10:00
10:30
11:00
Time (UTC)
Time (UTC)
Figure 9: Example 1. Additional results comparing the liquid water content and liquid water path with radar retrievals.
Ours (5 seconds)
Reference (30 seconds)
355
35
2,5
25
Height
20
15
3.9
10
6.2
0.0
12:00
10:00
10:30
11:00
Time (UTC)
Time (UTC)
10-1
10-3
10-2
kg m
Ours (5 seconds)
Reference (30 seconds)
m-
0.6
5.4
0.4
10:00
10:30
11:00
10:00
10:30
11:00
Time (UTC)
Time (UTC)
Figure 10: Example 2. Additional results comparing the liquid water content and liquid water path with radar retrievals.
19

---

Ours (5 seconds)
Reference (30 seconds)
3.5
55
2.5
25
Helght
20
di aulll
1.9
10
0.0
0.0
12:00
11:00
12:00
Time (UTC)
Time (UTC)
10
10-a
10
kg m
Ours (5 seconds)
Reference (30 seconds)
m-
0.4
0.4
Av M A.
11:00
11:30
12:00
11:00
11:30
12:00
Time (UTC)
Time (UTC)
Figure 11: Example 3. Additional results comparing the liquid water content and liquid water path with radar retrievals.
Ours (5 seconds)
Reference (30 seconds)
3.5
35
2,5
25
Height
15
3.9
10
6.2
0.0
11:00
12:00
Time (UTC)
Time (UTC)
10-1
10-2
10-2
kg m
Ours (5 seconds)
Reference (30 seconds)
mm
5.4
0.4
11:00
11:30
12:00
11:00
11:30
12:00
Time (UTC)
Time (UTC)
Figure 12: Example 4. Additional results comparing the liquid water content and liquid water path with radar retrievals.
20

---

Ours (5 seconds)
Reference (30 seconds)
3.5
35
2.5
25
Height
1.2
10
6.3
00
Time (UTC)
Time (UTC)
10aa
10"2
kg m
Ours (5 seconds)
Reference (30 seconds)
mm
0.4
0.4
10:00
10:30
11:00
10:00
10:30
11:00
Time (UTC)
Time (UTC)
Figure 13: Example 5. Additional results comparing the liquid water content and liquid water path with radar retrievals.
21

---

Appendix References
W. G. Durbin. Droplet sampling in cumulus clouds. Tellus, 11(2)202-215, 1959.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Proc. NeurIPS, 2022.
A. Pier Siebesma, Christopher S. Bretherton, Andrew Brown, Andreas Chlond, Joan Cuxart, Peter G. Duynkerke, Hongli Jiang, Marat Khairoutdinov, David Lewellen, Chin-Hof Moeng, Enrique Sanchez, Bjorn Stevens, and David E. Stevens. A large eddy simulation intercomparison study of shallow cumulus convection. Journal of the Atmospheric Sciences, 60(10):1201 1219, 2003.
Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile generation. arXiv preprint arXiv:24/2.01506, 2024.
22

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claims are backed by experimental evidence shown in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are discussed in Section ?
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
23

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The main details of method is described in Section and all other required implementation details can be found in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in the supplemental material?
Answer: [Yes]
24

---

Open access to our code and datasets will be made prior to the conference. The radar and satellite data used are publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides/Coeee Soaraddsse/ooeeStmmmiissooooeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
Ltspp..c/ppulli//guu /utie///ggiddeeedeeuuuutttiia wwtpobmmssiooooooiiiii.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3.4 describe the main training details. Any additional detail necessary is included in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Due to limitations in computational resources, we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 3.4 describe the compute resources used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section
I describe the potential impacts of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly refer to the license for ERA5 and the satellite data. Acknowl- edgements for the radar data will also be given after the blind review period.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We are releasing our code and datasets, which include relevant documentation
for use.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy hate:p///eerrippc.cc////ofeerees for what should or should not be described.
29


Reviews:
[{"review_id": "affO9MGufy", "reviewer": "Reviewer_XcpS", "summary": "The paper presents Cloud4D, a novel machine learning framework designed to estimate high-resolution, four-dimensional (3D space + time) cloud properties using synchronized ground-based cameras.\nAuthors mention a three-fold Contribution: \n* Cloud4D estimates liquid water content (LWC), cloud base height (CBH), and cloud top height (CTH) at a 25 m spatial and 5 s temporal resolution—a significant improvement over satellite and radar methods.\n* The system uses a transformer model guided by camera homographies to map 2D images into structured 3D cloud layer representations. \n*  A demonstration, on a two-month deployment with six cameras, that Cloud4D delivers an order-of-magnitude improvement in temporal resolution over space-borne products while achieving < 10% relative error against collocated radar retrievals.\n\nThe method to achieve the 3D LWC comprises of two main steps:\n* A 2.5D cloud layer model for estimating cloud layer parameters. \n* A sparse 3D transformer that refines these into 3D predictions. \n\nFinally an Horizontal Wind Estimation is presented. By tracking cloud movement over time in the 3D reconstruction, Cloud4D infers height-resolved horizontal wind vectors.", "questions": "* Important: Justification of 10% Relative Error - The abstract and main text claim \"<10% relative error against collocated radar retrievals,\" but this metric does not appear in any table or figure. Where is this value calculated, and can you provide quantitative evidence or direct references to support this claim? \n\n* Moderate: It seems that the training of Cloud4D is performed exclusively on synthetic data. Isn’t there a need to address the domain gap between synthetic and real-world observations when deploying the model on real camera and radar data?\n\n* Moderate: Did you consider implementing and comparing with other models (e.g., multi-view stereo, which is also referenced in your work)?\n\n* Minor: Ambiguity of \"K\" Symbol: In Section 3.1 and related equations, does the symbol \"K\" refer to the number of height planes sampled, the number of camera intrinsics, or both? Could you clarify its precise meaning and usage in the model pipeline?", "limitations": "While Cloud4D achieves much higher spatial and temporal resolution than satellites, it requires dense, site-specific camera deployments. In contrast, satellite data, though coarser, can be applied globally and generalized to any location without extra infrastructure. This makes Cloud4D less scalable and less universally applicable than satellite-based approaches.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n* The proposed methods achieves 25 m spatial and 5 s temporal resolution, an order-of-magnitude better than satellite products.\n* The authors introduce a homography-guided 2D-to-3D transformer pipeline, leveraging DINOv2, LoftUp, and a sparse transformer for efficient and accurate 3D cloud field estimation.\n* Height-resolved horizontal wind profiles are derived by tracking reconstructed 3D cloud fields over time, a capability not available in previous multi-view or satellite-based methods.\n* Demonstration of the method on a two-month deployment with six cameras, showing practical feasibility.\n* Using synthetic data from MicroHH (atmospheric simulation) and Terragen (cloud rendering) lets Cloud4D train and test its models on realistic, fully labeled 3D cloud scenes that aren’t possible to capture in the real world.\n\nMain Weaknesses: \n* The abstract and introduction repeatedly claim <10% relative error against collocated radar measurements, but this number does not appear in any table, figure, or explicit result in the main or supplementary material.  There are no clear, quantitative error tables or plots showing this metric, making it difficult to verify the headline claim.\n* The paper lacks comprehensive quantitative comparisons (e.g., with other methods or baselines) and detailed error analysis across different cloud types, conditions, or time periods.\n*  Generalizability to other sites, camera setups, number of views or cloud regimes is not tested or discussed in depth. Of course, this is an inherent limitation of the method, since deploying many sites and setups requires significant effort.\n* The method relies on radar retrievals for ground truth, which themselves have limitations and may not always be available or accurate, especially for shallow clouds.\n\nConclusion\n* This paper presents an innovative and state-of-the-art approach for 3D cloud reconstruction, leveraging a sophisticated 2D-to-3D transformer architecture and foundation models to achieve unprecedented spatial and temporal resolution. The use of only synthetic data for training the model  (even though it can raise domain adaptation concerns) is particularly interesting and, based on the partial comparisons to radar retrievals shown in Figure 3 and the quantitative results in Table 1, appears promising.\n* The main concern is the lack of direct evidence for the headline \"<10% relative error\" claim. This is repeated in the abstract and introduction, but to my understanding, no table, figure, or quantitative analysis substantiates it. I recommend a borderline reject, but encourage the authors to address this, as the work has significant potential.", "was_revised": true, "final_justification": "As concluded in my comment to authors rebuttal and considering the latest comments of the other reviewers I'm raising my rating.", "rebuttal": "We thank the reviewer for their feedback, and we appreciate that they find our approach to be innovative and with significant potential. Please find below our response to your comments.\n\n&nbsp;\n\n>Q1. Important: Justification of 10% Relative Error\n\nThank you for pointing this out. We agree that it is important for this to be clear.\nThe relative error specifically refers to our LWC predictions when compared to radar values and is calculated as\n\n$$\\dfrac{\\text{MAE(LWC of clouds)}}{\\text{mean(LWC of clouds from radar GT)}} = \\dfrac{0.029 gm^{-3}}{0.321 gm^{-3}} = 8.9\\\\%$$\n\nwhere the mean absolute error of our LWC predictions is the same as the value from Table 1 in the paper.\n\nWe will make this clearer in the paper.\n\n&nbsp;\n\n>Q2. Moderate: Isn’t there a need to address the domain gap between synthetic and real-world observations when deploying the model on real camera and radar data?\n\nIt would be ideal to train our model with real-world data, but there are currently no observing systems that are able to provide the full 3D observations of cloud fields at the spatial scale required. To fill this gap, we use high-resolution LES, which is a state-of-the-art simulation approach for clouds. To ensure that our model generalizes across different conditions, we simulate a number of cases with different meteorological environments.\n\nOur results validate the effectiveness of LES-based training, showing that real-world data is not required, although it may improve results further. For future work, it could be interesting to investigate how real-world data might be integrated with simulation-based training.\n\n&nbsp;\n\n>Q3. Moderate: Did you consider implementing and comparing with other models (e.g., multi-view stereo, which is also referenced in your work)?\n\nMulti-view stereo only reconstructs surfaces, so it is not able to estimate cloud properties across the 3D volume. However, we have tried volumetric methods including MVSNeRF [1], IBRNet [2], and Gaussian Splatting [3]. We found that all of these models severely failed due to the limited number of camera views and because the domain is very different from that on which they were trained. \n\nFor MVSNeRF and IBRNet, we additionally tried fine-tuning on our synthetic data but found that this did not significantly improve results. This is not unexpected as these models are generally trained on synthetic datasets with much more than six camera views. \n\n&nbsp;\n\n>Q4. Minor: In Section 3.1 and related equations, does the symbol \"K\" refer to the number of height planes sampled, the number of camera intrinsics, or both?\n\nThank you for pointing this out. K refers to the number of height planes sampled, and we will update the notation to avoid confusion with the camera intrinsics that are denoted with a bolded K. We denote the number of images (which are the same as the number of intrinsics) as N.\n\n&nbsp;\n\n>Weakness 1. Generalizability to other sites, camera setups, number of views or cloud regimes is not tested or discussed in depth. Of course, this is an inherent limitation of the method, since deploying many sites and setups requires significant effort.\n\nWe agree that testing the generalization to other sites would be useful. However, we would like to point out that our current model is trained exclusively on synthetic data and is never trained on data from our real-world test site. Only the camera poses in our synthetic data match our real-world setup; the other characteristics are decoupled from the real test site.  Our synthetic dataset includes a wide variety of simulated cloud morphologies, atmospheric conditions, and sun positions that are not unique to our physical location. Therefore, the model is forced to learn the underlying principles of 3D cloud reconstruction from multi-view imagery, rather than overfitting to the specific visual features (e.g., background, typical weather patterns) of a single site. Our real-world site, therefore, acts as a generalization test in itself.  \n\nWe have also investigated our model's performance across different conditions in our real-world dataset, including different numbers of camera views, cloud coverages, cloud base heights, and cloud geometric thicknesses. These results are included below, where camera pairs are randomly and uniformly dropped, and the radar GT has been used to classify the cloud coverage, mean cloud base height, and mean cloud geometric thickness, over one-hour segments in our real-world dataset.\n\nOur key takeaways from this investigation are the following:\n\n(1) The estimation of CBH and CTH worsens as the number of views decreases, but the LWC and the LWP are less affected. The decreasing CBH and CTH accuracy with dropped views matches established results in 3D vision, where increasing the number of views improves accuracy in 3D shape estimation.\n\n(2) We observed slightly worse performance for higher cloud base heights and also geometrically thicker clouds, but not to a significant extent. \n\n| Number of Dropped Views &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; Occ &nbsp; &nbsp; | &nbsp; LWC (gm⁻³) &nbsp; | &nbsp; LWP (kgm⁻³) &nbsp; | &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; | &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 0 | 0.70 | **0.029** | **0.063** | **189.58** | **295.77** |\n| 2 | **0.76** | 0.038 | 0.070 | 316.28 | 371.71 |\n| 4 | 0.68 | 0.037 | 0.072 | 325.03 | 383.16 |\n\n&nbsp;\n\n| Cloud Coverage &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; Occ &nbsp; &nbsp; | &nbsp; LWC (gm⁻³) &nbsp; | &nbsp; LWP (kgm⁻³) &nbsp; | &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; | &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 0.20 - 0.45 | 0.65 | 0.034 | 0.077 | **149.36** | **287.12** |\n| 0.75 - 0.90 | **0.75** | **0.023** | **0.048** | 230.92 | 304.67 |\n\n&nbsp;\n\n| Mean CBH (m) &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; Occ &nbsp; &nbsp; | &nbsp; LWC (gm⁻³) &nbsp; | &nbsp; LWP (kgm⁻³) &nbsp; | &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; | &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 800 - 1350 | **0.74** | **0.027** | **0.055** | 198.63 | **281.29** |\n| 1350 - 1650 | 0.63 | 0.033 | 0.077 | **173.36** | 321.73 |\n\n&nbsp;\n\n| Mean Cloud Thickness (m) &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; Occ &nbsp; &nbsp; | &nbsp; LWC (gm⁻³) &nbsp; | &nbsp; LWP (kgm⁻³) &nbsp; | &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; | &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 150 - 275 | 0.69 | **0.025** | **0.050** | 194.84 | **269.58** |\n| 275 - 500 | **0.71** | 0.033 | 0.076 | **184.09** | 323.12 |\n\n&nbsp;\n\n>Weakness 2. The method relies on radar retrievals for ground truth, which themselves have limitations and may not always be available or accurate, especially for shallow clouds.\n\nWe agree that radars have their own biases and limitations. We would like to note that the liquid water content retrieval of the radar also includes information from a coincident lidar and is also scaled to match a microwave radiometer [4]. It is the best possible evaluation dataset for this deployment, but we note the possibility of errors in this data.\n\n&nbsp;\n\n[1] Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo, Chen et al. (ICCV 2021)\n\n[2] IBRNet: Learning Multi-View Image-Based Rendering, Wang et al. (CVPR 2021)\n\n[3] 3D Gaussian Splatting for Real-Time Radiance Field Rendering, Bernhard et al. (SIGGRAPH 2023) \n\n[4] Cloudnet: Continuous Evaluation of Cloud Profiles in Seven Operational Models Using Ground-Based Observations, Illingworth et al. (AMS 2007)"}, {"review_id": "jg7JgpE6rR", "reviewer": "Reviewer_nv5b", "summary": "Authors propose Cloud4D, a method to estimate cloud properties, such as liquid water content (LWC), liquid water path (LWP), cloud base height (CBH) and cloud top height (CTH), from multiple ground-based images.  \n\nThe method can be summarized as follows:\n\n * collect N ground-based images of clouds, together with the camera poses and intrinsics;\n * estimate an homography that maps the image space to the cloud plane, in order to estimate the height of the plane the clouds reside on;\n * to estimate cloud properties, DINOv2 is applied to the images, followed by the LoftUp method to reproject the DINOv2 features to the image space. The previously learnt homography projects the features to the cloud plane;\n * the clouds of interest, i.e. cumulus clouds, are characterized by their vertically thin nature, meaning that the main variation in cloud structure happens in the horizontal plane. With this assumption, authors perform the cloud estimation layer as a 2Dto-2D task, using 2D convolutions, which estimate the desired cloud properties;\n * a 3D refinement phase is applied to obtain a 3D estimate from the 2.5D features using a sparse transformer initialized with a basic model of a cloud volume, which is further fused with the projected features derived from DINOv2 and LoftUp, reconstructing 3D profiles for LWC;\n * as a final step, wind motion estimation is provided leveraging the reconstructed 3D LWC cloud profiles and the high temporal resolution (5 s) of the camera. Wind tracking is performed using an off-the-shelf point tracker.  \n\nDue to the lack existing datasets on cloud properties, authors validated the method on simulated and observed data, acquired over a two-month deployment of the ground-based cameras. Results show superiority in estimation of cloud properties, although the compared methods are very different in nature and very scarce.", "questions": "Here are some questions and suggestions I have for the authors:\n\n * could the authors provide more context about how the system is meant to be deployed? From the paper is unclear to me whether the system is designed to provide monitoring over large areas, or whether it is rather designed to characterize cumulus clouds at a smaller scale, with the aim of informing the evolution of weather patterns. If the deployment intention is the former, I would like to know about the requirements in terms of number of cameras required to cover, say, a 100kmx100km area, and which communication/collection infrastructure would be required;\n * as mentioned in the weaknesses above, and as the authors recognize in the Limitations Section, I don;t see how the proposed method could be generalized to estimate cloud properties of different cloud types, namely thicker clouds higher in the atmosphere. Such clouds would violate the assumptions and design choices of Cloud4D;\n* authors are probably aware, but there are more suited satellite images available than Sentinel-2, e.g. PlanetScope from Planet Labs, which provide a higher temporal revisit time, and often provide multiple takes within minutes of one another. Such imagery could be helpful in validating the cloud profiles, as well as wind tracking module.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Estimating cloud properties with adequate spatial and temporal resolution is a very relevant and important task for accurate modeling of weather and climate variables. The paper therefore proposes a very interesting and alternative approach for accurate estimation of 4D cloud properties, clearly outlining the motivation behind design choices and validating the method on simulated and observed datasets.\n\nThe paper is original, very well written and clear. Due to the niche application, available validation datasets are scarce, as well as comparable methods. This leads to the use of ad-hoc simulated datasets and a limited observed dataset. Despite this, the potential of the method is in my opinion demonstrated, since authors seek to evaluate their method against very diverse methods for estimation of cloud properties, namely radar measurement, satellite images, and wind profiles.\n\nThe main strength I see is the careful design of the imaging and processing system for the modelling of cumulus clouds, which are indeed challenging to characterize with other methods, e.g. remote sensing. However, on the other hand, I feel that this assumption also represents the main weakness of the method, since I don't believe the 2.5D representation and 2D-to-2D cloud property estimation would be valid for more thick types of clouds and clouds higher in the atmosphere.", "was_revised": true, "final_justification": "I maintain my initial score, as the authors have satisfactorily responded to my questions, and to the ones of my colleagues. My main concerns regarded the deployment at scale of the proposed approach, as well as generalization to other types of cloud.\n\nAuthors provided details about possible deployment scenarios, and some pointers on generalization.", "rebuttal": "We thank the reviewer for their feedback and appreciate that they find our approach interesting and the target task to be very relevant and important. Please find below our response to your comments.\n\n&nbsp;\n\n>Q1. Could the authors provide more context about how the system is meant to be deployed?\n\nWe envision the following possible deployment scenarios:\n\n(1) Small-scale at a single site: this would allow studying cloud physical properties and processes, such as formation and dynamics, which is not possible with existing instruments. It could allow the development and evaluation of parameterisations for weather and climate models. It could also provide a satellite validation site.\n\n(2) Large-scale with sparse coverage: this would be deployed similarly to existing radar and meteorological instrument sites which span a large area but have non-overlapping coverage. The output of this deployment could feed into NWP models or be used for scientific studies.\n\n(3) Large-scale with dense coverage: while dense coverage is more difficult, here we would aim to cover one weather model grid box (about 25km x 25km) to provide a 3D reconstruction of a portion of the cloud field spanning the full grid-box element. This would be useful for validating and improving weather/climate models.\n\n&nbsp;\n\n>Q2. If the deployment intention is the former, I would like to know about the requirements in terms of number of cameras required to cover, say, a 100kmx100km area, and which communication/collection infrastructure would be required\n\nA dense deployment covering a 25km x 25km area would require the following:\n\n**Number of cameras**. The most basic scaling of our deployment would be to duplicate our current camera setup across a grid. As our current setup is covering 5km x 5km using 6 cameras, scaling up to the sizes of a weather model gridbox ~(25km x 25km) would require 6 * 25 = 150 cameras. Given that our cameras are commodity cameras, this would still be cheaper than existing measurement devices such as radars. For future work, it could be of interest to investigate alternative camera setups for larger areas.\n\n**Infrastructure**. The cameras have low power consumption and are powered by an external portable solar panel. Time synchronization and transfer of camera data to an external server is done using a standard 4G cellular connection through a Raspberry Pi.\n\n&nbsp;\n\n>Q3. I don’t see how the proposed method could be generalized to estimate cloud properties of different cloud types, namely thicker clouds higher in the atmosphere\n\nWe would like to highlight that while an *individual cloud* might be more vertically thick for other cloud types, the vertical thin and horizontally wide structure still holds for any overall cloud *field*.\n\nThis vertical structure is a property of the atmosphere in general and also motivates why weather models such as the Integrated Forecasting System (ECMWF) and GraphCast [1] have gridboxes which are also vertically thin and horizontally wide.\n\n&nbsp;\n\n>Q4. There are more suited satellite images available than Sentinel-2, e.g. PlanetScope from Planet Labs\n\nPlanetScope is a good candidate for testing in the future, but as data access is only granted on request, we have chosen to compare against MODIS and Sentinel-2, as their data is publicly available and commonly used for cloud physics research. \n\n&nbsp;\n\n[1] Learning skillful medium-range global weather forecasting, Lam et al. (Science 2023)"}, {"review_id": "9xDoungdmk", "reviewer": "Reviewer_e4kC", "summary": "This paper presents Cloud4D: a model which predicts cumulus cloud presence, liquid water content, liquid water path, cloud base height and cloud top height from ground based cameras at fine temporal and spatial resolutions. The motivation for this work is to tackle limitations in how such cloud systems are resolved in existing numerical / ML climate models or satellite-based detection models. The Cloud4D modelling framework uses homography and convolutional layers to process image features extracted from ground based camera images by DinoV2 into 2D measures of cloud properties. These cloud properties are then used as initial features in a 3D refinement network that estimates the liquid water content in a voxel. Wind speed is estimated by tracking the movement of reconstructed clouds. The model was trained using a synthetic dataset produced using large eddy simulations and evaluated on a real world dataset captured by cameras and radar over a two month period. The model was shown capture features in ground based observations for liquid water content and path relative to ERA5 reanalysis data and to align with satellite observations of clouds.", "questions": "The comparison with the VIP-CT method could have been described and explained in more detail. This setup for using VIP-CT was not introduced in the methods section and, given that VIP-CT is designed for satellite images, how effective is this as a comparison model when applied to ground based camera systems? \n\nIt appears as though this system in part relies upon ground based camera systems. Are these just radar images does the model require optical images too? If the model requires optical images, does this mean the system is only capable of operating during daytime? What impact does this have on being able to fully capture the dynamics of cloud systems and to utilise model outputs in downstream tasks? \n\nA related question is how scalable this method is if it relies on ground camera systems? I assume that means it cannot realistically have global coverage? What implications does that have for using the outputs of the Cloud4D model? I would be interested in the authors thoughts and comments on the benefits of investing efforts in furthering ground based systems as opposed to improving satellite-derived monitoring? Is there a middle ground where select ground based systems are used to generate training / evaluation data to refine satellite-based models?", "limitations": "There is a limitations section. However, see the comments in the questions section above regarding discussion of some limitations of this work. My rating now sits between borderline accept and accept. I would err towards the former as, while I found the method to be novel and the author's have explained its relevance / application, I still believe the validation in the paper to be quite limited.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "This paper picks off a challenging topic in trying to model cumulus clouds and fine spatial and temporal resolutions. It presents a relatively complex workflow to address this challenge with a model that converts several ground based images, captured at a point in time, into a 3D dataset of several cumulus cloud properties. In the main, this workflow was well presented and easy to follow. The authors also collect a detailed, but localised, ground truth dataset consisting of two months of real world camera observations to evaluate the model. This dataset is a valuable contribution and allows for evaluating the model in considerable detail. However, the localised nature of this dataset left questions as to how generalisable this approach is. From what I could see in the paper or supplementary material, the geographic location or season of this dataset was not presented. \n\nIt was good that the authors mentioned limitations of using ground based cameras due to weather-based occlusion. However, there was scope to discuss other limits of using ground based cameras including nighttime captures and scalability and what this means for broad applicability of the Cloud4D approach. \n\nThe discussion of the results was quite superficial in places. For example, a figure comparing Cloud4D wind speeds, ERA5 wind speeds and ground truth wind speeds was presented and not discussed. The main results table presents average errors across time and position of clouds / water content / wind speed within the viewed area. It would be good to comment on how much variability there was in model error and the characteristics of the error.", "was_revised": true, "final_justification": "I thank the author's for their engagement and discussion. I have revised my ranking upwards to accept. This is mainly a reflection of the novelty of the approach, the author's explanation of training synthetic data supporting generalisation, and providing additional validation on another dataset (while noting the challenges of accessing validation datasets for this task). I would still flag that the narrow validation undertaken here is a limitation and I felt at times the author's attempts to contextualise the application of this approach was quite general (i.e. suggesting ways it could work at many scales and for many applications as opposed to focusing on the core problem area of relevance).", "rebuttal": "We thank the reviewer for their feedback and appreciate that they acknowledge the difficulty of the target task and that they find our dataset to be a valuable contribution. Please find below our response to your comments.\n\n&nbsp;\n\n>Q1. The comparison with the VIP-CT method could have been described and explained in more detail\n\nVIP-CT takes as input satellite-based images ($\\mathbb{R}^{1 \\times H \\times W}$) and outputs a 3D field ($\\mathbb{R}^{N_x \\times N_y \\times N_z}$). This is the same as our setup, with the only difference being that satellite imagery is grayscale while ours is RGB. This only requires a simple change in the number of channels for the first convolutional layer.\n\nWe use the publicly available training code for VIP-CT and instead train using our synthetic dataset, which consists of ground-based images rather than satellite-based images.\n\nWe will add this description to the paper. Thank you for pointing this out.\n\n&nbsp;\n\n>Q2. VIP-CT is designed for satellite images, how effective is this as a comparison model when applied to ground based camera systems?\n\nA key difference between satellite images and ground-based images is that the viewing angles greatly differ across ground-based views. The lack of explicit camera geometry modeling in VIP-CT can therefore limit its effectiveness with our ground-based images. However, as there is no prior work that predicts volumetric 3D cloud properties from ground-based cameras, we use VIP-CT as the closest comparable model.\n\n&nbsp;\n\n>Q3. Are these just radar images does the model require optical images too?\n\nTo clarify, our model estimates full 3D physical cloud properties from **only optical (RGB) images**. No radar images are required, as we only use radar for the evaluation of our model.\n\n&nbsp;\n\n>Q4. If the model requires optical images, does this mean the system is only capable of operating during daytime?\n\nThe cameras we have deployed only operate during the daytime. However, we argue this is not a significant limitation as:\n\n(1) Cumulus clouds generally require solar heat to form over land and are therefore mostly present during daytime.\n\n(2) There are alternative cameras that can operate at night and are capable of observing clouds. \n\n&nbsp;\n\n>Q5. How scalable this method is if it relies on ground camera systems? I assume that means it cannot realistically have global coverage?\n\nOur setup is scalable in the sense that it uses commodity cameras, which are cheap, so that new sites can be easily deployed. However, we do not intend for the setup to be deployed in a manner such that global areas are densely monitored by ground-based cameras. We envision them being deployed sparsely at sites of interest to cover large areas. This is because satellites are still the best option for global coverage, and the measurements from Cloud4D are complementary, providing high spatial and temporal resolution measurements for focused areas. \n\n&nbsp;\n\n>Q6. Benefits of investing efforts in furthering ground based systems as opposed to improving satellite-derived monitoring?\n\nWe see the ground-based system as complementary to satellites for two reasons. Firstly, the ground-based system has a much higher temporal and spatial resolution. Even with advanced satellites, it would be impossible to get data at 5s timesteps at 25m resolution, useful for fast forming convective clouds like cumulus. Secondly, in the visible spectrum, satellites observe mainly the tops of the clouds, while ground-based cameras are able to provide complementary observations of cloud bases.\n\n&nbsp;\n\n>Q7. Is there a middle ground where select ground based systems are used to generate training / evaluation data to refine satellite-based models?\n\nAbsolutely, we believe this is a key potential application of our method. Ground-based cameras can provide additional spatial detail and temporal context, while satellites have the advantage of global spatial coverage.\n\n&nbsp;\n\n>Weakness 1. However, the localised nature of this dataset left questions as to how generalisable this approach is. \n\nWe agree that testing the generalization to other sites would be useful. However, we would like to point out that our current model is trained exclusively on synthetic data and is never trained on data from our real-world test site. Only the camera poses in our synthetic data match our real-world setup; the other characteristics are decoupled from the real test site.  Our synthetic dataset includes a wide variety of simulated cloud morphologies, atmospheric conditions, and sun positions that are not unique to our physical location. Therefore, the model is forced to learn the underlying principles of 3D cloud reconstruction from multi-view imagery, rather than overfitting to the specific visual features (e.g., background, typical weather patterns) of a single site. Our real-world site, therefore, acts as a generalization test in itself.\n\n&nbsp;\n\n>Weakness 2. From what I could see in the paper or supplementary material, the geographic location or season of this dataset was not presented. \n\nWe have not included the geographic location to minimize identifying information in the double-blind review process. However, we will include precise coordinates and timestamps together with the public release of our real-world dataset.\n\n&nbsp;\n\n>Weakness 3. It would be good to comment on how much variability there was in model error and the characteristics of the error\n\nFor the rebuttal, we have investigated our model's error across different conditions in our real-world dataset, including different numbers of camera views, cloud coverages, cloud base heights, and cloud geometric thicknesses. \n\nOur key takeaways from this investigation are the following:\n\n(1) The estimation of CBH and CTH worsens as the number of views decreases, but the LWC and the LWP are less affected. The decreasing CBH and CTH accuracy with fewer views matches established results in 3D vision, where increasing the number of views improves accuracy in 3D shape estimation.\n\n(2) We observed slightly worse performance for higher cloud base heights and also geometrically thicker clouds, but not to a significant extent.\n\nPlease see our response to reviewer XcpS for the full numerical results."}, {"review_id": "e05Wa2lcQ3", "reviewer": "Reviewer_oz8q", "summary": "This paper presents a scheme for 4D cloud estimation (3D cloud with time). In the first (course)-level, they leverage homography and in the second-level, they trained a model for more fine-grained cloud prediction. The authors use simulation for training and collect their own real-world datasets for validation. Previous work either uses implicit geometry or focus solely on cloud base heights, and this is the first work that scale to real-world scenarios.", "questions": "**Dataset Generation**:\n1. Could the authors elaborate on the synthetic dataset generation process?\n2. How was the data split between training and test sets?\n3. Please clarify the \"three scenarios\" mentioned in Line 180.\n4. Were different lighting conditions (such as sun azimuth) considered, and was lighting augmentation performed?\n5. How accurately does the LES simulation reflect real-world conditions, and is this a standard methodology?\n6. What types of real-world clouds cannot be simulated using cloud simulation?\n7. It would be beneficial to include visualizations comparing synthetic and real-world data.\n\n**Training and Fine-tuning**:\n1. Has consideration been given to using Sentinel-2 imagery for model fine-tuning?\n2. Given the two-month real-world dataset and Sentinel-2's average 5-day revisiting time, it would be valuable to explore whether real-world fine-tuning improves performance.\n3. Does scaling up the training dataset size help?\n\n**Quantitative Evaluation**:\n1. The paper lacks comprehensive quantitative evaluation of real-world data performance.\n\n**Alternative Approaches**:\n1. Have alternative methods such as NERF or DUST3R been considered for 3D estimation?\n2. How do these methods compare to the proposed approach?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength: \n1. This paper is well-written and the idea is brought forward pretty neat and clear. \n2. This is a very interesting cloud estimation problem and the authors move beyond 3D geometry (like CBH or CTH) and include LWC, LWP. This allows for precise estimation of the Earth atmosphere at a very fine-grained estimation.\n3. The idea of validation from a bird-eye view using Sentinel-2 and MODIS is interesting and helpful to validate this scheme and the results are pretty astonishing to me - Cloud prediction and shadow estimation have been a long under-explored problem and this piece of work can offer as a new labeling technique. \n\nWeakness:\n1. I think the comparison is not thorough and it would be helpful to include methods from other domains. But this does not compromise the quality and significance of this work", "was_revised": true, "final_justification": "The authors answered my questions and resolved my concerns. I recommend an acceptation of this work.", "rebuttal": "We thank the reviewer for their feedback and are happy to hear that they find the cloud estimation task very interesting and that they were impressed with the results in our satellite-based comparison. Please find below our response to your comments.\n\n&nbsp;\n\n### Dataset Generation\n\n>Q1. How was the data split between training and test sets?\n\nFor training, we use high-resolution LES, which is a state-of-the-art simulation approach for clouds. To ensure that our model generalizes across different conditions, we simulate a number of cases with different meteorological environments. In practice, we use a set of well-studied meteorological conditions and refer to these as “scenarios”.\n\nTo ensure minimal leakage between training and test sets for the synthetic part of the evaluation, we do the split based on the scenarios.\n\nFor training, we use the three scenarios mentioned in Line 180 of the paper: ARM [1], Cabauw [2], and RICO [3].\n\nFor testing, we use: BOMEX [4]\n\n>Q2. Please clarify the \"three scenarios\" mentioned in Line 180.\n\nSee answer to Q1 above.\n\n>Q3. Were different lighting conditions (such as sun azimuth) considered, and was lighting augmentation performed?\n\nYes, during the data generation, we randomly sample the sun position. We additionally sample the air density and the dust density, both of which also affect the lighting conditions.\n\nImage augmentations are applied during training, where we vary the saturation, hue, and brightness. For full details, we refer to the training details that can be found in Appendix A of the supplemental material.\n\n>Q4. How accurately does the LES simulation reflect real-world conditions, and is this a standard methodology?\n\nYes, LES is a standard methodology for the simulation of high-resolution clouds. It is commonly used for the study of cloud development, and the simulations typically agree well with real-world measurements for cases with shallow convection (as in our setting) [4]. \n\n>Q5. What types of real-world clouds cannot be simulated using cloud simulation?\n\nThere are no specific limits to the kinds of clouds that can be simulated. However, as the microphysics (behaviour of droplets and ice crystals) cannot be simulated directly, clouds involving complex precipitation processes or a mixture of liquid and ice are more challenging to simulate. We focus on non-precipitating shallow cumulus, which are commonly studied through simulation. \n>Q6. It would be beneficial to include visualizations comparing synthetic and real-world data.\n\n\nThank you for the suggestion. We will add an example of this to the supplementary.\n\n&nbsp;\n\n### Training and Fine-tuning\n\n>Q7. Has consideration been given to using Sentinel-2 imagery for model fine-tuning? Given the two-month real-world dataset and Sentinel-2's average 5-day revisiting time, it would be valuable to explore whether real-world fine-tuning improves performance.\n\nEven with a 5-day revisiting time, Sentinel-2 only visits the area ~12 times over the two months our cameras were deployed. Of these visits, some are completely clear skies, and others are dominated by non-cumulus clouds. Unfortunately, this means that there is not enough data for fine-tuning at the current stage.\n\n>Q8. Does scaling up the training dataset size help?\n\nYes, from initial experiments, we found that increasing the number of scenarios used for training helped with the robustness of our model. This is likely due to the increased diversity in meteorological conditions in our training data resulting in better generalization.\n\n&nbsp;\n\n### Quantitative Evaluation\n\n>Q9. The paper lacks comprehensive quantitative evaluation of real-world data performance.\n\nTable 1 in the paper shows quantitative metrics compared to radar retrievals using our two-month real-world dataset, where we have evaluated the performance on both retrieving physical cloud properties (liquid water content and liquid water path) and geometric properties (cloud occupancy, cloud base heights, cloud top heights). \n\nAs part of the rebuttal, we have additionally quantitatively investigated our model's error across different conditions in our real-world dataset, including different numbers of camera views, cloud coverages, cloud base heights, and cloud geometric thicknesses. \n\nOur key takeaways from this investigation are the following:\n\n(1) The CBH and CTH error increases as the number of views decreases, but the LWC and the LWP are less affected. The decreasing CBH and CTH accuracy with fewer views matches established results in 3D vision, where increasing the number of views improves accuracy in 3D shape estimation.\n\n(2) We observed slightly worse performance for higher cloud base heights and also geometrically thicker clouds, but not to a significant extent.\n\nFor additional details, we have included the full numerical results in our response to reviewer XcpS.\n\n&nbsp;\n\n### Alternative Approaches\n\n>Q10. Have alternative methods such as NERF or DUST3R been considered for 3D estimation? How do these methods compare to the proposed approach?\n\nDUSt3R only reconstructs surfaces, so it is not able to estimate cloud properties across the 3D volume. However, we have tried volumetric methods including MVSNeRF [5], IBRNet [6], and Gaussian Splatting [7]. We found that all of these models severely failed due to the limited number of camera views and because the domain is very different from that on which they were trained. \n\nFor MVSNeRF and IBRNet, we additionally tried fine-tuning on our synthetic data but found that this did not significantly improve results. This is not unexpected as these models are generally trained on synthetic datasets with much more than six camera views.\n\n&nbsp;\n\n[1] Large-eddy simulation of the diurnal cycle of shallow cumulus convection over land, Brown et al. (Quarterly Journal of the Royal Meteorological Society 2002)\n\n[2] An efficient parameterization for surface shortwave 3d radiative effects in large-eddy simulations of shallow cumulus clouds, Tijhuis et al. (Journal of Advances in Modeling Earth Systems 2023)\n\n[3] Controls on precipitation and cloudiness in simulations of trade-wind cumulus as observed during rico, vanZanten et al. (Journal of Advances in Modeling Earth Systems 2011)\n\n[4] A large eddy simulation intercomparison study of shallow cumulus convection, Siebesma et al. (Journal of the Atmospheric Sciences 2003)\n\n[5] Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo, Chen et al. (ICCV 2021)\n\n[6] IBRNet: Learning Multi-View Image-Based Rendering, Wang et al. (CVPR 2021)\n\n[7] 3D Gaussian Splatting for Real-Time Radiance Field Rendering, Bernhard et al. (SIGGRAPH 2023)"}, {"review_id": "WDlpxy2LFQ", "reviewer": "Reviewer_fVbw", "summary": "The paper introduces Cloud4D, a novel learning-based framework designed to estimate cloud properties at high spatial and temporal resolutions using synchronized ground-based cameras. The framework leverages a homography-guided 2D-to-3D transformer architecture to predict the full spatial distribution of liquid-water content and horizontal wind vectors. Through a two-month real-world deployment with six cameras, Cloud4D demonstrates a significant improvement in space-time resolution compared to state-of-the-art satellite measurements while maintaining single-digit relative error against collocated radar measurements. In summary, the paper presents a groundbreaking approach to cloud property estimation that addresses the limitations of traditional methods and offers a scalable solution for high-resolution cloud observations", "questions": "1. The authors mention that Cloud4D is primarily trained on cumulus clouds and focuses on single-layer cloud properties. While this is well-motivated for ground-based cameras, the limitation to specific cloud types and single-layer clouds restricts the framework's applicability. Could the authors elaborate on the challenges and potential strategies for extending Cloud4D to other cloud types (e.g., stratocumulus, cirrus) and multi-layer clouds?  How might the framework handle occlusions and distinguish between different cloud layers? \n﻿\n2. The authors should consider providing error bars or other statistical measures for the reported metrics. Additionally, explaining the factors contributing to the variability (e.g., train/test splits, initialization) would strengthen the statistical significance of the findings.\n﻿\n3. The paper compares Cloud4D with radar, satellite, and other instruments, but the comparisons with existing machine learning-based cloud retrieval methods are somewhat limited.How does the explicit modeling of camera geometry in Cloud4D contribute to its performance compared to the implicit learning approaches used?\n﻿\n4. The paper does not include a detailed theoretical analysis of the proposed framework, such as the assumptions made in the homography-guided 2D-to-3D transformation and the limitations of the sparse-voxel refinement approach.\n﻿\n5. The authors could include experiments or simulations to validate the framework's performance under various environmental conditions. How does Cloud4D handle partially obscured views or low visibility conditions? Are there any preprocessing steps or modifications to the framework that can mitigate the impact of these conditions?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "### Strengths:\n﻿\n- The paper is technically sound and presents a novel framework (Cloud4D) that effectively addresses the challenge of estimating cloud properties at high spatial and temporal resolutions. The proposed method demonstrates significant improvements over existing satellite measurements, achieving an order-of-magnitude enhancement in temporal resolution while maintaining high accuracy (less than 10% relative error) when compared to radar measurements.\n- The authors provide comprehensive experimental validation, including comparisons with radar, satellite, and wind profiler data, which strengthens the credibility of their claims. The use of a two-month real-world deployment with six cameras further supports the robustness of the framework.\n\n### Weaknesses:\n﻿\n- While the experimental results are promising, the paper does not provide error bars or confidence intervals for the reported metrics.\n﻿\n- Some sections could benefit from additional details. For example, the implementation details of the wind retrieval method could be elaborated further to provide a clearer understanding of how the horizontal wind vectors are estimated from the reconstructed cloud fields.\n﻿\n- While the combination of techniques presented in the paper is novel, some components of the framework, such as the use of vision foundation models (e.g., DINOv2) and sparse transformers, have been explored in other computer vision applications.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and appreciate that they find our approach \"groundbreaking\" and the evaluation to be comprehensive. Please find below our response to your comments.\n\n&nbsp;\n\n>Weakness 1. While the experimental results are promising, the paper does not provide error bars or confidence intervals for the reported metrics\n\nThe main source of randomness in our experiments comes from the seed used for the weight initialization and dataset sampling during training. The model takes over 280 H100 hours to train, so running multiple full training runs is not feasible. To test the model variability, we have included results below on the mean and the 2-sigma error over five runs with random seeds trained for 9k steps (the full training run takes 90k steps). The results show that our method is robust to randomness in initialization and data sampling.\n\n| &nbsp; &nbsp; &nbsp; &nbsp; Occ &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; LWC (gm⁻³) &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; LWP (kgm⁻³) &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; &nbsp; &nbsp; |\n| :---: | :---: | :---: | :---: | :---: |\n| 0.53 $\\pm$ 0.06 | 0.0319  $\\pm$ 0.018| 0.0764 $\\pm$ 0.0008 | 160.81 $\\pm$ 15.32 | 285.20 $\\pm$ 39.34 |\n\n&nbsp;\n\n>Weakness 2. the implementation details of the wind retrieval method could be elaborated further.\n\nAdditional implementation details of the wind retrieval method are given in Appendix B.3. We are happy to expand on this if there is anything missing. We will also release our code, which includes the full implementation details.\n\n&nbsp;\n\n>Weakness 3. While the combination of techniques presented in the paper is novel, some components of the framework, such as the use of vision foundation models (e.g., DINOv2) and sparse transformers, have been explored in other computer vision applications\n\nWe thank the reviewer for acknowledging the novelty of our proposed system. We agree that foundational models like DINOv2 and sparse transformers are established tools. Indeed, our motivation was precisely to leverage their representational power to solve the challenging task of estimating 3D cloud properties from 2D images, which has not been done before.\n\n&nbsp;\n\n>Q1. Could the authors elaborate on the challenges and potential strategies for extending Cloud4D to other cloud types (e.g., stratocumulus, cirrus) and multi-layer clouds?\n\nAs most cloud fields follow a vertically thin and horizontally wide structure, our method can be applied to other cloud types too. This could be accomplished by generating synthetic data for other cloud types. \n\nWhile our cloud layer estimation model is currently designed for single cloud layers, it could be extended to multiple cloud layers. For example, instead of predicting 2.5D cloud features $\\mathbb{R}^{N_x \\times N_y \\times 3}$, you could estimate additional channels to represent higher layers $\\mathbb{R}^{N_x \\times N_y \\times 3L}$, where L would be the number of cloud layers to predict and would realistically not need to be more than two or three. \n\n&nbsp;\n\n>Q2. How might the framework handle occlusions and distinguish between different cloud layers?\n\nOur current model is designed for single cloud layers. See our comment to Q1 above for suggestions on how this could be extended.\n\n&nbsp;\n\n>Q3. The authors should consider providing error bars or other statistical measures for the reported metrics\n\nSee answer to Weakness 1 above.\n\n&nbsp;\n\n>Q4. Explaining the factors contributing to the variability (e.g., train/test splits, initialization) would strengthen the statistical significance of the findings\n\n**Train/test splits.** To ensure minimal leakage between training and test sets for the synthetic part of the evaluation, we do the split based on different meteorological environments used to configure the LES-based generation of synthetic cloud data.\n\nFor training, we use the following three meteorological conditions: ARM [1], Cabauw [2], and RICO [3].\n\nFor testing, we use BOMEX [4].\n\n**Initialization.** We initialize our model weights using a Xavier uniform distribution.\n\n&nbsp;\n\n>Q5. How does the explicit modeling of camera geometry in Cloud4D contribute to its performance compared to the implicit learning approaches used?\n\nOur camera setup consists of sparse views that are spaced far apart. As a result, the input images from the different views will greatly differ. The explicit modelling of camera geometry makes it easier to learn how to predict full 3D cloud properties from the sparse input images.\n\n&nbsp;\n\n>Q6. Assumptions made in the homography-guided 2D-to-3D transformation and the limitations of the sparse-voxel refinement approach\n\nFind below a list of assumptions in our model:\n\n(1) The homography assumes a pinhole camera model with no radial distortion and accurate camera poses. These are common assumptions for 3D vision tasks.\n\n(2) The backprojection of DINOv2 features assumes the cloud layer being reconstructed is not occluded. This is generally not a problem for single cloud layers, as the cameras are upwards facing, such that occlusions are minimized. Additionally, while one camera might be partially occluded, the features from other unoccluded cameras will help average out errors.\n\n(3) The sparse transformer only refines in local areas of the estimated cloud from the cloud layer estimation model.\n\n&nbsp;\n\n>Q7. The authors could include experiments or simulations to validate the framework's performance under various environmental conditions\n\nWe have investigated our model's performance across different conditions in our real-world dataset, including different numbers of camera views, cloud coverages, cloud base heights, and cloud geometric thicknesses. These results are shown below, where the radar GT has been used to classify the cloud coverage, mean cloud base height, and mean cloud geometric thickness, over one-hour segments in our real-world dataset.\n\nFrom these results, we highlight that the model’s performance slightly worsens for higher cloud base heights and also geometrically thicker clouds, but not to a significant extent. \n\n| Cloud Coverages &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; Occ &nbsp; &nbsp; | &nbsp; LWC (gm⁻³) &nbsp; | &nbsp; LWP (kgm⁻³) &nbsp; | &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; | &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 0.20 - 0.45 | 0.65 | 0.034 | 0.077 | **149.36** | **287.12** |\n| 0.75 - 0.90 | **0.75** | **0.023** | **0.048** | 230.92 | 304.67 |\n\n&nbsp;\n\n| Mean CBH (m) &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; Occ &nbsp; &nbsp; | &nbsp; LWC (gm⁻³) &nbsp; | &nbsp; LWP (kgm⁻³) &nbsp; | &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; | &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 800 - 1350 | **0.74** | **0.027** | **0.055** | 198.63 | **281.29** |\n| 1350 - 1650 | 0.63 | 0.033 | 0.077 | **173.36** | 321.73 |\n\n&nbsp;\n\n| Mean Cloud Thickness (m) &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; Occ &nbsp; &nbsp; | &nbsp; LWC (gm⁻³) &nbsp; | &nbsp; LWP (kgm⁻³) &nbsp; | &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; | &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 150 - 275 | 0.69 | **0.025** | **0.050** | 194.84 | **269.58** |\n| 275 - 500 | **0.71** | 0.033 | 0.076 | **184.09** | 323.12 |\n\n&nbsp;\n\n>Q8. How does Cloud4D handle partially obscured views or low visibility conditions?\n\n**Partially obscured views.** To evaluate how Cloud4D might be affected by partially obscured views,  we have included results below on how Cloud4D is affected by uniformly dropping random camera pairs during evaluation. The results show that predictions for cloud base heights and cloud top heights worsen as views are dropped, but that liquid water content and liquid water path predictions are less affected. The decreasing CBH and CTH accuracy with missing views matches established results in 3D vision, where increasing the number of views improves accuracy in 3D shape estimation.\n\n| Number of Dropped Views &nbsp; &nbsp; &nbsp; | &nbsp; &nbsp; Occ &nbsp; &nbsp; | &nbsp; LWC (gm⁻³) &nbsp; | &nbsp; LWP (kgm⁻³) &nbsp; | &nbsp; &nbsp; CBH (m) &nbsp; &nbsp; | &nbsp; &nbsp; CTH (m) &nbsp; &nbsp; |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| 0 | 0.70 | **0.029** | **0.063** | **189.58** | **295.77** |\n| 2 | **0.76** | 0.038 | 0.070 | 316.28 | 371.71 |\n| 4 | 0.68 | 0.037 | 0.072 | 325.03 | 383.16 |\n\n**Low visibility conditions.** Our current implementation focuses on cases where no rain or fog is obscuring the views. Mitigating the impact of these conditions is left to future work (see Q9 for suggestions).\n\n&nbsp;\n\n>Q9. Are there any preprocessing steps or modifications to the framework that can mitigate the impact of these conditions?\n\nYes, here are some suggestions:\n\n(1) The model could directly learn to be robust to low-visibility conditions by including such cases in the training data. The simulation of rain or fog could be implemented using existing techniques in Blender. \n\n(2) The impact of low visibility conditions could be mitigated through preprocessing by using weather data to filter out data with rain or fog.\n\n&nbsp;\n\n[1] Large-eddy simulation of the diurnal cycle of shallow cumulus convection over land, Brown et al. (Royal Meteorological Society 2002)\n\n[2] An efficient parameterization for surface shortwave 3d radiative effects in large-eddy simulations of shallow cumulus clouds, Tijhuis et al. (Advances in Modeling Earth Systems 2023)\n\n[3] Controls on precipitation and cloudiness in simulations of trade-wind cumulus as observed during rico, vanZanten et al. (Advances in Modeling Earth Systems 2011)\n\n[4] A large eddy simulation intercomparison study of shallow cumulus convection, Siebesma et al. (Atmospheric Sciences 2003)"}]

Author rebuttal:

Meta-review: Summary: Cloud4D reconstructs 4D cumulus cloud fields + winds from six synchronized ground cameras.  The resolutiion: 25 m spatial and 5 s temporal resolution is better than alternative approaches.

Strengths:
Clear, novel geometry-aware pipeline (DINOv2+LoftUp+homography gives sparse 3D refinement)
Order-of-magnitude space-time resolution gain over satellites.
Good real world demo.

Weaknesses:
Real-world validation is narrow (single primary site, daytime, mostly shallow single-layer cumulus).
Limited baselines (VIP-CT adapted; NeRF-style methods fail but comparisons remain thin).

Decision and justification:
Substantial, original capability: high-resolution, physically meaningful 4D cloud reconstructions from commodity cameras.
Method is technically sound.
Decision: Accept (spotlight)