Title: Cloud4D: Estimating Cloud Properties at a High Spatial and Temporal Resolution
Venue: NeurIPS 2025
Accepted: yes
Keywords: AI for Science

Computer Vision

Climate Change

3D Reconstruction
Abstract: There has been great progress in improving numerical weather prediction and climate models using machine learning. However, most global models act at a kilometer-scale, making it challenging to model individual clouds and factors such as extreme precipitation, wind gusts, turbulence, and surface irradiance. Therefore, there is a need to move towards higher-resolution models, which in turn require high-resolution real-world observations that current instruments struggle to obtain. We present Cloud4D, the first learning-based framework that reconstructs a physically consistent, four–dimensional cloud state using only synchronized ground‐based cameras. Leveraging a homography-guided 2D‐to‐3D transformer, Cloud4D infers the full 3D distribution of liquid water content at 25 m spatial and 5 s temporal resolution. By tracking the 3D liquid water content retrievals over time, Cloud4D additionally estimates horizontal wind vectors. Across a two-month deployment comprising six skyward cameras, our system delivers an order-of-magnitude improvement in space-time resolution relative to state-of-the-art satellite measurements, while retaining single-digit relative error ($<10\\%$) against collocated radar measurements.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Cloud4D, a novel learning-based framework designed to estimate cloud properties at high spatial and temporal resolutions using synchronized ground-based cameras. The framework leverages a homography-guided 2D-to-3D transformer architecture to predict the full spatial distribution of liquid-water content and horizontal wind vectors. Through a two-month real-world deployment with six cameras, Cloud4D demonstrates a significant improvement in space-time resolution compared to state-of-the-art satellite measurements while maintaining single-digit relative error against collocated radar measurements. In summary, the paper presents a groundbreaking approach to cloud property estimation that addresses the limitations of traditional methods and offers a scalable solution for high-resolution cloud observations\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n﻿\n- The paper is technically sound and presents a novel framework (Cloud4D) that effectively addresses the challenge of estimating cloud properties at high spatial and temporal resolutions. The proposed method demonstrates significant improvements over existing satellite measurements, achieving an order-of-magnitude enhancement in temporal resolution while maintaining high accuracy (less than 10% relative error) when compared to radar measurements.\n- The authors provide comprehensive experimental validation, including comparisons with radar, satellite, and wind profiler data, which strengthens the credibility of their claims. The use of a two-month real-world deployment with six cameras further supports the robustness of the framework.\n\n### Weaknesses:\n﻿\n- While the experimental results are promising, the paper does not provide error bars or confidence intervals for the reported metrics.\n﻿\n- Some sections could benefit from additional details. For example, the implementation details of the wind retrieval method could be elaborated further to provide a clearer understanding of how the horizontal wind vectors are estimated from the reconstructed cloud fields.\n﻿\n- While the combination of techniques presented in the paper is novel, some components of the framework, such as the use of vision foundation models (e.g., DINOv2) and sparse transformers, have been explored in other computer vision applications.\n\nQUESTIONS:\n1. The authors mention that Cloud4D is primarily trained on cumulus clouds and focuses on single-layer cloud properties. While this is well-motivated for ground-based cameras, the limitation to specific cloud types and single-layer clouds restricts the framework's applicability. Could the authors elaborate on the challenges and potential strategies for extending Cloud4D to other cloud types (e.g., stratocumulus, cirrus) and multi-layer clouds?  How might the framework handle occlusions and distinguish between different cloud layers? \n﻿\n2. The authors should consider providing error bars or other statistical measures for the reported metrics. Additionally, explaining the factors contributing to the variability (e.g., train/test splits, initialization) would strengthen the statistical significance of the findings.\n﻿\n3. The paper compares Cloud4D with radar, satellite, and other instruments, but the comparisons with existing machine learning-based cloud retrieval methods are somewhat limited.How does the explicit modeling of camera geometry in Cloud4D contribute to its performance compared to the implicit learning approaches used?\n﻿\n4. The paper does not include a detailed theoretical analysis of the proposed framework, such as the assumptions made in the homography-guided 2D-to-3D transformation and the limitations of the sparse-voxel refinement approach.\n﻿\n5. The authors could include experiments or simulations to validate the framework's performance under various environmental conditions. How does Cloud4D handle partially obscured views or low visibility conditions? Are there any preprocessing steps or modifications to the framework that can mitigate the impact of these conditions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a scheme for 4D cloud estimation (3D cloud with time). In the first (course)-level, they leverage homography and in the second-level, they trained a model for more fine-grained cloud prediction. The authors use simulation for training and collect their own real-world datasets for validation. Previous work either uses implicit geometry or focus solely on cloud base heights, and this is the first work that scale to real-world scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n1. This paper is well-written and the idea is brought forward pretty neat and clear. \n2. This is a very interesting cloud estimation problem and the authors move beyond 3D geometry (like CBH or CTH) and include LWC, LWP. This allows for precise estimation of the Earth atmosphere at a very fine-grained estimation.\n3. The idea of validation from a bird-eye view using Sentinel-2 and MODIS is interesting and helpful to validate this scheme and the results are pretty astonishing to me - Cloud prediction and shadow estimation have been a long under-explored problem and this piece of work can offer as a new labeling technique. \n\nWeakness:\n1. I think the comparison is not thorough and it would be helpful to include methods from other domains. But this does not compromise the quality and significance of this work\n\nQUESTIONS:\n**Dataset Generation**:\n1. Could the authors elaborate on the synthetic dataset generation process?\n2. How was the data split between training and test sets?\n3. Please clarify the \"three scenarios\" mentioned in Line 180.\n4. Were different lighting conditions (such as sun azimuth) considered, and was lighting augmentation performed?\n5. How accurately does the LES simulation reflect real-world conditions, and is this a standard methodology?\n6. What types of real-world clouds cannot be simulated using cloud simulation?\n7. It would be beneficial to include visualizations comparing synthetic and real-world data.\n\n**Training and Fine-tuning**:\n1. Has consideration been given to using Sentinel-2 imagery for model fine-tuning?\n2. Given the two-month real-world dataset and Sentinel-2's average 5-day revisiting time, it would be valuable to explore whether real-world fine-tuning improves performance.\n3. Does scaling up the training dataset size help?\n\n**Quantitative Evaluation**:\n1. The paper lacks comprehensive quantitative evaluation of real-world data performance.\n\n**Alternative Approaches**:\n1. Have alternative methods such as NERF or DUST3R been considered for 3D estimation?\n2. How do these methods compare to the proposed approach?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Cloud4D: a model which predicts cumulus cloud presence, liquid water content, liquid water path, cloud base height and cloud top height from ground based cameras at fine temporal and spatial resolutions. The motivation for this work is to tackle limitations in how such cloud systems are resolved in existing numerical / ML climate models or satellite-based detection models. The Cloud4D modelling framework uses homography and convolutional layers to process image features extracted from ground based camera images by DinoV2 into 2D measures of cloud properties. These cloud properties are then used as initial features in a 3D refinement network that estimates the liquid water content in a voxel. Wind speed is estimated by tracking the movement of reconstructed clouds. The model was trained using a synthetic dataset produced using large eddy simulations and evaluated on a real world dataset captured by cameras and radar over a two month period. The model was shown capture features in ground based observations for liquid water content and path relative to ERA5 reanalysis data and to align with satellite observations of clouds.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper picks off a challenging topic in trying to model cumulus clouds and fine spatial and temporal resolutions. It presents a relatively complex workflow to address this challenge with a model that converts several ground based images, captured at a point in time, into a 3D dataset of several cumulus cloud properties. In the main, this workflow was well presented and easy to follow. The authors also collect a detailed, but localised, ground truth dataset consisting of two months of real world camera observations to evaluate the model. This dataset is a valuable contribution and allows for evaluating the model in considerable detail. However, the localised nature of this dataset left questions as to how generalisable this approach is. From what I could see in the paper or supplementary material, the geographic location or season of this dataset was not presented. \n\nIt was good that the authors mentioned limitations of using ground based cameras due to weather-based occlusion. However, there was scope to discuss other limits of using ground based cameras including nighttime captures and scalability and what this means for broad applicability of the Cloud4D approach. \n\nThe discussion of the results was quite superficial in places. For example, a figure comparing Cloud4D wind speeds, ERA5 wind speeds and ground truth wind speeds was presented and not discussed. The main results table presents average errors across time and position of clouds / water content / wind speed within the viewed area. It would be good to comment on how much variability there was in model error and the characteristics of the error.\n\nQUESTIONS:\nThe comparison with the VIP-CT method could have been described and explained in more detail. This setup for using VIP-CT was not introduced in the methods section and, given that VIP-CT is designed for satellite images, how effective is this as a comparison model when applied to ground based camera systems? \n\nIt appears as though this system in part relies upon ground based camera systems. Are these just radar images does the model require optical images too? If the model requires optical images, does this mean the system is only capable of operating during daytime? What impact does this have on being able to fully capture the dynamics of cloud systems and to utilise model outputs in downstream tasks? \n\nA related question is how scalable this method is if it relies on ground camera systems? I assume that means it cannot realistically have global coverage? What implications does that have for using the outputs of the Cloud4D model? I would be interested in the authors thoughts and comments on the benefits of investing efforts in furthering ground based systems as opposed to improving satellite-derived monitoring? Is there a middle ground where select ground based systems are used to generate training / evaluation data to refine satellite-based models?\n\nLIMITATIONS:\nThere is a limitations section. However, see the comments in the questions section above regarding discussion of some limitations of this work. My rating now sits between borderline accept and accept. I would err towards the former as, while I found the method to be novel and the author's have explained its relevance / application, I still believe the validation in the paper to be quite limited.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose Cloud4D, a method to estimate cloud properties, such as liquid water content (LWC), liquid water path (LWP), cloud base height (CBH) and cloud top height (CTH), from multiple ground-based images.  \n\nThe method can be summarized as follows:\n\n * collect N ground-based images of clouds, together with the camera poses and intrinsics;\n * estimate an homography that maps the image space to the cloud plane, in order to estimate the height of the plane the clouds reside on;\n * to estimate cloud properties, DINOv2 is applied to the images, followed by the LoftUp method to reproject the DINOv2 features to the image space. The previously learnt homography projects the features to the cloud plane;\n * the clouds of interest, i.e. cumulus clouds, are characterized by their vertically thin nature, meaning that the main variation in cloud structure happens in the horizontal plane. With this assumption, authors perform the cloud estimation layer as a 2Dto-2D task, using 2D convolutions, which estimate the desired cloud properties;\n * a 3D refinement phase is applied to obtain a 3D estimate from the 2.5D features using a sparse transformer initialized with a basic model of a cloud volume, which is further fused with the projected features derived from DINOv2 and LoftUp, reconstructing 3D profiles for LWC;\n * as a final step, wind motion estimation is provided leveraging the reconstructed 3D LWC cloud profiles and the high temporal resolution (5 s) of the camera. Wind tracking is performed using an off-the-shelf point tracker.  \n\nDue to the lack existing datasets on cloud properties, authors validated the method on simulated and observed data, acquired over a two-month deployment of the ground-based cameras. Results show superiority in estimation of cloud properties, although the compared methods are very different in nature and very scarce.\n\nSTRENGTHS AND WEAKNESSES:\nEstimating cloud properties with adequate spatial and temporal resolution is a very relevant and important task for accurate modeling of weather and climate variables. The paper therefore proposes a very interesting and alternative approach for accurate estimation of 4D cloud properties, clearly outlining the motivation behind design choices and validating the method on simulated and observed datasets.\n\nThe paper is original, very well written and clear. Due to the niche application, available validation datasets are scarce, as well as comparable methods. This leads to the use of ad-hoc simulated datasets and a limited observed dataset. Despite this, the potential of the method is in my opinion demonstrated, since authors seek to evaluate their method against very diverse methods for estimation of cloud properties, namely radar measurement, satellite images, and wind profiles.\n\nThe main strength I see is the careful design of the imaging and processing system for the modelling of cumulus clouds, which are indeed challenging to characterize with other methods, e.g. remote sensing. However, on the other hand, I feel that this assumption also represents the main weakness of the method, since I don't believe the 2.5D representation and 2D-to-2D cloud property estimation would be valid for more thick types of clouds and clouds higher in the atmosphere.\n\nQUESTIONS:\nHere are some questions and suggestions I have for the authors:\n\n * could the authors provide more context about how the system is meant to be deployed? From the paper is unclear to me whether the system is designed to provide monitoring over large areas, or whether it is rather designed to characterize cumulus clouds at a smaller scale, with the aim of informing the evolution of weather patterns. If the deployment intention is the former, I would like to know about the requirements in terms of number of cameras required to cover, say, a 100kmx100km area, and which communication/collection infrastructure would be required;\n * as mentioned in the weaknesses above, and as the authors recognize in the Limitations Section, I don;t see how the proposed method could be generalized to estimate cloud properties of different cloud types, namely thicker clouds higher in the atmosphere. Such clouds would violate the assumptions and design choices of Cloud4D;\n* authors are probably aware, but there are more suited satellite images available than Sentinel-2, e.g. PlanetScope from Planet Labs, which provide a higher temporal revisit time, and often provide multiple takes within minutes of one another. Such imagery could be helpful in validating the cloud profiles, as well as wind tracking module.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Cloud4D, a novel machine learning framework designed to estimate high-resolution, four-dimensional (3D space + time) cloud properties using synchronized ground-based cameras.\nAuthors mention a three-fold Contribution: \n* Cloud4D estimates liquid water content (LWC), cloud base height (CBH), and cloud top height (CTH) at a 25 m spatial and 5 s temporal resolution—a significant improvement over satellite and radar methods.\n* The system uses a transformer model guided by camera homographies to map 2D images into structured 3D cloud layer representations. \n*  A demonstration, on a two-month deployment with six cameras, that Cloud4D delivers an order-of-magnitude improvement in temporal resolution over space-borne products while achieving < 10% relative error against collocated radar retrievals.\n\nThe method to achieve the 3D LWC comprises of two main steps:\n* A 2.5D cloud layer model for estimating cloud layer parameters. \n* A sparse 3D transformer that refines these into 3D predictions. \n\nFinally an Horizontal Wind Estimation is presented. By tracking cloud movement over time in the 3D reconstruction, Cloud4D infers height-resolved horizontal wind vectors.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The proposed methods achieves 25 m spatial and 5 s temporal resolution, an order-of-magnitude better than satellite products.\n* The authors introduce a homography-guided 2D-to-3D transformer pipeline, leveraging DINOv2, LoftUp, and a sparse transformer for efficient and accurate 3D cloud field estimation.\n* Height-resolved horizontal wind profiles are derived by tracking reconstructed 3D cloud fields over time, a capability not available in previous multi-view or satellite-based methods.\n* Demonstration of the method on a two-month deployment with six cameras, showing practical feasibility.\n* Using synthetic data from MicroHH (atmospheric simulation) and Terragen (cloud rendering) lets Cloud4D train and test its models on realistic, fully labeled 3D cloud scenes that aren’t possible to capture in the real world.\n\nMain Weaknesses: \n* The abstract and introduction repeatedly claim <10% relative error against collocated radar measurements, but this number does not appear in any table, figure, or explicit result in the main or supplementary material.  There are no clear, quantitative error tables or plots showing this metric, making it difficult to verify the headline claim.\n* The paper lacks comprehensive quantitative comparisons (e.g., with other methods or baselines) and detailed error analysis across different cloud types, conditions, or time periods.\n*  Generalizability to other sites, camera setups, number of views or cloud regimes is not tested or discussed in depth. Of course, this is an inherent limitation of the method, since deploying many sites and setups requires significant effort.\n* The method relies on radar retrievals for ground truth, which themselves have limitations and may not always be available or accurate, especially for shallow clouds.\n\nConclusion\n* This paper presents an innovative and state-of-the-art approach for 3D cloud reconstruction, leveraging a sophisticated 2D-to-3D transformer architecture and foundation models to achieve unprecedented spatial and temporal resolution. The use of only synthetic data for training the model  (even though it can raise domain adaptation concerns) is particularly interesting and, based on the partial comparisons to radar retrievals shown in Figure 3 and the quantitative results in Table 1, appears promising.\n* The main concern is the lack of direct evidence for the headline \"<10% relative error\" claim. This is repeated in the abstract and introduction, but to my understanding, no table, figure, or quantitative analysis substantiates it. I recommend a borderline reject, but encourage the authors to address this, as the work has significant potential.\n\nQUESTIONS:\n* Important: Justification of 10% Relative Error - The abstract and main text claim \"<10% relative error against collocated radar retrievals,\" but this metric does not appear in any table or figure. Where is this value calculated, and can you provide quantitative evidence or direct references to support this claim? \n\n* Moderate: It seems that the training of Cloud4D is performed exclusively on synthetic data. Isn’t there a need to address the domain gap between synthetic and real-world observations when deploying the model on real camera and radar data?\n\n* Moderate: Did you consider implementing and comparing with other models (e.g., multi-view stereo, which is also referenced in your work)?\n\n* Minor: Ambiguity of \"K\" Symbol: In Section 3.1 and related equations, does the symbol \"K\" refer to the number of height planes sampled, the number of camera intrinsics, or both? Could you clarify its precise meaning and usage in the model pipeline?\n\nLIMITATIONS:\nWhile Cloud4D achieves much higher spatial and temporal resolution than satellites, it requires dense, site-specific camera deployments. In contrast, satellite data, though coarser, can be applied globally and generalized to any location without extra infrastructure. This makes Cloud4D less scalable and less universally applicable than satellite-based approaches.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}