Fourier Amplitude and Correlation Loss: Beyond Using L2 Loss for Skillful Precipitation Nowcasting

Authors: chiu-wai  yan
shi  quan
foo  van
hoan  trinh
dit-yan  yeung
ka-hing  wong
wai-kin  wong
hong  kong observatory
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 07:08:37

Abstract: Deep learning approaches have been widely adopted for precipitation nowcasting in recent years. Previous studies mainly focus on proposing new model architectures to improve pixel-wise metrics. However, they frequently result in blurry predictions which provide limited utility to forecasting operations. In this work, we propose a new Fourier Amplitude and Correlation Loss (FACL) which consists of two novel loss terms: Fourier Amplitude Loss (FAL) and Fourier Correlation Loss (FCL). FAL regularizes the Fourier amplitude of the model prediction and FCL complements the missing phase information. The two loss terms work together to replace the traditional L2 losses such as MSE and weighted MSE for the spatiotemporal prediction problem on signal-based data. Our method is generic, parameter-free and efficient. Extensive experiments using one synthetic dataset and three radar echo datasets demonstrate that our method improves perceptual metrics and meteorology skill scores, with a small trade-off to pixel-wise accuracy and structural similarity. Moreover, to improve the error margin in meteorological skill scores such as Critical Success Index (CSI) and Fractions Skill Score (FSS), we propose and adopt the Regional Histogram Divergence (RHD), a distance metric that considers the patch-wise similarity between signal-based imagery patterns with tolerance to local transforms.

Main paper: 
 introduction: 
 Precipitation nowcasting refers to the task of predicting the rainfall intensity for the next few hours based on meteorological observations from remote sensing instruments such as weather radars, satellites and numerical weather prediction (NWP) models. The development of a precise precipitation nowcast algorithm is crucial to support weather forecasters and public safety, as it could facilitate timely alerts or warnings on severe precipitation and mitigate their impact on the community through early preventive actions. Sharp precipitation nowcast imagery that is perceptually similar to the actual observations (such as radar images) is equally important for weather forecasters to comprehend how the severity of precipitation will evolve in space and time, as well as to diagnose the rapid evolution of the underlying weather systems in real-time forecasting operations.Besides the traditional optical-flow and NWP models, deep learning models have also been widely explored and adopted for precipitation nowcasting in recent years. The research community generally formulates the task as a spatiotemporal prediction problem, where a sequence of input radar or satellite maps is given, and the future sequence needs to be predicted or generated. Although multiple previous attempts proposed solid improvements to the model to grasp the spatiotemporal dynamics, deep learning models can result in blurry predictions in real-life datasets featuring precipitation patterns such as radar echo and satellite imagery. Consequently, they provide limited operational utility [1] in weather forecasts.The blurry prediction in multiple deep learning models is believed to be caused by the use of pixelwise losses such as the Mean Squared Error (MSE), which entangles the probability into model prediction. In other words, the uncertainty of the image transformation leads to obfuscation of the surrounding pixels in the prediction. Nevertheless, solely improving the model capability could not resolve this issue due to the high spatial randomness of the atmospheric dynamics. In order to suppress the ambiguity of the model output, an emerging approach is to utilize generative models such as generative adversarial networks (GANs) and diffusion models. In this paper, we introduce an alternative approach, which is to modify the loss function such that the model focuses on recovering the high-frequency patterns. By utilizing the Fourier domain, we would like to shed light on a deterministic, non-generative method that can sharpen the spatiotemporal predictions with negligible sacrifice to its correctness.To achieve the desired sharpness, we propose the Fourier Amplitude Loss (FAL), a loss term that improves the prediction of high frequencies by regularizing the amplitude component in the Fourier space. Supported by empirical validation, we further propose the Fourier Correlation Loss (FCL), a complementary loss term that provides information on the overall image structure. Furthermore, we have developed a training mechanism that alternates between FAL and FCL based on an increasing probability of employing FAL throughout the training steps. We name this combined loss function the Fourier Amplitude and Correlation Loss (FACL). FACL is computationally efficient, parameterfree, and model-agnostic and it can be directly applied to a wide range of state-of-the-art deep neural networks and even generative models. Extensive experiments show that compared to MSE, FACL results in forecasts that are both more realistic and more skillful (i.e., high performance with respect to several meteorological skill scores). To the best of our knowledge, we are the first to substantially replace the spatial MSE loss with spectral losses without using generative components on the spatiotemporal prediction problem, demonstrating the novelty and significance of our approach.Our main contributions are summarized as follows:• We propose the Fourier Amplitude and Correlation Loss (FACL), which is constituted by sampling between the Fourier Amplitude Loss (FAL) for regularizing the spatial frequency of the predictions to enable clarity and sharpness, and the Fourier Correlation Loss (FCL), a modified loss term that is cohesive with FAL to capture the overall image structure. • Theoretical and empirical studies show that FAL boosts the image sharpness significantly while FCL complements the missing information for accuracy. • We apply FACL to replace the MSE reconstruction loss in generative models. Results show that generative models with FACL perform better with respect to most of the metrics. • We propose the Regional Histogram Divergence (RHD), a quantitative metric to measure the distance between two signal-based imagery patterns with tolerance to deformations. RHD considers both the regional similarity and the visual likeness to the target.2 Related Works 
precipitation nowcasting as a spatiotemporal prediction problem: 
 Previous works generally formulate precipitation nowcasting as a spatiotemporal predictive learning problem. Given a sequence of observed tensors with length t: X 1 , X 2 , ..., X t , the problem is to predict the future k tensors formulated as follows:arg max Xt+1,...,X t+k p(X t+1 , ..., X t+k | X 1 , X 2 , ..., X t )Based on this formulation, numerous variations of convolutional RNN models were proposed to model both spatial and temporal relationships in the data. ConvLSTM [2] first proposed to integrate convolutional layers into LSTM cells, with the recurrence forming an encoder-forecaster architecture.PredRNN [3] replaced the ConvLSTM units with ST-LSTM units and modified the structure such that the hidden states flow in both spatial and temporal dimensions in a zigzag pattern. MIM [4] replaced the forget gate in ST-LSTM with another RNN unit, forming a memory-in-memory structure to learn higher-order non-stationarity. Moreover, advanced modifications such as reversed scheduled sampling [5] , gradient highway [6] , etc. [7, 8] were proposed to further improve the overall performance of the model. With the breakthroughs brought by transformers and self-attention mechanism, space-time transformer-based models such as Rainformer [9] and Earthformer [10] were proposed to model complex and long-range dependencies.On the other hand, CNN models have also been widely explored for the task as a video prediction problem. Inspired by the U-Net structure used in earlier works [11, 12] , SimVP achieves remarkable performance and efficiency by adopting an encoder-translator-decoder structure with mostly convolutional operations. Among the parts, the translator (temporal module) is found to benefit from MetaFormer (an architecture with both token mixer and channel mixer) in subsequent studies [13, 14] . TAU [15] further demonstrated the effectiveness of the structure by adopting depthwise convolution followed by 1 × 1 convolution as the temporal module.Conventional precipitation nowcasting tasks and video prediction tasks evaluate the output mainly with pixel-wise or structural metrics such as Mean Absolute Error (MAE), Mean Squared Error (MSE) and Structural Similarity (SSIM) Index. To better consider the hits and misses of signal-based reflectivity, the Critical Success Index (CSI; equivalent to Intersection over Union, IoU), Fractions Skill Score (FSS) and Heidke Skill Score (HSS) belong to another type of metrics widely used in meteorology. To distinguish these scores from those used in the traditional machine learning literature, we refer to this metric type as skill scores in the remaining sections of the paper.Traditional models can result in blurry predictions at longer lead times, causing difficulty in forecasting operations. To address it, recent works leverage generative models such as GANs and diffusion models to promote realistic forecasts which could bring more insightful observation to forecasting operations. DGMR [1] utilizes a GAN framework with discriminators in both the spatial and temporal dimensions to ensure that the predicted images are sufficiently realistic and cohesive. LDCast [16] uses latent diffusion to generate a diverse set of outputs for ensemble forecasting. Meanwhile, the literature in video generation strives to generate realistic output frames with generative models. PreDiff [17] introduces a knowledge alignment mechanism with domain-specific constraints while adopting a latent diffuser for quality forecasts. DiffCast [18] appends a diffusion component as an auxiliary module to improve the realisticity of the forecasts. It is worth mentioning that the literature in video generation [1, 19, 20, 21, 22] also exhibits potential in generating high-quality nowcastings despite not specifically being designed to handle precipitation. Unlike works in video prediction, instead of evaluating the output quality with pixel-wise similarity, perceptual metrics such as LPIPS [23] and Fréchet Video Distance (FVD) [24] are predominantly used.These works usually formulate the task as an unsupervised or semi-supervised learning problem with the results being non-deterministic based on a random prior, enabling the possibility of ensemble prediction. However, studying each prediction individually is less reliable as the prediction is unexplainably affected by the random prior. Furthermore, the inference efficiency of the diffusion model is poor due to the iterative nature of the reverse diffusion sampling process. Concerning the drawbacks of generative models, our method is proposed to be efficient, deterministic, and accurate at both the pixel and perceptual levels, bridging the advantages of both probabilistic video prediction and non-deterministic video generation.Spectral analysis in the Fourier space is a common practice for DNNs to study the features in terms of frequency. Rahaman et al. [25] proposed a property known as the spectral bias, which causes DNN models to be biased towards low-frequency functions. A follow-up study [26] theoretically showed that DNN models have a much slower convergence rate toward high-frequency components. Such observations motivate subsequent works to apply Fourier-based loss terms extensively in tasks such as super-resolution (SR) where fine details are crucial.Despite the existence of works that apply Fourier transform amid the model feed-forward pipeline [27, 28, 29, 30, 31] , here we focus on works that utilize spectral transform in the loss function or as a regularization term. Inspired by the JPEG compression mechanism, the Frequency Domain Perceptual Loss [32] compares the Discrete Cosine Transform (DCT) of the model output in 8×8 non-overlapping patches:L(y, ŷ) = c ⊙ ∥DCT(y) -DCT(ŷ)∥ 2 2, where c is a vector of constants computed from the quantization table and training set. The Focal Frequency Loss [33] compares the element-wise weighted Fast Fourier Transform (FFT) output:L FFL = 1 M N M -1 u=0 N -1 v=0 w(u, v)|FFT(y) u,v -FFT(ŷ) u,v | 2, where w(•, •) is a dynamic weight matrix and |•| refers to the absolute operator on complex numbers. Moreover, the Fourier Space Loss [34] decomposes the Fourier output (in complex) into amplitude and phase and measures their difference separately as a GAN loss component.Although these losses were proposed specifically for the SR task, we find the problem setting similar to spatiotemporal forecasting in terms of the requirement for high-frequency fine details and the involvement of a ground-truth label. While taking advantage of the Fourier space as a spectral analysis is intuitive, choosing the proper distance metrics and additional weighting is tricky. This motivates us to propose a new loss function with consideration of the spectral property on the spatiotemporal forecasting problem. 
our methods: 
 In this section, we start by arguing why a naive implementation of the Fourier loss does not benefit the model compared with the MSE loss in the image space. Then, we will discuss the motivation and details of our proposed FACL.An image X can be interpreted as a 2D matrix with the transformed Fourier series, F . The orthonormalized Discrete Fourier Transform (DFT) output and its corresponding inverse Discrete Fourier Transform are formulated as:EQUATIONwhere M and N are the height and width, respectively, of the image X.To constrain model convergence via the spatial frequency components of its prediction, one naive design is to regularize the L 2 norm of the displacement vector between the ground truth and prediction in the Fourier space apart from the image space. Parseval's Theorem shows that such design is linearly proportional to the spatial MSE loss, and the detailed proof can be found in Appendix B.Since this straightforward regularization does not differ from the MSE loss in the image space, the common adaptations from previous works are either to apply weighting on different frequencies or to decompose the Fourier features into amplitude |F | and phase θ F with the following definitions:EQUATIONwhere F real and F imag are the real and imaginary parts, respectively, of the complex Fourier vector F .As the spectral bias indicates the lack of attention to the high-frequency components, we encourage the model to consider high-frequency patterns by applying a loss on the amplitude of each frequency band. Similar to previous works, we first apply DFT to obtain the spectral information. Using Equation (3), we extract only the Fourier amplitudes (|F |) in the Fourier space and compare them inEQUATIONwhere F is the DFT output of X as formulated in Eq. (2) . Note that the formulation is subtly different from minimizing the L 2 norm of the displacement vector that prediction deviates from ground truth in the Fourier domain. The new formulation based on the Fourier amplitude of the images only is invariant to global translation. This reduces the spatial constraint induced by MAE and MSE losses.A detailed analysis can be found in Appendix D.Despite retaining the high-frequencies by dropping the Fourier phase, FAL alone is insufficient to reconstruct the image. As X → |F | is a many-to-one mapping, there exist multiple X to have the same Fourier amplitude matrix. Thus, simply minimizing Eq. ( 4) can likely converge to an undesirable critical point. A high-level interpretation is that only image sharpness is retained by this loss while the information regarding the actual shape and position is lost with the Fourier phase discarded. Hence, on top of FAL, we require another loss term to compensate for the missing information, leading to our upcoming proposal of the FCL term. An alternative perspective via the mathematical formulation can be found in Appendix C.To remedy the missing information resulting from FAL, there are several approaches to take the image structure into account. A straightforward way is minimizing the difference of the Fourier phase between the prediction and the label, but it fails as θ F obtained under DFT is discontinuous. Another approach is to compute the cosine distance in the Fourier domain without extracting θ F directly. However, our preliminary experiments reveal that such formulation is unstable in reconstructing the image structure. Ultimately, we propose to implement the correlation between the generated output and ground truth in the Fourier domain and adopt it as the Fourier Correlation Loss (FCL) in our proposed loss:EQUATION)where here is a shorthand for the summation over all elements of the Fourier features and * denotes the complex conjugate of the vector. FCL plays a significant role during training as it is responsible for learning the proper image structure while FAL can be treated as a regularization to promote the high-frequency components that FCL fails to capture.The formulation of FCL has a similar format to the Fourier Ring Correlation (FRC) and Fourier Shell Correlation (FSC) widely used in image restoration and super-resolution of cryo-electron microscopy [35, 36, 37, 38, 39, 40] . However, both FRC and FSC pre-define a specific region of interest (either a ring or a shell) on the Fourier features. In contrast, we extend the region of interest to the entire map, considering the global spectral bands with all frequencies. To ensure the score is real and commutative, we take the average of F F * and F F * in the numerator. The denominator performs normalization such that FCL only focuses on the image structure in the global view rather than the absolute brightness. Unlike FRC (without 1-), FCL spans the range [0, 2], where larger values refer to a negative correlation and smaller values refer to a positive correlation. Further analysis of FCL from the gradient aspect can be found in Appendix E.While it is straightforward to apply the overall loss function as a linear combination of FAL and FCL, we find it tricky to determine the weighting of the components in our preliminary studies. Instead, we offer a more controllable solution -to alternate FAL and FCL as shown below:EQUATIONwhere p is sampled randomly and uniformly in [0, 1] and P (t) is a pre-defined threshold decreasing during the training process as shown in Figure 1 . P (t) always decreases from 1 to 0 such that the model is first trained with 100% FCL that takes image structure into account, and then the models are more frequently trained with FAL which improves the image sharpness.Since FCL loses information on the overall brightness, the model could not achieve proper brightness at the early stage where FCL dominates the learning objective. To address it, we append a sigmoid function in the output layer of the model. This constrains the model output in the range [0, 1] to prevent the model from converging to a sub-optimal state with an undesirable range of output values.Overall, the following modifications are applied to the models:• Training loss function of the models involving FAL and FCL is formulated in Eq. ( 6).• A sigmoid layer is appended to the end of the model. For RNN models, the sigmoid function is applied before the output of the last RNN stack. • To coordinate with the decreasing threshold, the cosine annealing learning rate scheduler is used rather than the conventional reduce-on-plateau scheduler. One of the metrics that consider both the previous two factors is the Fractional Skill Score (FSS), which is widely used in meteorology. After splitting the image into N x × N y smaller patches, where N x and N y control the shift of precipitation events we tolerate, we obtain the FSS score as follows:EQUATIONwhere F i,j and O i,j refer to the fraction of predicted positives and fraction of observed positives, respectively, of the patch in the i-th row and j-th column. Based on this formulation, the intensities are free to reposition within the patch window, granting tolerance to translation and deformation. Nevertheless, one drawback of FSS is that the pixel range is only categorized into two classes: positives and negatives. For a threshold of 0.5, a pixel value of 0 is treated the same as a pixel value of 0.49, resulting in a huge error when viewing the per-patch precision. This means that the choice of threshold induces a bias in evaluating the forecasting performance of models.To improve the representation, we propose the Regional Histogram Divergence (RHD), a variation of FSS that exhibits smaller errors within a class. Instead of categorizing the pixel values into 'hits' and 'misses', we divide the values into n bins and count the frequency of each bin, obtaining a histogram for each patch. Next, we compare the average Kullback-Leibler (KL) divergence on the histograms. Mathematically, the RHD between two sets of bins can be expressed as:EQUATION, where F ′ i,j and O ′ i,j correspond to the predicted and observed discrete probability distributions, respectively, among the set of bins X of the patch in the i-th row and j-th column.Different from FSS where the proportions of positives are subtracted directly, RHD instead compares the distributional difference in the context of the histograms. This not only increases the precision of each class/bin, but also heavily penalizes blurs since blurring forms a Gaussian-like distribution in the histograms while sharp intensities should have an 'M-shape' bimodal distribution. If the patch-wise distribution of the two images is identical, the corresponding RHD is 0. The larger the RHD is, the more different the two sets of patches behave. Furthermore, RHD is formulated to be a mean KL divergence so it is always non-negative.For simplicity and consistency, we choose the number of bins to be 10, divided uniformly within the range [0, 1] for all datasets in our experiments. In real-life applications, non-uniform division can be applied for data in non-linear scales such as radar echo (in dBZ) to highlight specific ranges of values. When we compute the histograms, as 0 dominates in the imagery, we apply a threshold ϵ = 10 -5 to exclude all small intensities. 
experimental settings: 
 We evaluate the performance of our proposed method on a synthetic dataset and three radar echo datasets, namely, Stochastic Moving-MNIST, SEVIR [41] , MeteoNet [42] and HKO-7 [43] . A more detailed description for each dataset can be found in Appendix A. To show that our method is effective and generic, we selected ConvLSTM [2] and PredRNN [3] (reported in Appendix I), two RNN-based models with different recurrence paths; SimVP [44] , a CNN-based model and Earthformer [10] , a transformer-based model. We trained the models with two variants: conventional MSE and FACL (as formulated in Eq. ( 6)). To compare with generative models as references, we also report the results of LDCast [16] (latent diffusion) and MCVD [21] (denoising diffusion) for all datasets. For Stochastic Moving-MNIST, we report two more models, i.e., PreDiff [17] (latent diffusion) and STRPM [45] (GAN-based). Appendix M reports the detailed setup and hyper-parameters.In the upcoming sections, we will first present the setup and experimental results on the Stochastic Moving-MNIST dataset. After that, we will test the models with three real-world radar echo datasets. Extra studies on our methods are reported in the Appendix. Specifically, we report the ablation study of FAL, FCL and α in Appendix F, the running time of FACL in Appendix G, experiments on additional datasets in Appendix H, comparison with other potential loss functions in Appendix J, the performance when applying FACL to generative models in Appendix K, and characteristic analysis of RHD in Appendix L. To demonstrate the advantages of our method against counterparts for precipitation nowcasting, video prediction and video generation, we evaluate the models with a union of metrics from the areas. Specifically, we report the MAE and SSIM to show the pixel-wise and structural accuracy; LPIPS and FVD to show the deep perceptual similarity to ground truth; FSS and RHD to measure the similarity of the intensity distribution in different regions. For radar echo datasets, we further include the CSI and pooled CSI to reveal the models' capability of identifying potential extreme weather. Such a combination of metrics is believed to facilitate a comprehensive understanding of the pros and cons of the current state-of-the-art in precipitation nowcasting.The Moving-MNIST dataset has been a common benchmark to evaluate how well a model could predict motion in spatial preservation and temporal extrapolation. However, the nature of the Moving-MNIST is highly deterministic, which does not resemble the chaotic nature of the atmospheric system. Previous adaptations attempted to simulate the physical dynamics by introducing a set of complex motions such as rotation and scaling [10] or by applying an external force on collision [46] . We argue that the fundamental reason causing the blur in precipitation nowcasting is the intrinsic stochasticity of the motion caused by external factors unseen in the weather dataset, such as orographic effects, vertical wind shear, interaction with other weather systems, etc. Trained with such stochasticity, regular models with pixel-wise loss could consistently fail to provide quality prediction in the future lead time.To verify our claim, we introduce a simplistic modification to the Moving-MNIST dataset. The standard Moving-MNIST dataset contains handwritten digits sampled from the MNIST dataset moving and bouncing with a constant velocity (u 0 , v 0 ) on the 64 × 64 image plane. To introduce stochasticity, we perturb the velocity with a random Gaussian noise ϵ at each time step. Details of the perturbation are shown in Appendix A. In the upcoming sections, we dub this dataset Stochastic Moving-MNIST and apply the experimental setting to this synthetic dataset. The performance of combinations of different losses and models can be found in Table 1 and qualitative visualizations of the corresponding methods are shown in Figure 2 and Appendix N. Note that the Stochastic Moving-MNIST is used in both training and evaluation to ensure that the models are well exposed to motion randomness.In Table 1 , our modification drastically improves the sharpness and realisticity for all tested models, as reflected by the vast reduction in LPIPS and RHD. In particular, FACL reduces up to 57% of LPIPS and 71% of RHD for the ConvLSTM model. The pixel-wise and structural metrics between the two losses are comparable. On the other hand, generative models result in much poorer MAE and SSIM, with skill scores like FSS still being worse than most of the baseline models. In Figure 2 , we can observe that the model trained with MSE cannot reconstruct a clear spatial pattern, especially in the subsequent frames, while the model trained with FACL yields much sharper and higher quality outputs. Consequently, we can conclude that in this setting with the synthetic Stochastic Moving-MNIST, FACL demonstrates a significant improvement in perceptual metrics and skill scores, with the quality on par with that of generative models.In this section, we extend the previous setup to general radar-based datasets. Apart from the distance metrics used in the last section, we further report the CSI with different pooling sizes. Following the previous works [43, 10] , we measure multiple CSI scores with different thresholds ({16, 74, 133, 160, 181, 219} for SEVIR, {12, 18, 24, 32} for MeteoNet and {84, 117, 140, 158, 185} for HKO-7).The visualizations can be found in Figure 3 and more in Appendix N.The results of Table 2 are similar to the observations in Table 1 . Compared with the MSE baselines, FACL always improves the perceptual and skill scores. For sharp forecasts, the pooled CSI increases with the pooling size while for blurry forecasts, CSI shows no apparent difference based on pooling size. For some models, we observe a tiny decay in pixel-wise and structural metrics. For example, the Earthformer model trained with FACL on SEVIR has a 6.4% increase in MAE, which is believed to be a natural trade-off since pixel-wise metrics have no tolerance for spatial transformation. Despite poorer pixel-wise performance, the perceptual metrics and skill scores always improve, as further illustrated by Figure 3 that only FACL predicts fine-grained extreme values. Regarding those generative models, although they could perform the best in deep perceptual scores like LPIPS and FVD, we still observe that they usually result in poorer skill scores. Moreover, it is noteworthy that FACL does not add any new parameters to the model. The change in the metrics solely indicates that FACL induces a shift of focus from pixel-wise accuracy to image quality and prediction skillfulness.  
conclusion: 
 In this paper, we proposed the Fourier Amplitude and Correlation Loss (FACL). The two loss terms, Amplitude Loss (FAL) and Fourier Correlation Loss (FCL), encourage the model to focus on the Fourier frequencies and image structure correspondingly. Besides, we proposed a new metric, Regional Histogram Divergence (RHD), to measure the patch-wise similarity between two spatiotemporal patterns. We widely tested our methods on a synthetic dataset and three more real-life radar echo datasets, measured by metrics considering accuracy, realisticity and skillfulness. Extensive experiments reflect that our method yields sharper, more realistic and skillful forecasts with limited degradation in pixel-wise similarity.Despite the remarkable performance of the FACL loss, our methods still have room for improvement. First, we assumed the data to be monotonic radar echo, which might not generalize well to multimodal datasets featured in medium-range forecasts. Besides, our loss provides no regularization on temporal consistency, which may lead to the misalignment of temporal features between frames. The solution to these issues, however, will be open for future work. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Hyper-parameters and data processing are provided in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Most experiments are performed only once due to the high computational requirement of spatiotemporal prediction problems.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: The submitted work has no apparent negative societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The introduced dataset, Stochastic Moving-MNIST, is based on the existing dataset MNIST and the dataloader can be found in the code. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Stochastic Moving-MNIST. Based on the same method generating vanilla Moving-MNIST, we further update the velocity of the digits at time t:EQUATIONEach unit corresponds to a pixel in the 64 × 64 image. Note that the expected trajectory is unchanged under this modification. However, the biased trajectory is exposed to the model as a stochastic factor influencing model training. Due to such behavior, models trained with the MSE loss are expected to exhibit a motion blur pattern along the direction of motion.To forge the chaotic nature of the Earth system, N-body-MNIST [10] was proposed to study the effectiveness of deep learning models. Rather than linear translation as in the conventional Moving-MNIST, digits in N-body-MNIST follow the N-body motion pattern, exerting an attractive force between digits and causing each other to circulate. Following the default setting of the original paper, the frame size is set to be 64 × 64 with N = 3. We use the same training, validation and test sets which contain 20000, 1000 and 1000 sequences respectively provided by the official repository.SEVIR. The SEVIR dataset [41] is a spatiotemporally aligned dataset containing over 10,000 weather events in a 384km × 384km region in the US spanning a period of four hours from 2017 to 2019. Among the five channels provided, we extract the NEXRAD Vertically Integrated Liquid (VIL) data product for precipitation nowcasting. Following previous works [10, 47] , we predict the future VIL up to 60 minutes (12 frames) from 65 minutes of input frames (13 frames). We sample the test set from June 2019 to December 2019, leaving the remaining as the training set. In this section, we prove that the L 2 distance between ground truth and prediction in both the Fourier domain and image domain are equivalent from both the forward and gradient aspects.B.1 Showing that L 2 ( F , F ) = L 2 ( X, X)Parseval's Theorem (or the general one: Plancherel Theorem) describes the unitarity of the Fourier transform under proper normalization. Without normalization, we have the following relationship:N -1 k=0 |X k | 2 = 1 N N -1 k=0 |F k | 2 .This refers to the 1D case where F is the Fourier transformed output of X, and N is the vector length of both F and X. In the 2D case with F orthonormalized in Eq. ( 2), we have instead|X| 2 = |F | 2 ,where is used as a shorthand summing every element in the following 2D matrix. When we apply the L 2 loss to two orthonormalized Fourier matrices, we obtainEQUATIONdue to the linearity of the Fourier transform and the use of Parseval's Theorem.B.2 Showing that ∂L2(F, F ) ∂ Xkl = ∂L2(X, X) ∂ XklFrom Eq. ( 2), we continue and derive the gradient of the Fourier transform output F with respect to image X:∂F pq ∂X kl = 1 √ M N e -i2π( kp M + lq N )For every complex vector, the multiplication of itself and its conjugate is equal to the square of its amplitude, that isF F * = |F | 2 . Thus, ∂ ∂ Xkl |Fpq -F * pq | 2 = ∂ ∂ Xkl [|Fpq| 2 -F * pq Fpq -Fpq F * pq + Fpq F * pq ] = 1 √ M N [-F * pq e -i2π( kp M + lq N ) -Fpqe i2π( kp M + lq N ) + F * pq e -i2π( kp M + lq N ) + Fpqe i2π( kp M + lq N ) ].With the inverse Fourier transform defined in Eq. ( 2) and the assumption that X is always real, we can obtain:∂L2(F, F ) ∂ Xkl = 1 M N M -1 p=0 N -1 q=0 ∂ ∂ Xkl |Fpq -F * pq | 2 = - 1 ( √ M N ) 3 M -1 p=0 N -1 q=0 [F * pq e -i2π( kp M + lq N ) + Fpqe i2π( kp M + lq N ) -F * pq e -i2π( kp M + lq N ) -Fpqe i2π( kp M + lq N ) ] = - 1 M N [X * kl + X kl -X * kl -Xkl ] = - 2 M N [X kl -Xkl ] ∂L2(F, F ) ∂ Xkl = ∂L2(X, X) ∂ XklConsequently, the gradients of L 2 (F, F ) and L 2 (X, X) with respect to Xkl are equivalent. This result indicates that implementing L 2 in the Fourier domain without any weighting as a loss function does not affect the model performance.In Section 3.2, we claim that FAL works as a regularizer to maintain the frequency amplitude in the Fourier domain, but not the full loss function. In this section, we study the reason behind this statement from the perspective of gradient feedback. Before that, we start with deriving the derivative of |F | with respect to X kl :|F pq | 2 = F pq F * pq 2 |F pq | ∂|F pq | ∂X kl = F pq ∂F * pq ∂X kl + ∂F pq ∂X kl F * pq ∂|F pq | ∂X kl = 1 2 √ M N [e i(θpq+α pq,kl ) + e -i(θpq+α pq,kl ) ] = 1 √ M N cos(θ pq + α pq,kl ),where F pq = |F pq |e iθpq and α pq,kl = 2π( kp M + lq N ). From Eq. ( 4), we further derive its derivative with respect to Xkl and get:EQUATIONwhere |F pq | and | Fpq | are the Fourier amplitudes of the ground truth and prediction corresponding to the frequency (p, q) while θpq is the Fourier phase of prediction corresponding to the frequency (p, q).From Eq. ( 10), we note that θ pq , which corresponds to the position or the image structure of the ground truth frequencies (object) in the image space, is missing. In other words, the model never gets its parameters updated based on the phase of the ground truth. As a result, FAL only encourages the model to predict what has the same amplitude distribution in the Fourier domain, without considering the image structure. This theoretically shows the infeasibility of reconstructing the ground truth based on FAL alone, motivating us to adopt a second loss term to maintain the image structure.To effectively make use of FAL as a regularizer, we have to ensure that the model has sufficient time to learn the general image structure (the low-frequency pattern) such that FAL could provide better guidance on the remaining frequency components by exposing it more to FCL in the beginning of training process. In contrast, if the model cannot learn the low-frequency components before FAL becomes the dominant learning objective, the model will likely converge to a trivial solution. This claim is also empirically verified by our ablation study experiments where using FAL alone results in poor performance as shown in Appendix. F.This section discerns FAL and a naive L 2 loss in the Fourier space. By definition, the major difference between the two is whether the complex pattern or the amplitude is used in the comparison. The FAL loss term can be expanded as follows:FAL(X, X) = 1 M N M -1 p=0 N -1 q=0 (|F |pq -| F |pq) 2 = 1 M N M -1 p=0 N -1 q=0 (|F | 2 pq + |F | 2 pq -2|F |pq| F |pq) = 1 M N M -1 p=0 N -1 q=0 (X 2 pq + X2 pq ) - 1 M N M -1 p=0 N -1 q=0 2|F |pq| F |pq = 1 M N M -1 p=0 N -1 q=0 (X 2 pq + X2 pq -2Xpq Xpq)+ 1 M N M -1 p=0 N -1 q=0 2 XpqXpq - 1 M N M -1 p=0 N -1 q=0 2|F |pq| F |pq = L2(X, X) + 1 M N M -1 p=0 N -1 q=0 2Xpq Xpq - 1 M N M -1 p=0 N -1 q=0 2|F |pq| F |pq.Apart from the L 2 component which is equivalent to Eq. ( 9), we also obtain two extra terms, shorthanded as 2X X and 2|F || F |. To study the empirical effect of the two factors on the high-frequency components, we performed a simple experiment: we sampled an image from the Moving-MNIST dataset and performed two modifications over time: (1) applying Gaussian blur with a standard deviation of σ to the sample, and (2) translating the sample along the direction (t, t). Then we observed the trend of increment of the two factors, as shown in Figure 4 . The figure reflects a couple of characteristics of the FAL loss term. First, in the case of blurring, it behaves similarly to the standard L 2 loss with a tiny difference when σ gets very large. Intriguingly, FAL does not exhibit a different degree of sensitivity to different frequencies. However, for translation, the absolute difference 2X X -2|F || F | is almost equivalent to and thus cancels out the L 2 loss, causing the final FAL loss term to become very small. It is also noteworthy that when the two samples X and X are identical, both 2X X and 2|F || F | are zero and thus FAL is also zero. From the observations above, FAL is invariant to global translations and robust to one-directional local translation compared to L 2 . Because of such invariance, FAL is more robust against the spectral bias and could better fine-tune the frequencies of the output. Utilizing this behavior, the model could focus on the reconstruction of a clear signal without suffering from the influence of the randomness in translations. Moreover, it also shows that an arbitrary scaling between L 2 and the factor 2X X -2|F || F | could not result in a desired effect, since this causes the two terms to no longer overlap in the plot over t, leading to an increase in sensitivity to translation.To understand how FCL affects the model during training, we derive its derivative with respect to Xkl and obtain an interesting result with the aid of Plancherel's theorem:EQUATIONFrom Eq. ( 11), the ratio X kl to Xkl highly depends on the summation over the image domain, providing global information to Xkl , unlike the element-wise or pixel-wise relationship between X and X in the conventional MSE loss, L 2 ( X, X).To have an intuitive understanding of the conclusion above, we design a thought experiment to understand how FCL is different from L 2 ( X, X) here. Consider the case where the prediction has the same image structure as the ground truth but with different intensity, for instance, X = βX, where β is an arbitrary non-zero constant.Substituting X into Eq. ( 11), we have∂ ∂ Xkl FCL(X, X) = 0.Meanwhile, it is straightforward that∂ ∂ Xkl L 2 ( X, X) ∝ -(1 -β).With the above discrepancy, the behavior of FCL is substantially different from MSE in regard to overall brightness. That is, MSE is affected by both the image structure and the overall brightness but FCL is affected by the image structure only. Therefore, with FCL alone, we lose the pixel intensity. While applying the sigmoid function is one method to alleviate the drawback of the missing information, incorporating FAL which focuses on the intensity in particular could be viewed as a parallel complement to further stabilize the models.In previous sections, we showed that the Fourier phase of the ground truth, θ, is missing in the gradient of FAL. Hence, we claim that FAL alone is insufficient to be a reconstruction loss. Similarly, in the thought experiment conducted in Appendix E, we conclude that FCL does not consider the image intensity and sharpness. As a result, the two loss terms FAL and FCL have to be used together as a full reconstruction loss. Here, we report the empirical results of the ablation study for FAL and FCL in Table 3 . In Table 3 , the model trained with FAL does not produce meaningful output, as reflected by the abnormal values of the metrics and the faulty predictions in Figure 5 . This agrees with our statement that models trained with FAL alone cannot converge to proper local minima. Meanwhile, the model trained with FCL only exhibits behaviors similar to MSE as shown in Figure 5 . To sum up, either using FAL or FCL alone does not empirically produce the desired effect. However, combining the two loss terms together achieves a huge improvement to most of the metrics, which agrees with our theoretical analysis.Next, we study the effect of α, which controls the length of the fine-tuning process with FAL. The fine-tuning process encourages the models to predict sharper and brighter predictions. At the same time, this can also lead to overfitting of noises in the high-frequency components. From Table 4 and 5, the sharpness-aware metrics such as LPIPS, FVD and RHD can always be improved by setting α to non-zero. However, there is no apparent improvement or decay after α ≥ 0.2. Therefore, we believe setting α = 0.1 or 0.2 as a default value can strike a good balance between sharpness and pixel-wise performance. In the previous sections, we showed that the method is both effective and generic of models. In this section, we discuss the running time of FACL. Theoretically, FACL utilizes DFT, which has time complexity O(n 2 ) in the 1D case with vector length n. By leveraging the 2D Fast Fourier Transform (FFT), we could improve the time complexity to O(M N (log(M ) + log(N ))) for each pair of frames, where M and N correspond to the height and width of the samples. With the aid of deep learning frameworks such as PyTorch, such operations can be run in parallel and supported by GPU. Therefore, the computational load for FACL is light compared to the deep network architectures.During inference, the only difference between models trained with MSE and FACL is that the FACL one consists of a sigmoid layer at the end. Running in parallel, again, this operation is negligible.To Apart from the proposed Stochastic Moving-MNIST, previous works proposed N-Body-MNIST [10] , a dataset that utilizes multiple transformations to simulate the chaotic nature of the atmospheric conditions. We present the results in this section. This section is an extension of Table 2 featuring PredRNN. Due to the limitation of the model and consideration of training efficiency, we downscale the radar data to 128 × 128 with bilinear interpolation. The results are reported in Table .8. Note that the image resolution influences some metrics such as LPIPS and CSI with pooling, causing an unfair comparison with the results in Table 2 . Apart from the previous works discussed in Section 2, there has also been a series of attempts to improve the loss function. For instance, Balanced Mean Squared Error (BMSE) [43] was proposed to increase the weighting on heavy rainfall. Multi-sigmoid loss (SSL) [48] preprocesses the images with linear transformations and nonlinear sigmoid function before applying MSE. Tran et al. [49] tested SSIM and MS-SSIM and recommended MSE+SSIM to be the loss function. We also present the results by adopting these losses with ConvLSTM trained on Stochastic Moving MNIST. For SSL, we follow the paper by picking i ∈ [ 20 70 , 30 70 ] and c = 20. One point worth noting is that none of the methods other than FACL generates a sharp prediction qualitatively, despite occasionally higher pixel-wise performance. Besides, we can draw the following conclusions:• SSL improves the model performance in general, but still cannot generate clear output under stochastic motion. • Losses integrating SSIM (and also L1) "dissolves" the prediction to zero over time under uncertainty. Such an effect is especially significant for weaker models. • Weighted MSE (such as BMSE) only "tilts" the focus. BMSE severely over-predicts in exchange for an improvement in CSI, sacrificing all other metrics such as MAE, SSIM, LPIPS, FVD, FSS and RHD.Apart than replacing the MSE loss in video prediction, we also study the scenario where FACL is applied to video generative models. Specifically, we have tested three generative models using different generative methods: MCVD [21] , a diffusion-based model; STRPM [45] , a GAN-based model training a recurrent generator; and SVGLP [46] , a VAE-based model with its default loss function being a composition of the MSE term and KL-divergence term. We replace the MSE term in each of these models with FACL, and report the quantitative performance in Table 11 , while its visualization is reported in Figure 9 .In the table, FACL exhibits to be a good substitute for MSE even in the generative models as it improves most of the metrics. For both SVGLP and STRPM, replacing the reconstruction loss with FACL further improves the image quality of the prediction, as reflected by the significant drop in FVD and RHD and vast improvement in FSS. On the other hand, using FACL in MCVD is much less intuitive as the MSE loss fits the diffusion output to Gaussian noise. Since there is no point in studying the noise frequencies, the performance gain attributed to FACL appears trivial, resulting in comparable performance between MSE and FACL. In the paper, we utilized three types of metrics to measure the similarity between two image sequences: pixel-wise and structural metrics, perceptual metrics and meteorological skill scores. Each suffers from its kind of drawbacks. For example, pixel-wise differences such as MAE and MSE do not consider the overall image structure, causing great encouragement to blurry prediction. Perceptual metrics such as LPIPS and FVD are based on pre-trained deep learning models (usually on ImageNet), which can suffer from domain bias that does not favor signal-based images. Moreover, deep perceptual metrics can be insensitive to transformations such as global rotation and spatial flipping.To study the behavior of these metrics and compare them with RHD, we sample a random precipitation event (as visualized in Figure . 10) and apply the following transformations to the image:• Gaussian blur with kernel size 27 and σ = 15.• Translation by (4, 4) .• Clockwise rotation by 5 • .• Brightening by 2X if the pixel value is higher than 0.5.• Darkening by 2X if the pixel value is lower than 0.5.The former three transformations study the robustness of the metric under blurring and transformation. The brightening action simulates forecasts that overestimate and the darkening action simulates forecasts that underestimate. After obtaining the transformed images, we measure the evaluation metrics between the distorted images and their corresponding ground truth. The result is reported in Table . 12. FVD is not computed since it requires a larger set of data to form an image distribution.In the table, a couple of behaviors deserve to be pointed out. First of all, pixel-wise and structural metrics appear to be insensitive to blur, which exhibits a huge difference compared with the other transformations in Figure 10 . Such a characteristic discourages small translation which is undesired for precipitation nowcasting. Perceptual metrics such as LPIPS behave the opposite, where blur is the most penalized and value scaling (brightening and darkening) is the most rewarded transformation. Despite this, we believe LPIPS penalizes too little on brightening and darkening as they could result in wrong alerts for extreme weather. For the skill scores, we again observe that CSI with larger pooling tolerates more translation and rotation. In other words, CSI with a large pooling size can be a good metric to penalize blur. However, since CSI discourages false positives, low-range prediction usually wins in CSI which is undesirable for extreme weather forecast. On the other hand, FSS results in unstable behavior for brightening and darkening, due to a single threshold which causes a huge error within a binary cutoff. Among the metrics, we find that RHD is more robust to spatial and pixel-wise transformation while penalizing blur. With the multi-class behavior of RHD, it is also much more stable without bias over overestimation or underestimation. To sum up, RHD can be viewed as a generalization of FSS or CSI with consideration of multiple radii and pooling sizes. KL-divergence is adopted to measure the similarity of class distribution, replacing the binary segregation used commonly in the meteorological skill scores. Unlike SSIM being a normalized score from 0 to 1, only inspecting the magnitude of RHD is not meaningful. Instead, the users need to specify a fixed set of parameters such as window size and bin ranges, such that relative comparisons between two forecasts under the same set of parameters can provide useful evaluation feedback.This section extends the visualizations in Figure 2 and Figure 3 by including the remaining models used in the experiments. Figure 11 visualizes an example output of the remaining models on Stochastic Moving-MNIST and Figure 12 visualizes that of SEVIR. In addition, we further plot an event from HKO-7 and MeteoNet, as shown in Figure 13 and Figure 14 respectively. Answer: [Yes]Justification: The retrieval links of the data are provided in the readme file in the code.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The paper proposes novel loss functions designed to enhance the prediction quality of precipitation nowcasting models without the use of generative models. This non-generative approach notably sharpens the spatiotemporal predictions. All reviewers agree the technical contribution and soundness in the initial pass of reviewing. Also, the authors' rebuttal fully addressed some questions and suggestions raised by reviewers. The AC recommends accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes the FACL loss function and provides theoretical and empiral proofs on how it boosts clarity and structure for images. The paper also shows how the loss behaves with generative setups and additionally proposes a new metric that is tolerant to deformations.
strength_weakness: strength_weakness: - Instead of the stochastic modification to Moving-MNIST, [1] already introduced a chaotic yet deterministic N-Body MNIST to mimic the complexity of Earth system interactions, would have been interesting to see the performance on this dataset
- FVD and LPIPS calculated on the basis of models trained on natural images, difficult to see why it would extend to scientific data

[1] Gao, Z., Shi, X., Wang, H., Zhu, Y., Wang, Y. B., Li, M., & Yeung, D. Y. (2022). Earthformer: Exploring space-time transformers for earth system forecasting. Advances in Neural Information Processing Systems, 35, 25390-25403.
questions: questions: - Does the alpha parameter swing wildly between datasets? Any good strategy to figure this out?
- What are the benefits of RHD over a simple wasserstein distance between predicted and true distributions or their quantile error?
limitations: limitations: Yes
paper_summary: paper_summary: This paper propose a new Fourier Amplitude and Correlation Loss (FACL) to replace the traditional L_2 losses in precipitation nowcasting task. They evaluated the FACL on one synthetic dataset and three radar echo datasets, which demonstrates their method improve perceptual metrics and meteorology skill scores. Besides, they propose Regional Histogram Divergence (RHD) to improve the error margin in meteorological skill scores.
strength_weakness: strength_weakness: 1. The evaluation is incomplete and unconvincing. The csi-m of EarthFormer is much lower than the results in [1] (0.3982/0.44). Besides, it could include more methods, such as NowcastNet[1] and DiffCast[2].
2. In figure9, compared to generative methods such as LDCast and MCVD, the FACL methods still lack predictions for extreme value and the distribution of predictions given by FACL is slightly different to the ground truth.
3. The intuition of FACL has no strong connection with precipitation nowcasting.
4. The coefficients of FAL and FCL needs to be selected manually, which may hinder the applications of FACL in other datasets and tasks.

Minor:
Models adopt the “prediction and refine” may help improve the related work section. 

[1] Skillful nowcasting of extreme precipitation with NowcastNet
[2] Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling
questions: questions: Why does the author measure the CSI of SEVIR and MeteoNet with the same threshold? The data in SEVIR are VIL images, while the ones in MeteoNet are radar reflectivity (dbz).
limitations: limitations: The evaluation is incomplete and unconvincing;lack predictions for extreme value;The intuition of FACL has no strong connection with precipitation nowcasting
paper_summary: paper_summary: The paper introduces novel loss functions - the Fourier Amplitude Loss (FAL), Fourier Correlation Loss (FCL) and a Regional Histogram Divergence (RHD) - to improve the realism of the predictions of precipitation nowcasting models without the use of generative models. The loss is applied to established precipitation nowcasting methods and is seen to significantly improve the sharpness and overall realism of the predictions.
strength_weakness: strength_weakness: The authors sensibly use MSE loss as a baseline when evaluating their model. However, there is quite a long history of developing new losses for making precipitation nowcasting produce sharper images and improve the representation of extreme precipitation. As such, I would have wanted to see a paper introducing a new loss function do some comparisons to earlier attempts to improve on the MSE loss if it wants to claim state-of-the-art performance. See e.g.:
https://proceedings.neurips.cc/paper/2017/hash/a6db4ed04f1621a119799fd3d7545d3d-Abstract.html
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019EA000812
https://www.mdpi.com/2073-4433/10/5/244
questions: questions: You mention that you introduce FCL because the FAL doesn't fully constrain the spatial positions in the image. A seemingly simpler solution would be to use the FAL in combination with MSE. Have you tried this approach and if so, do you see advantages with using FCL instead?

Also, you are using a loss where you randomize between FAL and FCL and the probability of using FAL increases over time. You motivate this (lines 210-214) as it being tricky to find the correct weighting of FAL and FCL for a linear combination, but I don't quite understand, how is this not functionally equivalent to using a linear combination with the weight of FAL increasing over time?
limitations: limitations: The paper discusses some limitations of the approach, but one point that is not mentioned is that unlike generative approaches to producing realistic nowcasting predictions, the model trained with the new loss function cannot produce multiple outputs for the same input, and is thus unable to provide stochastic predictions where multiple outputs from the model are used to characterize the probability distribution of precipitation.

Comments: title: The updated commonts
Regarding new SOTA models
The updated commonts
The updated table for MeteoNet with new CSI thresholds (12, 18, 24, 32)
comment: Thank the authors for their efforts. I will increase my original score. I suggest including the discussion about  SOTA models in the paper, which can help the reader know when to apply the proposed FACL.
We thank the reviewer for the follow-up responses. FACL is proposed to enforce sharpened forecasts for **video prediction** (VP; in contrast to video generation) models. The models we can apply FACL to are mostly **SOTA video prediction models**. The 3 most highlighted models in the paper: ConvLSTM, SimVP and Earthformer, correspond to three types of common model architectures in learning the spatiotemporal patterns: RNNs, CNNs and ViTs. In this category, we are able to provide results on more SOTA models / strong models, such as SimVP v2.

Stochastic Moving-MNIST:
| Model | Loss | MAE | SSIM | LPIPS | FSS | RHD |
| --- | --- | --- | --- | --- | --- | --- | 
|  SimVP v2 (gSTA) | MSE | 171.58 | 0.7625 | 0.1846 | 0.7464 | 0.9479 |
|  SimVP v2 (gSTA) | FACL | 178.55 | 0.7622 | 0.0959 | 0.8210 | 0.3581 |

The SOTA models the reviewer previously suggested were **video generation models**. In Appendix I, we showed that replacing the MSE reconstruction loss with FACL in GAN/VAE improves different metrics to a considerable degree, but replacing diffusion loss with FACL does not influence the results significantly. Moreover, video generation models usually impose constraints to the data such as the image resolution and sequence length, causing difficulties to align with conventional video prediction models. 

---

To sum up,
1. Applying FACL to **SOTA video prediction models** is the most preferred way of using FACL, which brings huge improvement in both perceptual metrics and skill scores.
2. Applying FACL to **SOTA VAE/GAN-based models** improves the models.
3. Applying FACL to **SOTA diffusion-based models** has almost no effect.
4. We are unable to include some generative models due to lack of implementation source and discrepancy in experimental settings. It is too computationally demanding to train some models (e.g. DiffCast) on the same datasets in full resolution.
Thanks the authors for the detailed responses. I agree that the proposed FACL is a good alternative to MSE, but I still want to know if the proposed FACL can improve the performance of the SOTA model or what are the benefits of applying it to the sota model
Thank you for the thorough response. I particularly appreciate the comments on robustness over the alpha parameters, and am also pleased to see the performance on the additional dataset. I will increase my original score.
Here we present the updated table for MeteoNet with new CSI thresholds (12, 18, 24, 32) based on the reviewer's feedback. Models trained with FACL still largely outperform those trained with MSE. With the increase of the CSI pooling radius, the performance of the diffusion models also drastically increases.

| Model          | Loss  | CSI-m  | CSI_4-m | CSI_16-m |
| ---  | --- | --- | --- | --- |
| ConvLSTM  | MSE  | 0.4388 | 0.3989    | 0.3904      |
| ConvLSTM  | FACL | 0.4161 | 0.4876    | 0.6041      |
| SimVP         | MSE  | 0.4221 | 0.3748    | 0.3627      |
| SimVP         | FACL | 0.4008 | 0.4513    | 0.5772      |
| Earthformer | MSE  | 0.4004 | 0.3327 | 0.2946 |
| Earthformer | FACL | 0.3594 | 0.4038 | 0.5250 |
| LDcast         |  -        | 0.2353 | 0.3188   |  0.4804      |
| MCVD          | -         | 0.3645  | 0.4559 | 0.6148       |