Has the Deep Neural Network learned the Stochastic Process? An Evaluation Viewpoint

## Abstract

Abstract This paper presents the first systematic study of evalution of Deep Neural Network (DNN) designed and trained to predict the evolution of a stochastic dynamical system, using wildfire prediction as a case study. We show that traditional evaluation methods based on threshold based classification metrics and error-based scoring rules assess a DNN’s ability to replicate the observed ground truth (GT), but do not measure the fidelity of the DNN’s learning of the underlying stochastic process. To address this gap, we propose a new system property: Statistic-GT, representing the GT of the stochastic process, and an evaluation metric that exclusively assesses fidelity to Statistic-GT. Utilizing a synthetic dataset, we introduce a stochastic framework to characterize this property and establish criteria for a metric to be a valid measure of the proposed property. We formally show that Expected Calibration Error (ECE) tests the necessary condition for fidelity to Statistic-GT. We perform empirical experiments, differentiating ECE’s behavior from conventional metrics and demonstrate that ECE exclusively measures fidelity to the stochastic process. Extending our analysis to real-world wildfire data, we highlight the limitations of traditional evaluation methods and discuss the utility of evaluating fidelity to the stochastic process alongside existing metrics.

## 1 Introduction

DNNs are widely used for modeling physical systems such as fluid dynamics [], economics [], neuroscience [], and wildfire modeling []. In wildfire modeling, DNNs use multivariate observations (24 hours to a year) including weather, topology, and fire evolution to predict future fire-map (one day to 30 days ahead), identifying whether specific locations will be burned or not [, , ]. The DNN must learn the "rules" of fire evolution to make accurate predictions.

A fundamental characteristic of physical systems like wildfires is stochasticity, as their unpredictable evolution makes the "rules" of fire evolution stochastic []. Fire evolution can be represented by a random variable (RV) $Z_{t}$, indexed by time $t$ []. Under the stochastic assumption, multiple possible ground truths (GTs) exist, but we observe a single realization (observed GT). During training, DNNs learn stochastic interactions from one set of instances and are evaluated on another, each instance sampled from a different stochastic process. Current evaluation methods do not assume a stochastic process; they only test if the DNN can predict the observed GT (we call this property Fidelity to Realization, F2R). We assert that faithfulness to the underlying stochastic process is not currently being evaluated. Specifically, in a DNN failure scenario, our goal is to identify whether the failure is due to the DNN’s inability to replicate the observed GT or encountering a different stochastic process.

We propose a property of interest and an evaluation metric to measure that property. We call this property fidelity to Stochastic Process (F2SP) and it measures how faithfully the DNN can predict Statistic-GT, which represents the GT of the stochastic process. A significant challenge is the inaccessibility of Statistic-GT, with only single instances (observed GT) available. Using a stochastic framework, we formulate the property and establish criteria for a metric to be a valid measure of the proposed property. For thorough testing, we introduce a synthetic benchmark with varying stochasticity levels. Using empirical experiments, we show that current evaluation metrics (classification-based and scoring rules) cannot measure the property. We formally establish ECE as a suitable candidate and conduct experiments showing it exclusively measures F2SP, unlike conventional metrics. We note that low calibration error meets the necessary criterion for measuring F2SP, but it is not sufficient. Finally, we extend our study to the recently released real-world wildfire dataset, "Next Day WildFire Spread" [], demonstrating the limitations of current evaluation metrics and the utility of the proposed evaluation criterion (§5).

At a high level, we pose two research questions: [RQ1] Why should we evaluate a DNN’s ability to learn the stochastic process? [RQ2] How can we evaluate whether a DNN has learned the stochastic process? Our key contributions are:

- •
Under RQ1, we develop a framework to characterize forest fire as a stochastic process, highlighting its complexity unobservable in real-world data (§3). We show that in highly stochastic scenarios, current evaluation criteria F2R exhibits high variance, underscoring the need for a stochasticity-compatible evaluation criterion (§3.3).
- •
For RQ2, we establish the fixed property of interest for a stochastic process, i.e., the Statistic-GT. We demonstrate that, unlike prevalent metrics, ECE exclusively evaluates the proposed property (§4). This application of ECE, differs from its conventional use in deep learning for assessing uncertainty estimates, and simplifies adopting the proposed criteria.

Figure: Figure 1: [Left] Evaluation metrics on the stochastic-deterministic plane: Classification-based metrics measure Fidelity to Realization (F2R), scoring rules measure Fidelity to Statistic (F2S), and we propose calibration error measures Fidelity to Stochastic Process (F2SP) (details in §3.4). F2SP captures properties not measured by F2R and F2S. [Right] Different sources of randomization in the synthetic forest fire datasets (details in §3.1). A stochastic process results only with stochastic fire evolution rules.

## 2 Background and Related Works

Formulation of DNN prediction and evaluation for Wildfire Spread. For a literature survey on DNN-based wildfire modeling, refer to §B.1. Formally, wildfire evolution is modeled on a grid of size $H\times W$ ($64\times 64$ in this work), where each grid-cell $(i,j)$ at time $t$ is denoted by a binary random variable $M_{t,(i,j)}$, indicating fire presence ($1$) or absence (0 0). Let $B_{t}\in\{0,1\}^{H\times W}$ encapsulate the fire occurrence status across all cells at time $t$. Additionally, let $O_{t,(i,j)}\in\mathbb{R}^{n}$ represent an $n$-dimensional vector of observational variables (e.g., vegetation, terrain, weather conditions) for each cell at time $t$ []. The DNN predicts the joint conditional distribution $\hat{P}_{t+T}$ of fire occurrences $T$ timesteps into the future using past fire maps $B_{1:t}$ and observational variables $O_{1:t}$:

$$ $\hat{P}_{t+T}\coloneqq\hat{P}(B_{t+T}|B_{1:t},O_{1:t})$ (1) $$

Post-prediction, we use the evaluation metric $S(B_{t+T},\hat{P}_{t+T}):(\{0,1\}^{H\times W}\times[0,1]^{H\times W})\to
\mathbb{R}$ to measure prediction accuracy by comparing the observed fire map $B_{t+T}$ with the predicted fire map $\hat{P}_{t+T}$. Different $S$ measure different system properties (see §3.4).

Stochasticity in wildfire evolution and challenges in stochastic modeling.
Stochasticity emerges from the interaction of various components—vegetation, terrain, weather conditions, and human activities, e.g., fire suppression efforts []. These factors add noise to the rules of fire evolution that DNNs aim to learn. The subtle variations in interaction rules can lead to significantly divergent outcomes from a given state, a phenomenon akin to the “butterfly effect" []. Real-world data limitations, typically presenting only a single outcome from a range of possibilities, obscure the true nature of stochastic interactions. To address the limitations of real-world datasets, we use a cellular automata-based forest fire model to simulate the range of possibilities (see §B.2 for applications of cellular automata in wildfire modeling). This approach allows us to study the stochastic dynamics at play in wildfire evolution, offering insights that are unconfined by the limitations of real-world data.

Related Works in Stochastic Video Prediction.
While deterministic DNNs are effective in predictable contexts (e.g., car movement on a straight trajectory), video prediction struggles with sporadic unpredictable events (e.g., car turning) [, ]. Therefore, stochastic DNNs predict a range of potential outcomes with high contrast. For evaluation, the authors report the score from the generated samples that best matches the observed GT, testing for fidelity to the observed GT []. Unlike the sporadic nature of stochasticity in video prediction, wildfires exhibit stochasticity consistently []. Our study diverges from evaluation strategy used in stochastic video prediction, emphasizing fidelity to statistics/stochastic-process as a more suitable strategy for evaluating a DNN for high-risk wildfire prediction. By providing stochastic interpretation and stochasticity-compatible evaluation methods, we aim to enhance the reliability of DNNs in stochastic wildfire prediction.

Figure: Figure 2: (a) The first four rows display four distinct realizations of fire evolution from the same fire seed location (initial condition). The last row shows the Statistic-GT of fire evolution, with each pixel representing the Bernoulli distribution parameter from which the realizations are sampled from; (b) [Left] Snapshot of forest-fire evolution in NetLogo using a 64x64 grid of agents, [Right] Flowchart depicting a single agent’s evolution rules inpired from Rothermel [].

## 3 Forest Fire as a Stochastic Process and Evaluation Strategies

### 3.1 Simulating a forest fire as a stochastic process

To empirically generate the stochastic process, we use an agent-based modeling framework called NetLogo to simulate a Forest-Fire model []. In an agent-based formulation of Forest Fire evolution, each pixel on a spatial grid, is treated as an agent. Agents interact with neighbors (in the Moore neighborhood) using stochastic rules. The local interactions collectively drive the global fire evolution, often manifesting in complex, fractal-like patterns (see Figure 2.b), simulating a wide spectrum of outcomes. We first establish deterministic interaction rules for neighboring agents (shown in Figure 2.b), and then add stochastic noise in the interactions (details in §C.1).

Empirical Stochastic Process (ESP). We represent an ESP using 1000 Monte Carlo (MC) simulations, each initiated with identical conditions (fire seed, forest configuration) and run until fuel depletion. This approach ensures variations in fire evolution stem exclusively from stochastic interactions. To make the rules stochastic, we introduce $p_{\text{ignite}}$, the probability of fire after ignition conditions are met (i.e., $q>q_{\text{threshold}}$). In deterministic scenarios, $p_{\text{ignite}}$ is set to 100, ensuring ignition once the threshold is reached. A $p_{\text{ignite}}$ value of 95 indicates a 95% chance of ignition under the same conditions. The S-Level parameter is defined as $(100-p_{\text{ignite}})$, with higher S-Level values indicating more randomness in agent interactions, and an S-Level of 0 corresponding to deterministic interactions. Figure 2.a depicts four distinct realizations of fire evolution (MC-1 to 4).

Distinguishing Random Instances of Deterministic Fire versus Stochastic Evolution of Fire. Figure 1 shows the sources of randomness in our synthetic dataset: forest configuration, fire seed location, and fire evolution rules. In deterministic evolution, the forest configuration and fire seed location are randomized, but fire evolution follows a single pathway, creating random instances of deterministic fire. In contrast, the ESP allows multiple evolutionary paths from the same initial conditions, resulting in stochastic fire evolution. When all elements are randomized, we obtain random instances of stochastic fire evolution, used for training DNNs in §4.1.

### 3.2 Micro and Macro Random Variable in ESP for Forest Fire Modeling

Figure: Figure 3: Illustrates the stochastic framework with Macro and Micro Random Variables (RVs) for forest fire evolution. ESP is shaded in red.

To formalize the modeling of stochastic fire evolution, we leverage insights from statistical mechanics []. At the microscopic level, individual agent behavior is captured through the Micro RV, $M_{t,(x,y)}$, while at the macroscopic level, the system behavior is represented by the Macro RV, $Z_{t}$. Figure 3 summarizes proposed framework.

Micro-Level Modeling. At the micro level, each grid point $(x,y)$ at time $t$ is modeled using a Micro RV, $M_{t,(x,y)}$, which follows a Bernoulli distribution with an expectation $E[M_{t,(x,y)}]$ equal to its probability $p$. The ensemble of Micro RVs across the grid, denoted as $\{M_{t,(x,y)}|1\leq x\leq H,1\leq y\leq W\}$, represents the grid-level micro representation of fire evolution. This Micro RV Map represents GT of the statistic, and termed Statistic-GT. In practical scenarios, a single realization (represented by one MC simulation) of the stochastic process (represented by the Statistic-GT) is observable. Parameters for each Micro RV are derived from the ESP by normalizing the burn frequency of each pixel at each timestep, across the MC simulations. The fifth row in Figure 2. (a) displays the Micro RV map for the ESP at S-Level 20, where each pixel’s value, $E[M_{t,(x,y)}]$, indicates the burn probability at that location and time, conditioned on the initial fire seed and forest configuration.

Macro-Level Modeling.
The Macro RV, $Z_{t}$, represents the collective state of the system at time $t$. Formally, we define $Z_{t}$ as the sum of the Micro RVs across the grid:
$Z_{t}=\sum_{x=1}^{H}\sum_{y=1}^{W}M_{t,(x,y)}$.
Applying the Central Limit Theorem for a large number of Micro RVs ($\approx 10^{3}$), $Z_{t}$ can be modeled using a Normal distribution, characterized by its mean $E[Z_{t}]$ and variance $Var[Z_{t}]$. Sampling from $Z_{t}$ provides a macro-state value, representing the aggregate state of all agents. It should be noted that multiple microstates can correspond to the same macrostate value. The parameters of the Macro RV are extracted from the ESP by recording the number of unburnt trees (macrostate) at each time step. Using the 1000 MC simulations, we create a distribution at each time step to compute $E[Z_{t}]$,$Var[Z_{t}]$.

Figure: Figure 4: Characterizes the Macro RV $Z_{t}$: (left) ESP representing $Z_{t}$ across different S-Levels: mean is the center line (Statistic-GT) and variance is the shaded region; (right) $Var[Z_{t}]$ over time for selected S-Level test cases used in this study. S-Level 20 shows chaos (peak variance).

### 3.3 Characterizing the S-Level Test Cases

Figure 4. (a) shows the Macro RV $Z_{t}$ over time with colors changing from red (S-Level=0) to black (S-Level=50) in increments of 5. The central line and shaded areas denote $E[Z_{t}]$ and $Var[Z_{t}]$. Figure 4.(b) displays $Var[Z_{t}]$ over time for select S-Levels used in our study. Key observations include: At S-Level 0 (deterministic), the absence of $Var[Z_{t}]$ implies a singular evolutionary path, an assumption implicit in current remote-sensing-based wildfire modeling [, , ]. As S-Level increases, $Var[Z_{t}]$ peaks at S-Level 20, indicative of a chaotic system with multiple fire evolution pathways, aligning with real-world fire behavior []. The $Var[Z_{t}]$ indicates the system’s tendency towards exploring diverse macrostates, with increasing variance indicating greater unpredictability.

Impact of macro-variance on the current evaluation strategy. As shown by $Var[Z_{t}]$, a system with stochastic interactions explores a vast state space, but typically only a single realization is observed. This makes the current evaluation strategy of testing fidelity to observed GT sensitive to the system’s macro-variance. Classification-based metrics, relying on thresholding, often misalign predictive outcomes with the observed GT (see §E.1). An effective evaluation strategy for such stochastic systems should not judge a DNN solely on replicating the observed GT; a DNN may model $Z_{t}$ accurately without replicating the observed GT. Therefore, a stochasticity-compatible strategy should assess fidelity to the Statistic-GT, which represents the complete stochastic process.

### 3.4 Testing Fidelity to Stochastic Process

As stated in Equation 1, given a DNN’s prediction $\hat{P}_{t+T}$ and the observed GT fire map $B_{t+T}$, the evaluation metric $S(B_{t+T},\hat{P}_{t+T})$ measures a specific property. The property of interest in the forest fire stochastic process is the Statistic-GT, represented by $P_{t+T}$. For a stochastic process, at a given timestep $t+T$, there can be multiple possible realizations, while there is only one $P_{t+T}$. Evaluating fidelity to Statistic-GT potentially offers stability over the long term (§4.2). The key challenge is testing fidelity to Statistic-GT using only observed GT, as Statistic-GT is not directly observable.

#### 3.4.1 Expected Calibration Error for testing Fidelity to Stochastic Process

We formally show that calibration error satisfies the necessary condition for evaluating the DNN’s fidelity to Statistic-GT (F2SP in Figure 1).
Consider a 64x64 grid where each cell $(i,j)$ is associated with a Bernoulli random variable $X_{ij}$, representing the presence (1) or absence (0) of a particular class. Let $\mathbf{X}=\{X_{ij}\mid i,j=1,\ldots,64\}$ be the set of all such variables. Given a DNN that outputs a predicted probability $\hat{p}_{ij}$ for each $X_{ij}$, we define the set of predicted probabilities as $\hat{P}=\{\hat{p}_{ij}\mid i,j=1,\ldots,64\}$. The predicted probabilities $\hat{p}_{ij}$ are grouped into $M$ bins $I_{m}=\left(\frac{m-1}{M},\frac{m}{M}\right]$. Let $B_{m}$ be the set of indices whose predicted probabilities fall into bin $I_{m}$:
$B_{m}=\{(i,j)\mid\hat{p}_{ij}\in I_{m}\}$. For each bin $B_{m}$, the fraction of positives is defined as:

$$ $\text{frac}(B_{m})=\frac{1}{|B_{m}|}\sum_{(i,j)\in B_{m}}y_{ij}$ $$

where $y_{ij}$ is the true class label for grid point $(i,j)$. A calibration curve plots $\text{frac}(B_{m})$ against the representative predicted probability $p_{m}$ for each bin $I_{m}$ (often taken as the midpoint of the bin). If the model is perfectly calibrated:
$\text{frac}(B_{m})=p_{m},\text{for all bins }I_{m}$. As a scalar summary statistic of calibration, we use the ECE []:
$\text{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{N}\left|\text{frac}(B_{m})-p_{m}\right|$.

ECE satisfies the necessary condition for evaluating fidelity to Statistic-GT. Using basic probability, we can show that $\text{frac}(B_{m})$ is an unbiased and consistent estimator of the true conditional probability $P(Y=1\mid\hat{P}\in I_{m})$. For a perfect predictor that predicts Statistic-GT, $p_{m}=P(Y=1\mid\hat{P}\in I_{m}),\forall\,I_{m}$, leading to low ECE (see §E.6 for calibration curves of the perfect predictor). Conceptually, calculating the fraction of positives $\text{frac}(B_{m})$ can be viewed as marginalizing over the subset of data points whose predicted probabilities fall within the bin $I_{m}$; this treats the data points as independent and ignores the dependencies between $X_{ij}$ for all $(i,j)\in B_{m}$. Therefore, low calibration error does not satisfies the sufficient criterion for assessing F2SP.

Application of ECE in Deep Learning vs. Our Work.
ECE and calibration curves are commonly used in deep learning to measure the quality of uncertainty estimates []. Predictive uncertainty in traditional deterministic tasks like image classification manifests in two forms: data uncertainty from measurement errors and model uncertainty due to the model’s unfamiliarity with certain data points []. Unlike these uncertainties arising from limitations in the DNN pipeline, our work encounters aleatoric (irreducible) uncertainty, which stems from the macro variance inherent in the physical system. Our application of ECE assesses whether the DNN has accurately learned the stochastic interactions and the associated stochastic process, rather than just measuring the quality of predictive uncertainty, which, while still possible, is secondary to evaluating our proposed criteria.

#### 3.4.2 Baseline evaluation metrics: classification-based and scoring rules.

Fidelity to Realization. Precision, Recall, and AUC-PR are favored for their ability to delineate type-1 and type-2 errors. These metrics are standard in DNN-based wildfire prediction due to the binary nature of fire spread outcomes and their direct link to decision-making in high-risk scenarios. They exclusively evaluate how faithfully the thresholded predictions match the observed GT.

Fidelity to Statistic. Metrics like Mean Squared Error (MSE) evaluate probabilistic forecasts and belong to the class of proper scoring rules []. A proper scoring rule $S$ ensures that the expected score, under the ground truth (GT) distribution $D_{\text{gt}}$, is minimized by an accurate forecast distribution $D_{f}$:
$\mathbb{E}_{y\sim D_{\text{gt}}}[S(y,D_{\text{gt}})]\leq\mathbb{E}_{y\sim D_{
\text{gt}}}[S(y,D_{f})].$
A scoring rule is strictly proper if this minimum is unique, guaranteeing that the best score corresponds to the forecast matching the GT distribution. However, with finite samples, scoring rules may not fully capture forecasting errors []. According to the Brier Score decomposition [], MSE splits into Calibration and Refinement:

$$ $\text{Brier Score}=\underbrace{\sum_{m=1}^{M}\frac{|B_{m}|}{N}(\text{frac}(B_{ m})-p_{m})^{2}}_{\text{Calibration}}+\underbrace{\sum_{m=1}^{M}\frac{|B_{m}|}{ N}\text{frac}(B_{m})(1-\text{frac}(B_{m}))}_{\text{Refinement}}$ (2) $$

While we have discussed Calibration Error (in §3.4.1), the Refinement term is a property of the system, and measures the variability of actual outcomes, captured by $Var[Z_{t}]$. As observed in Figure 4, as the s-level or forecast horizon increases, $Var[Z_{t}]$ increases, which raises the refinement value. The decomposition highlights the key difference between MSE and Calibration Error; the latter solely evaluates the DNN’s alignment with Statistic-GT, excluding the impact of $Var[Z_{t}]$ discussed in §3.3. We refer to scoring rules as Fidelity to Statistic to highlight this distinction from F2SP.

## 4 Benchmark Experiments

### 4.1 Generating the trained DNN models

We modify a single-layer convLSTM architecture chosen for its simplicity []. The convLSTM-CA architecture retains spatial information, essential for learning cellular automata rules (§D.1). The dataset contains 1,000 forest fire simulations for each S-Level (0, 5, 10, 15, 20), with each simulation including random fire seeds and forest configurations (Stochastic Train/Test in Figure 1). Each simulation represents one random realization of a fixed stochastic process, where different S-Levels indicate realizations from different processes. For each S-Level, we train a single DNN model using 700 simulations. The training data, segmented into 60-frame intervals, involves the DNN observing the first 10 frames in RGB and predicting the next 50 frames using binary segmentation (1 for burnt, 0 for unburnt). We use Binary Cross Entropy (BCE) loss for training. In the main paper, we show results for ConvLSTM-CA. We extend our evaluation to multiple DNN architectures in §E.2, including convLSTM variants [] (multi-layer, spatial bottlenecks) and a DNN architecture called Attentive Recurrent Neural Cellular Automata (AR-NCA) [], which specializes in modeling locally interacting discrete dynamical systems like forest fire evolution.

Figure: Figure 5: DNN trained on one S-Level, tested on another, evaluated by (a) (1 - AUC-PR)$\downarrow$, (b) (1 - Recall)$\downarrow$, (c) MSE$\downarrow$, and (d) ECE$\downarrow$. Diagonal behavior, evident in ECE, shows the metric’s ability to assess whether DNN has learned the correct stochastic process.

Figure: Figure 6: Long Horizon Performance of the DNN using (a) AUC-PR, (c) MSE, and (d) ECE [calibration curve in inset]. ECE demonstrates that the DNN’s long horizon predictions remain stable.

### 4.2 Testing ECE’s ability to assess fidelity to stochastic process

ECE vs. Baselines. In our simulation of the ESP, S-Level uniquely determines the Statistic-GT at each timestep. We hypothesize that an appropriate evaluation metric performs best when the S-Level of the trained DNN matches the test split’s S-Level. To test this, we conduct an experiment with five DNN versions from §4.1, each trained at a specific S-Level. We evaluate each DNN across five test splits, each at a different S-Level. The results are shown in the matrix heatmap in Figure 5, one for each evaluation metric. Rows represent evaluation scores for a DNN trained at a specific S-Level, and columns show scores for each test split at corresponding S-Levels. Only the ECE matrix shows clear diagonal behavior, indicating optimal scores when training and test S-Levels match. The MSE matrix shows some diagonal trends, but its measure of Refinement obscures this pattern (§3.4.2). Classification metrics like AUC-PR exhibit no such pattern. We confirm the general applicability of these findings across other DNN architectures: convLSTM variants [] and AR-NCA [] (§E.2).

Long horizon evaluation behavior of ECE.
As posited in §3.4, evaluating fidelity to Statistic-GT potentially offers long-term stability (§4.2) due to the presence of a single Statistic-GT. We hypothesize that an appropriate error evaluation metric should maintain a constant and low value throughout the prediction window. Figure 6 shows DNN performance using AUC-PR, MSE, and ECE at the same S-Level test split, mirroring the diagonal in Figure 5. Each evaluation score aggregates predictions and GT across 300 test simulations per time step. While baselines show declining scores with increasing S-Level and longer prediction horizons, ECE scores remain stable, indicating faithfulness to the Statistic-GT. Experimental results show that when DNNs are trained and tested on different S-Levels, long-horizon ECE performance drops noticeably (see §E.3). In the special case of deterministic evolution (S-Level 0), all three evaluation metrics agree. For a perfect predictor, Refinement is 0 for deterministic evolution since $\text{frac}(B_{m})=0$ or $1$ (§3.4.2), resulting in stable MSE. AUC-PR (F2R) is effective since there is only one possible realization.

## 5 Application to Real World Wildfire Prediction

Finally, we extend our study to a real-world wildfire dataset. Since this is real-world data, Statistic-GT cannot be calculated as only one realization of $Z_{t}$ is observed, yet the system is highly stochastic. Unlike the neat division of s-levels in the synthetic dataset, the real-world dataset is composite, meaning the individual samples have different, unobservable s-levels. Further, the agents are heterogeneous, and the interaction rules are far more complex than in the synthetic dataset. This section addresses two main themes: (1) addressing the disconnect between negative evaluation scores and positive qualitative observations discussed in the Next Day Wildfire Spread (NDWS) dataset [], and (2) using multiple DNNs to simulate scenarios and select the most suitable DNN. In this context, we discuss how ECE can complement existing metrics and guide the selection of DNNs.

Figure: Figure 7: DNN takes as input fire mask (left column) alongside 11 observational variables to forecast the next-day fire mask (right column), compared against the observed ground truth (GT) in the middle column. Evaluation through F2R metrics indicates suboptimal performance, whereas the MSE score, despite being low, does not adequately reflect the DNN’s practical efficacy.

### 5.1 Dataset and DNN Models used

Dataset. The NDWS dataset compiles historical wildfire incidents into two-dimensional regions at a 1 km resolution using 11 observational variables: elevation, wind direction and speed, temperature extremes, humidity, precipitation, drought index, vegetation, population density, and Energy Release Component (details in §C.2) []. The dataset documents 18,545 wildfire events, providing sequential snapshots of fire spread at times $t$ and $t+1$ day.

DNN Models. DNN input is a spatial map of 11 observational variables and fire spread at time $t$. DNN output is binary map for fire spread at time $t+1$. We use five architectures: Conv-AE, a convolutional autoencoder from Huot et al. []; Conv-CA, a modified Conv-AE without the spatial bottleneck (details in Appendix D.1); AR-NCA, an Attentive Recurrent Neural Cellular Automata for locally interacting discrete systems like forest fires []; SegFormer, a transformer-based segmentation model []; and U-Net, a convolution-based model [].

### 5.2 Revisiting the Evaluation of Conv-AE

Experiment. Huot et al. reported Conv-AE’s AUC-PR score of 0.284 on the test split. They acknowledged that “while the metrics on the positive class seem low," qualitative visualizations showed that "fires are predicted," "predicted fires are roughly in the target location," and there is "good recognition of larger fires" []. Similar observations are made in Figure 7, displaying the fire mask at time $t$, the next-day fire mask at time $t+1$ (observed GT), and the Conv-AE forecast. The low scores suggest poor performance, yet qualitative insights suggest otherwise. We hypothesize this disconnect stems from the highly stochastic nature of wildfires, compromising classification-based metrics. Our aim is to re-evaluate the DNN’s predictions considering our new insights.

**Table 1: Evaluation Metrics for Conv-AE stratified by Dice Coefficient values measuring Fire Map Overlap (FMO) between $t^{th}$ and $t+1^{th}$ day. ($\uparrow$ means higher is better)**
| FMO (DC) | Sup. | Precision$\uparrow$ | Recall$\uparrow$ | AUC-PR$\uparrow$ | MSE$\downarrow$ | ECE$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| 0.9-1.0 | 1 | 0.000 | 0.000 | 1.000 | 0.000 | 0.001 |
| 0.8-0.9 | 4 | 0.000 | 0.000 | 0.445 | 0.001 | 0.006 |
| 0.7-0.8 | 3 | 0.771 | 0.578 | 0.693 | 0.003 | 0.008 |
| 0.6-0.7 | 75 | 0.571 | 0.700 | 0.624 | 0.013 | 0.019 |
| 0.5-0.6 | 116 | 0.475 | 0.606 | 0.512 | 0.019 | 0.025 |
| 0.4-0.5 | 145 | 0.373 | 0.501 | 0.355 | 0.017 | 0.022 |
| 0.3-0.4 | 142 | 0.324 | 0.381 | 0.283 | 0.019 | 0.022 |
| 0.2-0.3 | 217 | 0.281 | 0.284 | 0.217 | 0.017 | 0.014 |
| 0.1-0.2 | 150 | 0.206 | 0.121 | 0.128 | 0.017 | 0.013 |
| 0.0-0.1 | 836 | 0.085 | 0.028 | 0.044 | 0.007 | 0.005 |
| Overall | 1689 | 0.346 | 0.311 | 0.247 | 0.012 | 0.012 |

Results. Huot et al. reported overall scores for Precision, Recall, and AUC-PR. We use the Dice Coefficient (DC) between the fire mask at time $t$ and the next-day fire mask to quantify fire evolution. Evaluation scores stratified by DC values are presented in Table 1. Higher DC values indicate greater fire map overlap, signifying gradual fire progression, while lower values indicate abrupt changes. Smaller fires tend to have lower overlap, while larger fires have higher overlap. For instance, a DC of 0.08 in Figure 7 signifies a complete shift in the fire front, while a DC of 0.64 shows slower progression from time $t$ to $t+1$. Table 1 shows that all metrics generally decrease as overlap decreases. However, lower MSE and ECE scores for small overlaps indicate fewer probabilistic errors and better modeling of the stochastic process by the DNN. The calibration curve (see §E.5) confirms the DNN makes overconfident predictions in the middle probability region but well-calibrated predictions at the extremes. For example, locations predicted to burn at $p=0.80$ did indeed burn 80% of the time. This supports the DNN’s ability to learn the stochasticity in fire map evolution, addressing the disconnect between positive observations and negative traditional evaluation scores in [].

**Table 2: Performance of DNNs on NDWS Dataset, with ranks in parentheses by each evaluation metric.**
| DNN | AUC-PR$\uparrow$ | MSE$\downarrow$ | ECE$\downarrow$ |
| --- | --- | --- | --- |
| Conv-AE [] | 0.2473 (5) | 0.0124 (4) | 0.0119 (4) |
| Conv-CA | 0.2631 (4) | 0.0146 (5) | 0.0207 (5) |
| AR-NCA [] | 0.2790 (2) | 0.0099 (2) | 0.0012 (1) |
| SegFormer [] | 0.2727 (3) | 0.0100 (3) | 0.0020 (2) |
| U-Net [] | 0.3302 (1) | 0.0096 (1) | 0.0023 (3) |

### 5.3 Conflict in DNN Rankings

Conflicts in model rankings are a vital component of model selection []. We simulate selecting the optimal DNN from a candidate set and explore how ECE complements existing metrics. Table 2 shows overall evaluation scores (AUC-PR, MSE, and ECE) for five DNNs on the test split, revealing rank conflicts among the metrics. This raises a crucial question: Which metric most accurately reflects model performance? As per our insights, each metric assesses different performance facets: AUC-PR measures Fidelity to Realization, MSE measures Fidelity to Statistic (including system macrovariance), and ECE measures Fidelity to the stochastic process. The choice depends on the application priorities. For example, while MSE scores for AR-NCA, SegFormer, and U-Net are similar, AR-NCA’s ECE is about half that of the others. If prioritizing Fidelity to the Stochastic Process, AR-NCA is the best choice. Conversely, if Fidelity to Realization is more critical, U-Net, with its competitive ECE, is preferable. The choice relies on estimating system randomness. For highly random systems, Fidelity to Statistic and Fidelity to the Stochastic Process are more reliable; Fidelity to Realization works better in low randomness. Ultimately, this study advocates a multi-faceted evaluation approach, emphasizing Fidelity to the Stochastic Process as crucial component.

## 6 Conclusion and Limitations

We propose a new evaluation criterion to assess a DNN’s ability to learn the stochastic interactions in forest fire evolution. Through controlled experiments with a synthetic framework, we show that ECE exclusively evaluates this capability compared to classification-based metrics or MSE. Notably, ECE reveals that long-horizon prediction performance remains stable if interaction randomness is constant, unlike current metrics like AUC-PR and MSE. With a refined understanding of each class of evaluation metric, we apply our insights to a real-world wildfire dataset. Here, we address the disconnect between positive qualitative assessments and negative performance scores and demonstrate how ECE can resolve conflicts in DNN rankings. Our work not only sheds light on the complexities of evaluating DNNs in stochastic systems but also paves the way for developing more robust and interpretable evaluation strategies for such systems.

Limitations of ECE in stochastic contexts.
ECE requires a sufficient number of samples for convergence, typically around 10 grids of size 64×64 in our synthetic dataset (§E.4). Additionally, ECE exhibits lower discriminative capabilities compared to classification-based metrics (§E.2). This is because, according to the Brier Score decomposition (Eq. 2), calibration error excludes the refinement term, which measures the sharpness of predictions and captures the discriminative capabilities of a DNN. As with all statistical measures, qualitative context should always be considered.

Applications beyond wildfire prediction and Future Work. Our study’s implications extend beyond wildfire prediction, offering insights into evaluating DNNs in binary classification tasks within physical and social systems, such as epidemic spread [, ] and rumor propagation [, ]. These systems exhibit complex behaviors from simple local interactions. Traditional metrics focusing on fidelity to observed GT may not fully capture a DNN’s ability to predict the underlying stochastic process. A low score in these metrics doesn’t necessarily indicate a failure to learn the system. Future work will extend our experiments to such domains to validate our findings.

## Appendix A Overview of the Supplementary Material

- •
Literature Survey B
–
Models for wildfire modeling B.1
–
Cellular Automata (CA) models in wildfire applications B.2
- •
Background C
–
Forest fire simulation in the NetLogo model C.1
–
Observational variables in Next Day Wildfire Spread dataset C.2
- •
DNN Characterization D
–
Design Rationale behind convLSTM-CA and conv-CA D.1
–
Qualitative Visualizations of DNN Forecasts D.2
–
Visualizing the DNN’s predictions using Empirical Stochastic Process D.3
- •
Additional Results E
–
Impact of macro-variance on the evaluation metrics E.1
–
Generalization study of ECE to other DNNs E.2
–
Long range behavior of ECE E.3
–
Asymptotic guarantees of ECE E.4
–
Calibration Curve on NDWS dataset E.5
–
Calibration Curve of the perfect predictor predicting Statistic-GT E.6

## Appendix B Literature Survey

### B.1 Models for wildfire modeling

Currently, forest fire spread models are categorized into three classes: empirical, semi-empirical, and physical. Empirical models, e.g., DNN modeling using remote sensing data [], analyze fire data statistically without exploring combustion mechanisms. Semi-empirical models, often the preferred choice, like [, ] integrate physical laws, such as heat transfer, but necessitate resource-intensive ground surveys for calculating model parameters []. Physical models, involving complex equations for heat dynamics [], are too complex for broad application. All these classes of models are deterministic and do not explicitly assume fire dynamics to be stochastic.

**Table 3: Selected works in Wildfire Prediction**
| Work | Window Size | DNN | Evaluation Metric |
| --- | --- | --- | --- |
| [] | Obs.: T<br>Pred.: T+24h | CNN | F1-score,<br>Recall, Accuracy |
| [] | Obs.: [T-52w, T]<br>Pred.: T+5w | CNN,<br>LSTM | AUC-ROC, MSE |
| [] | Obs.: T<br>Pred.: T+24h | CNN | AUC-PR,<br>Precision, Recall |

DNN-based modeling of Wildfires. A critical application area is wildfire prediction, where the need for improved forecasts is driven by the growing frequency and severity of wildfires [, ]. Leveraging the broad spatial and temporal coverage of remote sensing data from satellites [] and aircraft-based sensors [], DNN models are trained to capture fire evolution dynamics.
Conventional modeling tools, such as FARSITE [], have high operational costs due to reliance on ground-based data collection.
The growing application of DNNs in wildfire prediction, especially with remote sensing data, is evident in key studies [, , ] summarized in Table 4 and reviewed comprehensively in []. For example, FireCast [] outperforms traditional models by 20% in predicting 24-hour wildfire perimeters using satellite imagery [] (see §B.1 for a taxonomy on wildfire models). These DNNs integrate various covariates like vegetation, terrain, and weather conditions; for example, Huot et al. use a 64 x 64-pixel grid, each containing 11 observational variables []. DNNs operate in two phases for wildfire prediction: learning fire evolution rules during observation, and applying this knowledge for future predictions. DNNs are trained using Binary Cross Entropy (BCE) loss. Post-training, the DNNs are evaluated using evaluation metrics [] in Table 4(^1^11AUC-ROC is not recommended due to class imbalance [, ].).

### B.2 Cellular Automata (CA) models in wildfire applications.

CA models, with each pixel acting as an individual agent, offer a natural way to simulate stochastic interactions. These models, defined by discrete space and time, and marked by local spatial interactions, align well with the geographical nature of forest fires []. In CA-based forest fire models, each cell on fire is an agent capable of spreading the fire based on neighborhood interaction rules, leading to emergent behaviors that mirror real-life fire propagation patterns [, ]. CA models have been employed for various purposes in wildfire modeling, such as learning fire spread rules [], learning parameters influencing agent interactions [], and investigating chaos in fire evolution using Mean Field techniques [].

## Appendix C Background

### C.1 Forest fire simulation in the NetLogo model

The simulation starts on a $64\times 64$ grid, with each pixel initialized as a ‘tree’ or ‘no-tree’. Agents have heat values $q_{(i,j)}$ crucial for the heat transfer in forest fire evolution. Fire seeds, placed at randomized (or fixed) locations, provide initial heat to agents. The initial condition is set as $q_{(i,j)}=I_{\text{seed}}\times q_{\text{threshold}}$ for seed locations $(i,j)$, and $q_{(i,j)}=0$ otherwise, where $q_{\text{threshold}}$ is the ignition threshold and $I_{\text{seed}}$ amplifies seed heat values. A ‘tree’ agent accumulates heat from activated neighbors in its Moore neighborhood, in line with heat transfer mechanisms described by [], following the equation:

$$ $q_{(i,j)}(t+\Delta t)=q_{(i,j)}(t)+\sum_{(k,l)\in N_{R}}\mathbf{1}_{(k,l)}(t)q _{(k,l)}(t)$ $$

The indicator function $\mathbf{1}_{(k,l)}(t)$ ensures only ‘fire’ state agents contribute to heat transfer. An agent’s heat value $q_{(i,j)}$ exceeding $q_{\text{threshold}}$ triggers a state change from ‘patch’ to ‘fire’, and then to ‘ember’ in the next time step. ‘Ember’ agents radiate heat at a rate of $q_{\text{die}}$ to adjacent non-fire patches, gradually losing heat until radiation ceases. This process ends when ‘ember’ agents darken, indicating $q_{(i,j)}$ falling below a certain threshold, thus terminating heat radiation and transitioning to the ‘dead’ state.

### C.2 Observational variables in Next Day Wildfire Spread dataset

**Table 5: Observational variables and their description in the Next Day Wildfire Spread dataset []**
| Observational Variable | Description |
| --- | --- |
| Elevation | Terrain height above sea level |
| Wind Direction | The direction from which the wind originates |
| Wind Speed | Velocity of the wind |
| Minimum Temperature | Lowest daily temperature |
| Maximum Temperature | Highest daily temperature |
| Humidity | Amount of water vapor in the air |
| Precipitation | Amount of rain, snow, etc., that falls |
| Drought Index | Measure of dryness indicating drought conditions |
| Vegetation | Vegetation indices indicating plant health and coverage |
| Population Density | Number of individuals per unit area |
| Energy Release Component (ERC) | Indicator of fire potential energy release |

## Appendix D DNN Characterization

### D.1 Design Rationale behind convLSTM-CA and conv-CA

Figure: Figure 8: (a) Auto-regressive training of the DNN, (b) Softmax probability map from a convLSTM with a bottleneck, and (c) Softmax probability map from convLSTM-CA with no bottleneck.

The ConvLSTM architecture is chosen for its ability to efficiently model spatiotemporal systems, aligning well with the characteristics of the Forest-Fire system. Our modified version (see Figure 8.(a)), convLSTM-CA, places a ConvLSTM cell with a $3\times 3$ kernel between an Encoder (with a $3\times 3$ kernel) and a Decoder (with a $1\times 1$ kernel). The Encoder takes an RGB image and transforms it into a latent tensor during a 10-timestep observation window. This tensor is then processed by the ConvLSTM cell, maintaining its spatial dimensions, before the Decoder produces a burnt map grid of softmax probabilities.

This design choice of preserving spatial dimensions is crucial for modeling the cellular automata application. Reducing the spatial dimensions of the latent tensor is a popular design choice in video prediction models such as recurrent neural network-based [, ] and simple CNN-based models []. This reduction is often employed to decrease computational complexity and to capture essential spatial features while discarding less informative details. However, we observe that adding a bottleneck causes individual pixels (agents) to lose their identity. Maintaining these dimensions helps preserve each agent’s identity, a critical factor for developing a DNN that minimizes mis-calibrated forecasts arising from limited model capacity. For instance, as seen in Figure 8. (b), using a compressed latent dimension results in a forecast cloud around predictions for deterministic fire evolution scenarios, which does not accurately reflect the system’s true evolution. In contrast, as shown in Figure 8.(c) an uncompressed latent space yields predictions without a forecast cloud, aligning closely with the deterministic system’s true evolutionary rules. Conv-CA used in Section 5.3, is a version of Conv-AE [] with the spatial bottleneck eliminated.

### D.2 Qualitative Visualizations of DNN Forecasts

Figure: Figure 9: Qualitative Visualizations of the DNN’s forecasts for four different MC simulations. Frames are shown every five time steps. The DNN observes the first 10 time steps (first 2 frames) and predicts the next 50 time steps (last 10 frames). We can observe the difference between the DNN’s forecasts and the statistic GT, which arises because of the determinism that is injected into the DNN’s predictions due to its observation of the first 10 frames of the fire evolution.

### D.3 Visualizing the DNN’s predictions using Empirical Stochastic Process

Figure: Figure 10: shows the correlation between Statistic-GT and the DNN forecast, indicating that the DNN is learning to predict the statistic representing the stochastic process. Fidelity to Statistic-GT is what we aim to measure.

Figure: Figure 11: (a) Qualitative visualizations of forest-fire snapshots for different S-Level (one per row) showing statistic-GT, observed GT, and the corresponding DNN Prediction (raw forecast values); (b) Histogram shows the frequency of DNN’s forecast values for different S-Level values. We can observe that the DNN’s prediction becomes less confident with an increase in S-Level

In Figure 10, we plot the mean Statistic-GT against the corresponding mean DNN Forecast. The correlation between them suggests that the DNN is learning to predict the Statistic-GT (details in §D.3). This capability is linked to the use of BCE loss (a proper scoring rule), which promotes calibrated forecasts [].
Figure 11.a shows the DNN’s Statistic-GT, Observed-GT (MC sample), and DNN predictions for three different S-levels. We can observe that higher S-Levels correlate with greater ‘cloudiness’ in DNN output. Figure 11.b illustrates histograms of DNN predictions across 1000 simulations for each S-Level ESP. In deterministic settings, predictions are primarily binary (1 or 0), but at S-Level 20, predictions span the 0-1 range, indicating increased predictive uncertainty in highly stochastic scenarios.

## Appendix E Additional Results

### E.1 Impact of macro-variance on evaluation metrics

Figure: Figure 12: Shows the SD of the evaluation metrics vs. $Var[Z_{t}]$ for S-Level 20; higher SD indicates greater sensitivity to $Var[Z_{t}]$ and reduced reliability.

Methodology. Under RQ1, we further investigate the impact of $Var[Z_{t}]$ on currently used evaluation metrics: classification-based and scoring-rule. For this experiment, we perform inference on the DNN trained with S-Level 20, a setting that exhibits chaotic behavior typical of wildfire evolution. The test dataset comprises 1000 MC simulations of the S-Level 20 ESP test case, all with identical initial conditions. We calculate the evaluation metric independently for each timestep of each MC simulation. Each simulation at a given timestep $t$ has a specific $Var[Z_{t}]$, reflecting macro-variance at that moment. We measure the Standard Deviation (SD) of the evaluation metric across the 1000 MC simulations against $Var[Z_{t}]$ to assess the metric’s sensitivity to macro-variance. Since all simulations originate from the same stochastic process, an ideal metric should be faithful to the stochastic process, and demonstrate minimal sensitivity to individual realizations of the ESP.

Results.
Figure 12 shows the SD of evaluation metrics against $Var[Z_{t}]$. The significant increase in SD for classification-based metrics (Precision, Recall, AUC-PR) highlights their heightened sensitivity to macro-variance and reduced reliability. Precision and Recall suffer high variance in stochastic settings due to their reliance on thresholding, which can produce predictive outcomes misaligned with the actual GT. This issue stems from the incompatibility of "thresholding" with stochastic processes, extending beyond the common critique of arbitrary threshold selection, such as a 0.5 cutoff []. Although AUC-PR is less sensitive than other threshold-based metrics, it still encounters challenges from its integration of False Positives (FP) through Precision, where the traditional concept of FP loses relevance because any non-zero forecast by the DNN renders both outcomes (1 and 0) plausible. Conversely, MSE demonstrates less sensitivity to macro-variance due to its basis as a strictly proper scoring rule [], ensuring greater resilience.

### E.2 Generalization study of ECE to other DNNs

Figure: Figure 13: Evaluation metric scores for four different DNNs, from strongest (leftmost) to weakest (rightmost) using evaluation metrics (a) ECE, (b) MSE, (c) AUC-PR, and (d) Recall. While MSE, AUC-PR, Recall depict marked degradation in performance, ECE remains fairly stable, though slight increase is observed. In general, ECE shows lower variation on the synthetic dataset, indicating that it has a lower discriminating power.

Details about the DNN Architectures: (1) AR-NCA []: AR-NCA involves a recurrent cellular attention module that couples long short-term memory (LSTM) and cellular self-attention, (2) multi-layer ConvLSTM (num_layers=2), (3) convLSTM-btk-2: convLSTM with a spatial bottleneck that downsamples by 2, (4) convLSTM-btk-4: convLSTM with a spatial bottleneck that downsamples by 4. Using the deterministic case as the basis (train-test s-level=0), DNN quality is of the order $1\approx 2>3>4$.

### E.3 Long range behavior of ECE

Figure: Figure 14: DNNs trained on S-Level 10 (orange) and S-Level 20 (blue) and evaluated on S-Level 20. Scores reported by evaluation metrics (a) AUC-PR, (b) MSE, and (c), ECE. Since each DNN has learnt different stochastic interactions, divergence in DNN-S-Level 10 can be observed in ECE which asseses fidelity to stochastic process.

In this experiment we evaluate what happens when the DNN learns from one s-level and is evaluated on the other. In this specific example, one DNN is trained on S-Level 20 and the other is trained on S-Level 10. They are both evaluated on S-Level 20. Figure 14 shows the performance of the two DNNs reported by AUC-PR, MSE, and ECE. AUC-PR doesn’t show any difference between the two DNNs, as it’s not evaluating the statistic. MSE and BCE show visible difference. ECE shows the steepest difference because the DNN learnt a different noisy interaction rule, affecting the probabilistic accuracy. The system variance common in both test cases, reduces the gap in MSE.

### E.4 Asymptotic guarantees of ECE

Figure: Figure 15: Testing the asymptotic guarantees of ECE. Each sample is a grid of predictions at t=59 in the test split of the synthetic forest dataset. As the number of samples increase in the calculation, ECE converges. We can observe, that for the asymptotic guarantees to kick in, sufficient number of samples are required.

### E.5 Calibration Curve on NDWS dataset

Figure: Figure 16: [Left] displays the histogram of forecasts generated by Conv-AE; [Right] shows a Calibration Curve that illustrates forecast accuracy in an interpretable manner. This curve suggests (1) the DNN’s tendency towards overconfidence in mid-range forecasts, and (2) the accuracy of its probabilistic predictions is better for forecasts at the lower and upper ends of the probability spectrum.

### E.6 Calibration Curve of the perfect predictor predicting Statistic-GT

Figure: Figure 17: Calibration of a perfect predictor (predicting Statistic-GT) for different ESP test cases.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: - I don't understand the second part of the critical question, "is it encountering different stochastic behaviours" (different from what)? how is the "differentness" relevant?
- While it's pretty clear to me how to use this immediately in my work, I think anyone who wasn't already aware they wanted exactly this might struggle. Could you provide something like a "practical users guide" for non-domain experts?
 - if the clarity of the plots can be improved, the naming of the stat/metric you're introducing, and improve it's "usability" to the community, I would be happy to upgrade my score. You've done great work and this would bring the paper to the level it deserves.
soundness: 4
strengths: Great paper, wonderfully practical and insightful; I've been looking for something like this for 5+ years! Nice eval on real-world data.
I started writing a thing I would like to you add and then discovered it was already in the paper (long horizon behaviour)
summary: the metric of expected calibration error is introduced and studied as a way to capture fidelity of a learned representation to an underlying stochastic process (rather than a single realization of that process, as with typical metrics like AUC or MSE).
weaknesses: While overall the paper is very clear, some of the captions and explanations of the experiments/insights from them and how they tie to the figures could be improved. 
Some specifics:
- first fig should say what you mean by realization, and F2R and F2SP should be bolded (not ital) to make them easy to find in the text. Observed GT should be explained a bit more, or maybe it would be enough to move the sentence currently after F2R to be the second sentence of the paragraph.
 - Fig 5 is unclear to me. What is the data, what is S-level, why is it "good" that the 20 vs 10 lines are far apart? All of this should be clear from the caption
 - the clarity wanes a bit as the paper goes on, and it's a bit confusing that you call it ECE vs. F2SP vs Statistic-GP. Do these different namings really serve something? It could be a lot more clear if you just have one naming.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: A question arises regarding Figure 1: Can ECE be an effective metric for measuring F2R compared to other available metrics? Figure 1 suggests that the answer may be *no*.

An important indicator that ECE is a reliable measure is its diagonal pattern, showing low scores only when training and test S-Levels align, as illustrated in Figure 4. Could the authors provide theoretical insights to support this indicator?
soundness: 2
strengths: The topic of evaluating DNNs within stochastic complex systems is both intriguing and important.

In the primary evaluations, the author conducted experiments across various settings, including different DNN architectures, comparisons with multiple evaluation metrics, and diverse simulation tasks.

The main text clearly explains the difference between ECE in classical assessment and stochastic process settings.
summary: This paper presents a study evaluating deep neural networks (DNNs) within stochastic complex systems, emphasizing the importance of Expected Calibration Error (ECE) in measuring fidelity to stochastic processes. The findings are validated through multiple experiments and comparisons.
weaknesses: The paper is somewhat difficult to follow. For example, providing a brief introduction to the structure of each section would enhance clarity, particularly in Sections 2 and 3. Additionally, it is difficult to grasp the main messages conveyed by the table in Figure 2(b). Furthermore, in lines 229–240, the macro-level concept is introduced abruptly, which may disrupt the clarity and readability of the main text.

The main findings' practical applicability appears limited. In real-world scenarios, data generally provides only a single observed outcome centered on observable ground truth (line 117). Since the primary evaluation is simulation-based, the controlled stochasticity falls short of capturing real-world complexity. The Statistic-GT is basically derived by normalizing the frequency of target state occurrences across multiple Monte Carlo simulations.

Minors:

M1. The original text for the abbreviation RV is not given.

M2. In Table 1, what about the possibility of recovery in the Host-Pathogen problem?

M3. In line 152, maybe consider using an alternative symbol for Moore neighborhood, instead of $\mathcal{N}$ (normally representing Gaussian distribution).

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 1
questions: L50: Is --> is (lowercase)
Fig1: no need to write the whole name, you can use acronyms because they're already defined in the text, however MSE is not defined at this point.
L88: fidelity to realization --> F2R (it was already defined previously, so you can use the acronym)
L99: the notation of the dimension of the real vector O_t is confusing, what is (R^n)^(H x W), is n = H x W? If so, make that explicit.
Table 1: some rows end with full stop, other don't. Please make it consistent. Either all with or all without.
I find it odd to place Figures in columns as Figure 1 (which has a large top white margin) and Figure 3. I would suggest column figures into one row figure with multiple subfigures as you did with Figure 2. 
L201: Isn't the indicator variable already defined as B_t in L99? Why defining again with different notation?
L298: MSE already defined in text previously, no need to write the whole name again.
L516: ECE already defined in text previously, no need to write the whole name again.
Table 2 and Table 7: highlight the best performing DNNs.
soundness: 3
strengths: The paper makes a significant contribution by introducing the concept of Fidelity to Stochastic Process (F2SP), a novel evaluation criterion specifically designed to assess a DNN's ability to learn the underlying stochastic interactions in complex systems.

The authors provide a rigorous formalization of F2SP within a stochastic framework, establishing clear criteria for its valid measurement. The use of Expected Calibration Error (ECE) as an evaluation metric is well-justified.
summary: This paper presents a study on evaluating deep neural networks designed to forecast the evolution of stochastic complex systems. The authors identify a gap in traditional evaluation methods—such as threshold-based classification metrics and error-based scoring rules—which focus on a model's ability to replicate observed ground truth but fail to assess how well the model has learned the underlying stochastic process. To address this issue, they introduce a new property called Fidelity to Stochastic Process, representing the DNN's ability to predict the statistical ground truth of the stochastic process.

The paper proposes using the Expected Calibration Error (ECE) as an evaluation metric that satisfies the necessary conditions for assessing fidelity to statistical ground truth. This work underscores the importance of capturing the underlying stochastic processes in deep neural networks  evaluations for complex systems.
weaknesses: I found it hard to read the paper because there was a lack of consistency in the acronyms, the authors would redefine them in several parts of the text again and again. I addressed my comments on text in the questions section. 

In the tables, the best neural networks based on each criterion are not highlighted, which makes it difficult to the reader to infer and correlate the arguments in the text. I addressed my comments on text in the questions section. 

The focus of the paper is primarily on binary or discrete prediction tasks, leaving out regression tasks where the definition of calibration is more complex. While the authors acknowledge this and suggest it as an area for future work, the current scope limits the immediate applicability of the findings to a broader range of problems involving continuous outcomes.

Additionally, the use of the NDWS dataset, which is restricted to next-day predictions, prevents the assessment of ECE over longer time horizons, which are common in many complex systems. Could you elaborate on how future work might address this limitation? 

The paper highlights the lack of open-source complex system datasets as a barrier to broader validation. Are there any ongoing initiatives or plans to develop, collect, or standardize such datasets?

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: As mentioned in the "Weaknesses" part.
soundness: 3
strengths: The paper offers a new perspective on evaluating DNNs by considering DNNs as stochastic processes and uses a widely used criteria in Bayesian Deep Learning application to assess the fidelity to stochastic process. This work clearly explains the Expected Calibration Error is used to assess DNN modes in three synthetic cases and one real world case.
summary: This work offers a new perspective on evaluating DNNs in stochastic complex systems by emphasizing  the importance of capturing underlying the stochastic process. Traditional evaluation methods assess the DNN’s ability to replicate the observed ground truth but fail to measure the DNN’s learning of the underlying stochastic process. This paper proposes a new property called Fidelity to Stochastic Process, representing the DNN’s ability to predict the ground truth of the stochastic process, and introduces an evaluation metric that exclusively assesses fidelity to  the ground truth of the stochastic process. The Expected Calibration Error is used to evaluate the fidelity to ground truth of statistic process. Empirical experiments on synthetic datasets (including wildfire, host-pathogen, and stock market models) and real-world wildfire data are used to show the measurement of fidelity to stochastic process by Expected Calibration Error.
weaknesses: This paper is well organized and well written, several minor issues should be addressed: (1) The explaination of figures is not sufficient, e.g., in Figure 2 (1), the label for x-axis is not specified (I guess it is time?), either add a label or explain it in the captions. Same problems also exist in Figure 4. (2) This work examines ECE on three synthetic environments (forest fire, host-pathogen and stock market models) and a real world wildfire spread dataset. I can tell that these datasets are all multivariate either for classification or regression. Maybe due to the limit of pages, the authors didn't include the experiments on images. I suggest the authors add some discussions or comments in the paper.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: I'm curious about how the author evaluates ECE at time $t$ based on Statistic-GT $P_{t}$. Do we have to simulate it again from $t=0$ for $N$ times or we can sample states from $t-1$ and go forward $N$ times (the system is Markov)? Can we still apply ECE on Statistic-GT when the system is not Markov?
soundness: 2
strengths: 1. Evaluating model fidelity on the stochastic system is significant and has wide applications.
2. The paper is well-motivated and both the dataset and experiments are thorough.
summary: This paper introduces a novel stochasticity-compatible evaluation strategy for assessing existing models in the context of complex systems. The author justifies the Expected Calibration Error (ECE) as suitable for assessing the model fidelity of stochastic systems through both simulation environments and real-world data.
weaknesses: 1. Although the author attempts to explain the difference between their work and ECE in deep learning in Lines 282-288, it appears to me this work is still a direct application of using ECE to evaluate the model performance on a stochastic system. The author is encouraged to discuss more in-depth about the distinction between ECE in the proposed method (stochasticity comes from evolving in the environment, aka, Statistic-GT) and ECE in previous works (stochasticity comes from the output distribution).
2. In Lines 243-244, the author claims that Statistic-GT is more stable than classification-based metrics, but I could not find any evidence related to calculating ECE on Statistic-GT is less sensitive to the system variance than MSE. Is there any theoretical support for using ECE over MSE on stochastic systems with different noise levels and could the author clarify it a bit more?

Meta-review: The reviewers unanimously recommend acceptance (8-6-6-8-6). The paper presents a significant contribution for the evaluation of neural networks designed to forecast the evolution of stochastic complex systems. Reviewers recognize the importance of the work and the quality of the results. The author-reviewer discussion has been constructive and has led to a number of improvements to the paper, in particular regarding its presentation. The reviewers have raised some concerns about the practical applicability of the findings (e.g., the focus on binary prediction tasks, the limited horizon in the experiments), but the authors have provided convincing arguments arguing for the relevance of their work nonetheless. No major concerns have been raised by the reviewers. For these reasons, I recommend acceptance. I encourage the authors to implement the changes discussed with the reviewers in the final version of the paper.

Decision: Accept (Poster)