Rare event modeling with self-regularized normalizing flows: what can we learn from a single failure?
Authors: Charles Dawson
Van Tran
Max Z. Li
Chuchu Fan
Venue: ICLR 2025 Poster (2025)
Keywords: rare event modeling
normalizing flows
Bayesian inverse problems
Primary area: applications to robotics, autonomy, planning
TL;DR: Modeling rare events like failures in safety-critical systems requires learning from a very limited amount of data, which is challenging for existing inference tools. We develop a self-regularized framework for severely data-constrained problems.

Abstract: Increased deployment of autonomous systems in fields like transportation and robotics have seen a corresponding increase in safety-critical failures. These failures can be difficult to model and debug due to the relative lack of data: compared to tens of thousands of examples from normal operations, we may have only seconds of data leading up to the failure. This scarcity makes it challenging to train generative models of rare failure events, as existing methods risk either overfitting to noise in the limited failure dataset or underfitting due to an overly strong prior. We address this challenge with CalNF, or calibrated normalizing flows, a self-regularized framework for posterior learning from limited data. CalNF achieves state-of-the-art performance on data-limited failure modeling and inverse problems and enables a first-of-a-kind case study into the root causes of the 2022 Southwest Airlines scheduling crisis.

Published as a conference paper at ICLR 2025
RARE EVENT MODELING WITH SELF-REGULARIZER NORMALIZING FLOWS: WHAT CAN WE LEARN FROM A SINGLE FAILURE?
Charles Dawson
Van Tran
Department of Aeronautics and Astronautics Massachusetts Institute Technology cbd@mit. eduu
Department of Applied Mathematics
Harvard University
vantrane college www.aar.aae.cc.. edu
Max Z. Li
Chuchu Fan
Department of Aerospace Engineering University of Michigan maxzli merzziimmiii ch. edu
Department of Aeronautics and Astronautics
Massachusetts Institute of Technology
chuchu@mit.
ABSTRACT
Increased deployment of autonomous systems in fields like transportation and robotics has led to a corresponding increase in safety-criicaal failures. These failures are difficult model and debug due the relative lack of data: while normal operations provide tens of thousands of examples, we may have only seconds of data leading up to the failure. This scarcity makes it challenging to train generative models of rare failure events, existing methods risk either overfitting to noise in the limited failure dataset or underfitting due to an overly strong prior. We address this challenge with CALNF, calibrated normalizing flows, self- regularized framework for posterior learning from limited data. CALNF achieves state-of-the-art performance data-limited failure modeling problems and enables a first-of-a-kind case study of the 2022 Southwest Airlines scheduling crisis.
INTRODUCTION
When complex systems fail, the first step towards recovery understanding the factors that lead to failure. Modeling failures and other events is challenging because the limited amount of available data. While much work has been done on preemptive failure prediction in simulation (Corso et al.)
2022 O' Kelly al. 2018 Sinha et al. 2020 Dawson & Fan 2023 Delecki al. 2023 Zhong
et al.
2023). and online failure detection (Keipour et al. 2022 Hendrycks et al. 2018 Gudovskiy
al.
2022 Kang al. 2022; Najari
| 2002 Garg et (023), relatively little work has been
done on post-event failure modeling from observational data.
In failure modeling problems, we seek infer the hidden factors contributing to an observed failure by learning a posterior distribution over those factors. When observational data are plentiful, deep generative models can be powerful tools for solving these problems, but these methods struggle when only few examples are available. To solve rare event modeling problems that arise in domains like robotics and networked cyberphysical systems, we need new ways training generative models in data-constrained settings.
Formally, we frame rare event modeling as data-constrained Bayesian inverse problem where we aim to infer the distribution of latent variables from noisy observations of stochastic process I ~ (a y). where 0 are unknown process parameters and are known context variables, all (1) (6)
real-valued (tturt 2010). Given nominal observations Do {=) .30 1i=1 and much smaller set of target observations D. = {-(( LWWN Nez where N₂ < No. we aim to learn an approximation of the posterior distribution
(x(2) R (=1(x)) )
(1)

---

Published as a conference paper at ICLR 2025
Note that this problem is distinct from the rare event simulation problem considered in O' Kelly et al. (2018); Sinha et al. ) Gao et al. (2023). , and Dawson & Fan (2023). While we focus on learning the distribution of failures from small, fixed number of weal-world data points, these methods assume
the ability to sample and label an arbitrary numher of new failure examples in simulation.
In data-conttr settings, common approach to solving inverse and few-shot generative modeling problems is to use the nominal observations to train a deep model of the prior distribution, then use this prior regularize To (e.g. by penalizing the divergence between go and the learned prior, Asim et al. (2020); Liu et al. Abdolllzaeeh et al. (2023); Zhang et al. (2019); Ojha et al. (2021); Higgins et al. (2016)). Unfortunately, it can be difficult to specify the appropriate amount of regularization a priori, particularly when the distribution of 2 differs between the nominal and target datasets (as if often the case in failure modeling problems).
In this paper, we address this challenge by developing CALNF, or calibrated normalizing flows. To make full use of available data, CALNF amortizes inference over both the nominal and target datasets, learning a shared representation for both posteriors. To prevent overfitting. CALNE first learns a low-dimensional embedding for a family of candidate posteriors then searches over this low-dimensional space to find an optimal representation of the target posterior. Our method achieves state-of-the thrttrrrttttttttteeeeeeeea-- performance on a range of data-constrained inference benchmarks, including several with wolldw data from autonomous systems. We apply our method to post-mortem analysis of the 2022 Southwest Airlines scheduling crisis, which stranded more than 2 million passengers and led more than $750 million in financial losses (Rose) 2023); our first-of-a-kin analysis suggests mechanism by which failure propagated through the Southwest network.
2 BACKGROUND
Because the posterior in Eq. I1 is typically intractable to evaluate exactly. inverse problems are typically solved approximately, often using variational inference (Stuart 2010). These methods approximate the true posterior by maximizing the evidence lower bound (ELBO, Kingma & Welling (2013)), often using deep non- non-parametric representations like normalizing flows for do (Tabak & Vanden- 2010 Rezende & Mohamed 2015).
L(),0,D)
E E [o =;, log (=(=)] (2,0)ED (s))
particular challenge in constraind settings is that deep models tend to overfit the particu- lar target samples used for training, as shown in Fig. [p. As the following result illustrates. the representational power of these models makes them highly sensitive when trained on small datasets.
Lemma 1. Let D = {2 be sparse dataset with distance O ((IN)-1/(+11) between points (the precise limit is given in the appendix), and let (z) be a model capable of representing any L-Lipschitz probability density. If (DD are the parameters of the maximum likelihood estimator given DD then the optimal solutions trained on datasets differing by one point DD = DD U {2(1) and
D2 = -DU (20)) will differ by Wasserstein distance Wz (fo(D.) ):G(D.)) = || -(
'We provide these results for maximum likelihood estimation, but similar results can be shown for maximum a posteriori or maximum ELBO contexts.
(6) GT
(b) Inbalannd distoer:
(c) KL regularted, P-0.01
KL regulutted, $-19
(c) CaINF (ouns)
Figure 1: Inference in data-constrained environments. (a) The ground truth distribution. (b) An imbalanced dataset. (c) When the regularization strength / is too small, deep models overfit to noise in the target dataset. (d) When 3 is too large, the learned distribution underfits and struggles to distinguish between nominal and target distributions. (c) Our method learns a more accurate reconstruction of the target distribution using syperparameteerinsensitt self-regularizztion.
2

---

Published as a conference paper at ICLR 2025
The proof relies on the fact that the optimal L-Lipschitz maximum likelihood estimator is go(D) (2)
Eg(tle 8(2 - 2()), where 6(z) = max(0, a L| L||=|]) is an L-Lipschitz approximation of (22 and constant given in the appendix. Lemma T Jimplies that small changes in the training data can lead to large changes in the learned distribution; as N becomes small, this sensitivity increases.
Two common strategies for reducing this sensitivity in non-sparse settings are bootstrapping (Efron 1992), which trains an ensemble of models on random subsamples of the target data, and prior regularization [Asim et al. 22000), which uses model trained on nominal data regularize the target posterior. Unfortunately, both of these methods are difficult to apply to data-constrained problems.
Bootstrapping, or ensemble, methods are theoretically well-motivated (Bfronn 1992 Breiman (996) but do not fully avoid data sensitivity issues when applied to deep models (Nixon el al. 2020). In
fact, it can be shown that bootstrapping has no effect on data sensitivity in highly data-constraing environments.
Lemma Consider the setting from Lemmul 7 Let Dj be K subsampled datasets created by sampling N points with replacement from the original D. Let ((D)) be parameters of the maximum likelihood estimator for each D; and define the ensemble model Gensemble (2) =E (x) / K. As K - oc, the ensemble model recovers the solution of the non- non-bootstrapped problem; i.e. Gensemble (2) - (o(D)
The second strategy, prior regularization, is common in the few-shot learning literature, particularly for image generation tasks (Asim et al.) 2020 Higgins al. [2016] Abdollahzadeh et al., 2023).
These methods use large dataset to learn the nominal posterior distribution, then use that nominal posterior to regularize the target posterior:
Φ = argmax (
[((), 0, Do)
Φ = argmax, ((0, 0, D.) - -BD (10o 9o)
This penalty allows the large number of nominal samples to regularize the distribution learned from the target data, but there are two issues with this approach. First, is difficult to choose an appropriate penalty strength 3 a priori, as illustrated using the toy example in Fig. | If 33 is too small, the model will overfit to the target data (Fig. [ but if /3 is too large then the model will underfit the target in favor of learning the nominal distribution (Fig. It) Moreover, even if we were able to select an optimal B, many practical failure modeling problems involve a large shift between the nominal and target distributions, in which case regularizing between these distributions may not be appropriate.
In order to effectively train deep generative models of rare failure events, we will need to address two key questions. First, building upon prior regularization: can we adapt the amount of regularization based on the available data rather than specifying a regularization penalty a priori? Second, building upon bootstrapping: can share information between model components to learn robustly in data-constrained contexts?
2.1 RELATED WORK
number of recent works have explored various forms of prior regularization, mostly in the context of image generation tasks (see AAboolaahzadeh et al. (2023) for survey). AAsi et al. (2020) use a deep prior model trained on large image datasets regularize single-shot maximum likelihood estimation for image denoising. Liu et al.] (2023) combine prior regularization with graduated optimization for image reconstruction, gradually increasing the regularization strength over multiple inference rounds. Ojha et al. (2021) pre-train a model on large open image dataset, then fine-tune to smaller image datasets with a loss function that preserves relative difference and similarity across domains, Kong et al. (2022) focus generative adversarial networks (GANs) and regularize the generator's latent space by sampling points in between sparse target data. Other works rely on data augmentation, either using hand-designed heuristics Wang al. (2020) which must be re-derived for each new problem domain, or pre-trained generative models Tran et al. (2017); Zheng et al. (2023). which require a minimum amount of data from similar examples.
METHOD: CALIBRATED NORMALIZING FLOWS
To address the challenges from Section we propose novel framework for rare event modeling: calibrated normalizing flows, or CALNF, This method involves two steps, illustrated in Fig. First, learn low-dimensional embedding for a family of probability distributions that includes both the
3

---

Published as a conference paper at ICLR 2025
nominal posterior and several candidate target posteriors. We then search over this low-dimensional space to find the optimal representation of the target posterior. To ensure that the embedding space is well behaved, we apply regularization between candidate target posteriors. This self-reg self-regulatiization, combined with implicit regularization from our use of a single, shared representation for both the target and nominal posteriors, allows CALNF efficiently learn the posterior without overfitting.
oood
B.nay
K.RDA
Base Distribution
ceco
20
Flow Network
[eacse]
Calibration Label
poooa
Figure 2: (Left) CaINF architecture: A normalizing flow is trained on random subsamples of the target data and the full nominal dataset, using one-hot labels to identify different subsamples (o) and the zero vector to identify the nominal data (o). The model is calibrated by freezing the model parameters and optimizing the label on the entire target dataset (*)) (Right) Target candidates: The nominal posterior go (z( 0) (blue) and the family of candidate distributions for the target posterior (2; A1,) shown for varying values of the calibration label.
The first step in our framework is to learn embedding for family of candidate probability distributions. To do this, we randomly sample subsets of the target data D( .... D(K) then train single conditional normalizing flow (2(2; c) to learn mapping from a -ineennnnio label to the posterior distribution conditioned on each of these subsets. We identify the posterior for each target subset using one-hot labels 1, and use the zero label 0x to identify the nominal posterior; i.e.:
4 (2; 0x) x p(z[Do),
go(2; 1,) p(= i L,.,.., K
Once posteriors have been learned for each of these subsets, we calibrate the model by holding the model weights @ constant and searching for an optimal label e* such that (o(2; c*) ~ p(=[Dr).
This training process shown in more detail in Algorithm T This algorithm modifies the standard variational inference training process in two ways: by training on multiple random subsets of the target data, and by interleaving model updates and label calibration. To begin, we split the target training data into K random subsets with one-hot labels and train the model to learn the posterior for each subset. Each subset Dj is created by independently drawing N, samples from D, with replacement. We denote the empirical ELBO given dataset D
C
1
Σ (x.Y)ED (2:0)
E
[log p(x, 2; log (o(2;))]
(4)
D
The model trained with three objectives: to maximize the ELBO on each target subset (with one-hot labels), the nominal data (with a zero label), and the full target dataset (using the calibrated label c*). In addition, to improve the conditioning of the embedding space, we add a regularization term on the divergence between the posterior distributions learned for each target subset, yielding the loss:
K
K
L(),e) = K c (0,1.., L (o. 0k. Do) - C(o.c,D, -
1
Σ
> i
DEL (90(: (90((11). (:0::111) 1,))
(5)
This additional regularization term reflects the fact that the target subsamples D. are identically distributed, so the divergence between the candidate posteriors learned for each subsample should be small, and we show empirically that CALNF performs well for a range of 3. The mixture label c* is 1/K] and updated to maximize c* arg max, L(ő, c, D,). In practice, this equivalent to jointly optimizing (ø, c*) = arg min L(d, c²), but we show these optimization steps separately in Alg. [ to emphasize the different objectives.
initialized at [1/K,...
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Calibrated Normalizing Flows
Input: Nominal data Do, target data D,, step size y, number of target subsamples K. self-regularization / Output: Model parameters @ and calibrated label c*
for 1..... K do
D. + -leeement random subsample of D end for Initialize o, e*
while & not converged do
Update model 4- Φ v. [T,
Update calibration e* f- c² - (olcall0) end while
3.1 THEORETICAL ANALYSIS
The secondary optimization of c* is one of the main differences between CALNF and traditional bootstrapped ensembles (which combine the target candidates a mixture model rather than opti- mizing in lower-dimensional label space). In this section, provide theoretical motivation for this decision, showing that learning a mapping from labels e to candidate distributions implicilly regularizes the learned target posterior.
In particular, consider the Wasserstein metric W2(P1, P2) inf, E2 P2 || - 22] 1/2 where y is a coupling probability distributions P1 and P2 The result show that CALNF, in addition to explicitly regularizing the divergence between target candidates, also provides implicit regularization of the W2 metric between the learned nominal and calibrated target posteriors.
Theorem 1. If the flow mup fø(2, e) is L-Lipschitz in the second argument, then the Wasserstein distance between the nominal and target posteriors is bounded; W2 (1o(2, (x), ga(2)))))))) c²), S L||e"]
A proof is included in the appendix, along with L for common normalizing flow architectures.
4 EXPERIMENTS
4.1 BENCHMARK PROBLEMS
This section describes the benchmark problems used in experiments: the first is newly developed for our study, but the rest are previously-publishe 2022). We also include the toy 2D) problem in Fig. (No 103, N₁ = 20). More details provided in the appendix, and we provide open-source code and data.
benchmarks (Keipour et al.] 2021 Deng et al.)
Air traffic disruptions We develop a stochastic queuing model of the Southwest Airlines network using publicly available arrival and departure data, based on (Pyrgiotis et al. 2013). The latent variables represent travel times, runway delays, and overnight aircraft reserves. The context includes flight schedules, and observations include actual departure and arrival times. Nominal and failure data from between Dec. through Dec. 20 and Dec. 21 through Dec. 30, respectively, are used for the four busiest airports. The four-airport sub-network has 24 latent variables. We train using No = and N. = 4 and evaluate on held-out failure data points (each data point is single day with between 88-102 flights).
Aerial vehicle control We consider failure detection benchmark for unmanned aerial vehicles (UAVs) using the ALFA dataset (Keipour et al. 202T). This dataset includes real-world data from UAV during normal flight and during failures with various deactivated control surfaces. The latent variable a parameterizes the nonlinear attitude dynamics and has 21 dimensions, y includes the current and commanded states, and x is the next state. We train on 10 nominal trajectories (No = 2235) and 1 failure trajectory (N₂ = 58) and evaluate held-out failure trajectory (69 points).
Geophysical imaging Seismic waveform inversion (SWI) is a well-known geophysics problem used benchmark for inference and physics-informed learning (Gouveia & Scales 1998 Deng
5

---

Published as a conference paper at ICLR 2025
Table 1: ELBO (nats/dim) on held-out anomaly data on benchmark problems. 2D and SWI use unseen synthetic data for the test set; all other cases withhold half of the target data for testing. Mean and standard deviation across four seeds are reported. 'scaled by x10-3
2D nats/dim f
SWI nats/dim²
UAV nats/dim
ATC nats/dim' t
KL-regularized (3 0.01)
-3.22
44.72
2.87
-0.26
Kl.rregularized 0.1)
10.03
44.7 0.339
3.26
-2.23
-l-regularized = 1.0)
-1.04
44.3
3.02:
-2.23
W2-regularized (3 = (0.11)
-4.5840.18
36.7:3.03
-1.7524.53
-5.90=2.54
Wa-regularized (3 0.1)
-2.9540.14
36.7+3.02
-1.5444.31
-5.80=2.45
Wa-regularized (3 -.0)
-1.67.0.05
36.712.94
2.13315.79
-6.5724.09
Ensemble
-0.440..4
4..42
6.65+4..s
-2.23
CaINF (ours)
+0.90
46.4
7.55
-2.1140.13
et al. 2022 Zhang et al. 2016). SWI seeks to infer the properties of the Earth's 's subsurface using equation. This model uses latent
seismic measurements, which simulated using the elastic variables for subsurface density, context for the source signal, and observations æ for the seismic measurements (Richardson) (023). The latent space has 100 dimensions (a 10 10 grid). We train on No- 100 and N₂ = 4 and evaluate on 500 synthetic samples.
4.2 BASELINES AND METRICS
Our main claim that our CALNF framework is an effective way to learn the posterior when a small number of target data points are available. The most relevant comparisons are to the prior regularization and bootstrapping ensemble methods discussed Section 2 particular, we compare against three baselines from previously-publiseed literature: prior regularization using KL divergence (based on Asim et al. (2020)) prior regularization using Wasserstein divergence (based on FFinlyy et al.) (2020) and Onken et al. (2021)), ) and ensemble method based mixture of normalizing flows (adapted from the generative ensembles proposed in Choi et al.] (2019)). The two prior regularization baselines are sensitive the hyperparameter B, we report results for a range
[0.01, 1.0]. CALNF uses K and 3 = 1.0 for all problems, and Figs. 1 and [2 in the appendix shows the sensitivity of our method varying K and B. We use K 5 components for the ensemble baseline.
Since the large amount of nominal data makes it easy to fit the nominal distribution, we compare primarily on the basis of the evidence lower bound L computed on held-out target data, reporting the mean and standard deviation over four random seeds. When useful, we also provide visual comparisons of the posterior distributions learned using different methods.
4.3 RESULTS& & DISCUSSION
Our main empirical results shown in Table Our method achieves better performance held-out target data than baselines on all problems. Of course, CALNF's improved performance comes at the cost of increased training time, requiring K additional likelihood evaluations per step relative the KL- and W2 Wz-regularized methods (and the same number of evaluations as the ensemble method). To qualitatively understand the difference in performance between these methods, Fig. compares the learned target posteriors on the SWI problem (which lends itself to easy visualization) with the ground truth in 3.. We see that the KL- and W Wg-regularized and ensemble methods (Fig. 33-d) do not infer the correct density profile from the target data, while only our method (Fig. 3e) is able to infer the correct shape. Fig. shows CAI.NF's prediction the UAV problem on a held-out failure trajectory after training only results suggests that our method is able to appropriately balance the information gained from the nominal distribution with the limited number of target data points.
other failure trajectory (and 10 nominal trajectories). These
We also provide the results of an ablation study in Table in the supplementary material, comparing the ELBO achieved when omit the calibration step (using constant c), omit the nominal data, and remove the subsampling step. These results indicate that most the performance improvement from CALNF is due to training random subsamples of the target data. We observe that in cases

---

Published as a conference paper at ICLR 2025
Nominal
Target
(a) menn
(b) KL reg.
(c) Wer.g
(d) Ensemble
(c) CaINF (ours
Figure 3: Seismic waveform inversion. (a) The ground truth nominal and target density profiles.
(b-d) The posteriors fit using KI. and W2 regularization and CALNF (ours is the only method able to correctly infer the target density profile).
with plentiful nominal data (like the UAV problem), training on zero-labeled nominal data also substantially boosts performance. The appendix includes further results on training time and the sensitivity of CALNF to varying (Fig. 11, 3 (Fig. 12). and target dataset size N, (Fig. 12
4.4 USING CALNF FOR ANOMALY DETECTION
Although main focus in this paper is modeling failure events, using our method to detect previously-unsedd failures. To use CALNF for anomaly detection, we train the normalizing flow and calibration label described above, then the ELBO L(Ø, c², {a, y)) as the score function to classify a novel datapoint {>r, y). Table 2 shows the area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUCPR) for this CALNF-based anomaly detector on the UAV and SWI problems, compared with the supervised anomaly detector proposed (Gudovskiy et al., 2022 Kang al. 2022 Rudolph al. (20TT) and with hand-tuned KL- and Wa-regularized variants. We were not able to test the ATC problem in this setting due lack data, and the 2D toy example is too simple to be informative, as all methods achieve near-perfect classification. Although CALNF is designed for posterior learning rather than anomaly detection, we find that it achieves more consistent results on this downstream task than existing methods, likely due to CALNF's resistance to overfitting.
apply the posteriors learned
Table 2:
Anomaly detection performance on held-out data, re- porting mean and standard deviation across four seeds.
0.5
AUROC
AUCPR
SWT
UAV
SWT
UAV
033
NF-AD
0.744.0.03
0.65 00.17
0.770
0.56-0.23
NF-ADKL
0.74
0.74
0.77
0. 0.70+a.11
NF-ADw;
0.65
0.54
0.63
0.41 -0.08
12.5
175
Ensemble
0.75
0.5.0.0
0.78
0.36.
CaINF
0.7920.02
0.70
0.83
0.66+1.04
Figure 4: Single-shot UAV failure dynamics predicted by CALNF.
4.5 CALNF FOR FEW-SHOT INFERENCE ON IMAGE DATA
Although image modeling is not the focus this work, for completeness we also include results applying CALNF few-shot image generation. To use CALNF for this task, replace the underlying normalizing flow with a conditional Glow architecture (a type of normalizing flow specialized for image modeling Kingma & Dhariwal (2018), using the calibration label as the
7

---

Published as a conference paper at ICLR 2025
conditioning input. We adapt the MNIST (LeCun al. 2010) and CIFAR-10 10 (Krizhevsky 2009 datasets to create a nominal dataset with all training examples from a single class and a target dataset with 64 examples from second class. In this case, there is underlying stochastic process and no context variables (i.e. we directly observe the images I ~ po(=]2) 8(x - 2)), and the empirical ELBO 4 reduces to the average negative log likelihood of the training data. As result, minimizing the loss learns an approximation of the training image distribution.
Each method is trained on all examples from the nominal class and the limited set examples from the target class, and we report the negative log-likelihood on held-out test data, then we measure the negative log-likelihood on held-out images from the target class to test generalization beyond the limited training examples. Table 3 and Fig. Show the results of this experiment. We find that CAINF able to generalize hetter than either a standard or ensemble model, as measured by a higher log likelihood on the held-out data
Table 3: Log-likelinne Log-likelihood (bits/dim) on held-out im- ages, reporting mean and standard deviation across four seeds. Higher is better.
Glow (vanilla)
Glow (ensemble)
-llow CalNP
MNIST
CIFAR-10
Glow (vanilla)
5.77
148.6
5.
Glow (ensemble)
5.77
85.91
5.4
Glow + CALNE
5.99
+3.3
Vanilla
9
9
8
9
9
44
Swp
9
9
9
9
9
Figure 6: Log-likelihood (bits/dim) on MNIST validation set; CALNF reduces over- fitting to sparse examples.
Figure 5: Samples with CALNF (bottom) are more diverse than those without (top).
5 CASE STUDY: 2022 SOUTHWEST AIRLINES SCHEDULING CRISIS
In this section, we apply CALNF to a post-mortem analysis of the 2022 Southwest Airlines scheduling crisis. Between December 21 and December 3hth 2022, a series of cascading delays and cancella- tions severely disrupted the Southwest network, starting in Denver and spreading across the United States. The disruption occurred in roughly two stages, as shown in Fig. 1 in the appendix. In the first stage, from 12/21 to 12/24, weather and operational difficulties caused cancellations to increase from 5% baseline to over 50% of scheduled flights. In the second phase between 12/25 and 12/29, Southwest flight dispatchers started preemptively canceling flights and ferrying crew between airports to reset the network, canceling up to 77% of scheduled flights before returning to noam-normal operations on 12/30. Southwest ultimately canceled more than 16,000 flights, affecting more than 2 million passengers. and the airline lost substantial revenue and later paid $140 million penalty imposed by the US Department of Transportation (28% of its 2023 net income; (Rose 2023 )
This incident has been the subject of extensive investigation, with report from Southwest Air- lines (Southwest Airlines 2023), testimony before the US Senate from the Southwest Airlines Pilots Association (SWAPA; (Murray (023)), ) and press coverage (Rose) 2023 Cramer & Levenson 2022). These sources propose number of hypotheses on the root cause of the 2022 incident. While there is broad agreement that winter weather was a major factor, sources differ on the role of other factors; e.g. the SWAPA report emphasizes poor crew management, while press coverage emphasizes the point-to-point nature of the Southwest network.
Given this context, we have two goals for our case study. First, we are interested in identifying changes in the network state that coincided with the disruption, and how those disrupted parameters aim to produce generative model of the
compare to the nominal state of the network. Second, disrupted network conditions for use as a tool for network design and analysis (e.g. as a simulation environment for stress-testing future scheduling and recovery policies might).
8

---

Published as a conference paper at ICLR 2025
30-
2330
3330
10
0.1
0.3
0.3
0.3
servic
Figure 7: The posterior distribution of departure service times at DEN, MDW, DAL, and LAS. Service times increased only at airports that saw severe weather (DEN, MDW, and DAL).
5.1 IMPLEMENTATION
We focus on the first four days of the scheduling crisis, prior to the wave of manual interventions aimed at resetting the network. We conduct our analysis at two spatial resolutions, considering subnetworks of the and 10 busiest airports the Southwest network, respectively. More details on model are included in the appendix, along with key for relevant airport codes in Table 6
5.2 RESULTS
Localized delays due winter weather. Our first observation confirms a common explanation for the disruption: that localized delays at airports across the US coincided with winter weather. Fig. 7 shows CALNF's posterior estimates of nominal and disrupted service times (a proxy for taxi, deicing. and ATC delays) at the four busiest airports. Of these four, only those that experienced severe cold temperatures (DEN, MDW, and DAL) saw an increase in average service time, while there was no significant increase at LAS, which did not have severe weather. This result agrees with official accounts that identify winter weather and a lack of deicing equipment critical airports like DEN as contributing factors (Southwest Airlines) 2023] Cramer & Levenson (022). However, the more important question how these localized service delays cascaded into the nationwide disruption.
Cascading f'allures due to aireraft flow interruption. Our main finding comes from modeling the movement of aircraft within the network. The number of aircraft starting the day at each airport provides an important measure of robustness, since if there are insufficient aircraft to meet demand, then departing flights must be delayed or canceled. Aireraft deficits can also cascade through the network, as down-stream airports are deprived of the aircraft needed to serve scheduled departures. Because aircraft distribution data are not publicly available, we must use our method infer
0.4
Relative aircraft surplus/defict
0.0
Dak
-0.4
PHOX
-0.6
LAS
2022-12-21
2022-12-22
2022-12-25
2022-12-24
Figure 8: CALNF's inferred posterior estimates of the distribution of Southwest aircraft at the start of the first four days of the disruption, normalized by the number scheduled departures at each airport; positive/negative indicates more/fewer aircraft than in the nominal case, respectively. CALNF suggests that LAS, DAL, and PHX accumulated a large aircraft deficit over the course of the disruption.

---

Published as a conference paper at ICLR 2025
Fig. shows results from using CALNF to infer the distribution of aircraft reserves in the top- 10 network over each of the first four days of the disruption. CALNF finds that there was no detectable deviation from the nominal aircraft distribution on the first day of the disruption, but infer steadily increasing deficit at LAS, DAL. and PHX over the following three days. The fact that the aireraft deficit these airports continued worsen may have been a factor in Southwest's decision to 'hard reset" the network by ferrying empty planes between airports.
Beyond inferring these hidden parameters in the Southwest network, CALNF's results also suggest possible causal mechanism by which aircraft deficits at LAS, DAL, and PHX propagated to the rest of the network. Although the Southwest network famously operated in a point-to-poin manner, aircraft typically visit multiple airports in sequence on a given day; and disruptions at any of these intermediate destinations can lead to "missing aircraft" at downstream nodes. For example, although LAS and PHX did not experience severe weather during this period, nearly 50% of aircraft ultimately bound for LAS or PHX pass through either DEN or MDW (which did see weather-related delays). Our analysis in Fig. 8 suggests that trends in aircraft reserves at key airports like LAS, PHX, and DAL might be valuable early warning signs for detecting future disruptions.
Generative modeling Once we have learned the nominal and disruption posteriors for the South- west network, we can use these as generative models for stress-testing proposed modifications to the Southwest network scheduling system. In future work, we hope to explore how these generative models can be used to design more resilient schedule recovery algorithms.
6 CONCLUSION
In this paper. we propose a novel algorithm for rare-event modeling, developing a data-constrained posterior inference tool that uses subsampling and calibration strategy to avoid overfitting to sparse data. We apply our algorithm to failure analysis and inverse problems, achieving competitive performance range of benchmarks with both simulated and real data. We also apply algorithm to a real-world failure modeling problem, providing new insight into the factors behind the 2022 Southwest Airlines scheduling crisis.
Limitations & future work The primary limitation of our work is that training CALNF on randomly sampled subsets of the target dataset incurs an additional training cost (as shown in Table in the appendix). Although there is no inference-time penalty, CALNF requires K + 1 evaluations of the joint likelihood p(z,2; y) per training step (one for each of K subsamples and once for updating c²), compared to K evaluations for the ensemble and one evaluation for the KI.- and Wa-regularized methods. Our method also requires one additional hyperparameter than the baselines, which adds to implementation complexity, and it is more difficult to train in parallel than the ensemble model. However, even though CALNF is slower to train, achieves results in the low-data regime that are not possible using faster methods. For example, Fig. 3 shows how our method solves a seismic imaging problem that none of the competing baselines can solve, and CALNF yields better, more consistent performance downstream anomaly classification tasks. We would also like emphasize that there is no additional cost for for our method at inference time, other than the negligible cost of passing the calibrated label c" as additional input the normalizing flow.
second limitation is that our method does not provide an estimate of the risk of failure. Estimating the probability of failure is challenging due to the size of the dataset, but we hope that future work will close this gap; e.g. providing theoretical bounds using large deviation theory (Dembo & Zeitouni 2010). A final limitation is that we implicitly assume that failure examples share some structure with nominal ones, and so learning shared representation for both cases (as CALNF does) is helpful. If the failure examples are drawn from radically different distribution than nominal data, the implicit regularization from this shared representation (discussed in Theorem d may not be useful.
Broader impact This paper aims to provide tools to understand the causes of past failures and prevent future incidents. We hope that our work will help enable more comprehensive data-driven approach to safety analysis for complex systems, including cyberphysical systems and complex infrastructural networks. There is some potential for negative impact (e.g. a bad actor attempting to infer the properties of a safety-critica system to prepare an adversarial attack). but we believe that the potential benefits for designing more robust systems outweigh these concerns.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We include source code for all experiments in a zip file in our supplementary materials, including README files with instructions for installing all required dependencies and scripts for recreating all experimental results reported in this paper, including all hyperparameters and random seeds used.
We also provide an open- implementation of our algorithm and air traffic simulator at https: 7/gi thub. com httpo//dwwwoo////aaaaa
ACKNOWLEDGEMENTS
This work was partly supported by the National Aeronautics and Space Administration (NASA) ULI grant 80NSSC2200070, Air Force Office of Scientific Research (AFOSR) grant FA9550-23-1-0099 and the MIT-DSTA program. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the sponsors.
REFERENCES
Zuko (open-source library). The Probabilists, January 2024.
Milad Abdollabzadeh, Touba Malekzadeh, Christopher T. H. Teo, Keshigeyan Chandrasegaran, Guimeng Liu, and Ngai-Man Cheung. Survey on Generative Modeling with Limited Data, Few Shots, and Zero Shot, July 2023.
Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: Mitigating representation error and dataset bias. In Proceedings of the 37th International Conference on Machine Learning, pp. 399-409 PMLR, November 2020.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Joern-Henrik Jacobsen. Invertible Residual Networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 573-582. PMLR, May 2019.
Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research, 20(1):973-977 January 2019. ISSN 1532-4435.
Leo Breiman. Bagging predictors. Machine Learning, 24(2):123-140, August 1996. ISSN 1573-0565, doi: 10.1007/BF000588555
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and foemn-Hentt Jacobsen. Residual Flows for Invertible Generative Modeling. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Hyunsun Choi, Eric Jang, and Alexander A. Alemi. WAIC, but Why? Generative Ensembles for Robust Anomaly Detection, May 2019.
Anthony Corso, Robert Moss, Mark Koren, Ritchie Lee, and Mykel Kochenderfer. A Survey of Algorithms for Black-Bol Safety Validation of Cyber-Phystiee Systems. Journal of Artificial Intelligence Research, 72:377 January 2022. ISSN 1076-9757. doi: 10.1613/jair.I.1222 1.1716.
Maria Cramer and Michael Levenson. What Caused the Chaos at Southwest. The New York Times, December 2022. ISSN 0362-4331.
Charles Dawson and Chuchu Fan. A Bayesian approach to breaking things: Efficiently predicting and repairing failure modes via sampling. In 7th Annual Conference Robot Learning, August 2023.
11

---

Published as a conference paper at ICLR 2025
Harrison Delecki, Anthony Corso, and Mykel Kochenderfer. Model-based Validation Probabilistic Inference. In Proceedings of The Sth Anmual Learning for Dynamics and Control Conference, pp. 825-837. PMLR, June 2023.
Amir Dembo and Ofer Zeitouni. Large Deviations Techniques and Applications, volume 38 of Stochastic Modelling and Applied Probability. Springer. Berlin, Heidelberg, 2010. ISBN 978-3- doi: 10. 11-10079777---62-03333-0-
642-03310-0 778-3-642-033111
Chengyuan Deng, Shihang Feng, Hanchen Wang, Xitong Zhang, Peng Jin, Yinan Feng, Qili Zeng, Yinpeng Chen, and Youzuo Lin. OpenFWWI: Large-scale Multi-structural Benchmark Datasets for Full Waveform Inversion. In Thirty-Sixth Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, June 2022.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 1-12-2, Red Hook, NY, USA, December 2019. Curran Associates Inc.
Bradley Efron, Bootstrap Methods: Another Look the Jackknife. In Samuel Kotz and Norman Johnson (eds.), Breakthroughs in Statistics: Methodology and Distribution, pp. 569-593. Springer, New York, NY, 1992. ISBN 1778-1-4612-43800 doi: 00.10079978-14412444388
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M Oberman. How to train your neural ODE: The world of Jacobian and Kinetic regularization. Proceedings of the 37th International Conference on Machine Learning, volume 119 of ICML'20, pp. 3154-3164. JMLR. July 2020.
Zhenggqi Gao, Dinghuai Zhang, Luca Daniel, and Duane S. Boning, Rare Event Probability Learning by Normalizing Flows, October 2023.
Kunal Garg, Charles Dawson, Kathleen Xu, Melkior Ornik, and Chuchu Fan. Model-Free Neural Fault Detection and Isolation for Safe Control. IEEE Control Systems Letters, 7:3169-3174 2023.
ISSN 2475- 1456.
doi: 10. 101119911
Wences P. Gouveia and John A. Scales. Bayesian seismic waveform inversion: Parameter estimation and uncertainty analysis. Journal of Geophysical Research: Solid Earth, 103(B2):2559-2779, 1998. ISSN 2202. doi: 10. 10.1099/97]800233.
Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1819-1828, IEEE Computer Society, January 2022. ISBN 978- 10-----6660-915.5
doi: 11119//0AAV11444.22200111...
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep Anomaly Detection with Outlier Exposure. In International Conference on Learning Representations, September 2018
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning Basic Visual Concepts with Constrained Variational Framework. In International Conference on Learning Representations, November 2016
Chin-Was Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural Autoregressive Flows. In Proceedings of the 35th International Conference on Machine Learning, pp. 2078-2087. PMLR, July 2018
Zhuangwei Kang, Ayan Mukhopadhyay, Aniruddha Gokhale, Shijie Wen, and Abhishek Dubey. Traffic Anomaly Detection Via Conditional Normalizing Flow. 2022 IEEE 25th International Conference Intelligent Transportation Systems (ITSC), pp. 2563-2570, Macau, China, October 2022. IEEE Press. doi: 10. 10.1109/TSC551 40.2002..99226611
Azarakhsh Keipour, Mohammadreza Mousaei, and Sebastian Scherer. ALFA: A dataset for UAV fault and anomaly detection. The International Journal of Robotics Research, 40(2-3):5155522 February 2021. ISSN 0278-3649. doi: 10.1 10.1177107733442906666
12

---

Published as a conference paper at ICLR 2025
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, December 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Im- proved Variational Inference with Inverse Autoregressive Flow. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Chaerin Kong, Jeesoo Kim, Donghoon Han, and Nojun Kwak. Few-Shot Image Generation with Mixup-Based Distance Learning. In Computer Vision ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 563-580, Berlin, Heidelherg, October 2022. Springer- Verlag. ISBN 978-3-031-19788-33 doi: 10. 00007/978-3-031-1978888
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online].
Available: http://yann.lecum www.://w...o////////nn.ssss 2, 2010.
Tianci Liu, Tong Yang, Quan Zhang, and Qi Lei. Optimization for Amortized Inverse Problems. In Proceedings of the 40th International Conference on Machine Learning, pp. 22289-22319. PMLR, July 2023.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations, February 2018
Casey Murray, Strengthening airline operations and consumer protections, February 2023.
Naji Najari, Samuel Berlemont, Grégoire Lefebvre, Stefan Duffner, and Christophe Garcia. Robust Variational Autoencoders and Normalizing Flows for Unsupervised Network Anomaly Detection. In Leonard Barolli, Farookh Hussain, and Tomoya Enokido (eds.). Advanced Information Net- working and Applications, pp. 281 292, Cham, 2022. Springer International Publishing. ISBN 00--3-030-9557-44 doi: 10. 00.100077773333 3- 1------00999-0 9000--99577--22
Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why Are Bootstrapped Deep Ensembles Not Better? In "I Can't 't Believe It's 's Not Better!" NeurIPS 2020 Workshop, December 2020.
Matthew O' Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, and John C Duchi. Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation. In Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot Image Generation via Cross-domain Correspondence. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10738-10747, June 2021. doi: 10.1109/CVPR44444.20001
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport. Proceedings of the AAAI Conference on Artificial Intelligence 35(10):9223 2202222222222,222222222222223322 May 2021. ISSN 2374-3468. doi: 10. 1.11609/aai.v335l0 17113.
Nikolas Pyrgiotis, Kerry M. Malone, and Amedeo Odoni. Modelling delay propagation within an airport network. Transportation Research Part C: Emerging Technologies, 27:60-75, February 2013. ISSN 0968-090X. doi: 10. 10.1016/f.trc.201 1.05.017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, ICML 15, pp. 1530-1538, Lille, France, July 2015. JMLR.org.
Alan Richardson. Deepwave. Zenodo, September 2023.
13

---

Published as a conference paper at ICLR 2025
Joel Rose. Southwest will pay a $140 million fine for its meltdown during the 2022 holidays. NPR, December 2023.
Marco Rudolph, Bastian Wandt, and Bodo Rosenhahn. Same same but DifferNet: Semi-supervised defect detection with normalizing flows. In Winter Conference on Applications of Computer Vision (WACV), January 2021.
Aman Sinha, Matthew O' O'Kelly, Russ Tedrake, and John Duchi. Neural bridge sampling for evaluating safety-critical autonomous systems. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, pp. 6402-6416 6416, Red Hook, NY, USA, December 2020. Curran Associates Inc. ISBN 978-1-71382-9544
Southwest Airlines. Final Summary and Action Plan, 2023.
A. M. Stuart. Inverse problems: Bayesian perspective. Acta Numerica, 19:451-559, May 2010.
ISSN 1474-0508, 0962-4929. doi: 00.1017//096249991000000
Esteban G. Tabak and Eric Vanden-Eijnden. Density estimation by dual ascent of the log- likelihooddd Communicutions in Mathematical Sciences, 8(1):217-233, March 2010. ISSN 1539-6746, 1945- 0796.
Toan Tran, Trung Pham, Gustavo Cameiro, Lyle Palmer, and Ian Reid. A Bayesian Data Augmentation Approach for Learning Deep Models. In Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Fabrice Rossi. On the expressivity of bi-Lipschitz normalizing flows. In Proceedings of The 14th Asian Conference on Machine Learning, pp. 1054-1069. PMLR, April 2023.
Yaqing Wang, Quamming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a Few Examples: A Survey on Few-shot Learning. ACM Computing Surveys, 53(3):63: June 2020. ISSN 0360-0300. doi: 10.1 155/338622.
Jian Zhang, Chenglong Zhao, Binghing Ni, Minghao Xu, and Xiaokang Yang. Variational Few-Shot Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1685-1694, 2019.
Ran Zhang, Claudia Czado, and Karin Sigloch. Bayesian Spatial Modelling for High Dimensional Seismic Inverse Problems. Journal of the Royal Statistical Society Series C: Applied Statistics, 65 (2):187-213, February 2016. ISSN 0035-9254. doi: 10. 111l//sc..11111 12118.
Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward Understanding Generative Data Aug- mentation. In Shirty-Seventh Conference on Neural Information Processing Systems, November 2023.
Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided Conditional Diffusion for Controllable Traffic Simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 3560-3566, May 2023, doi: 10. L1010///CRA68891100000000000011
A LIPSCHITZ CONSTANTS FOR CONDITIONAL NORMALIZING FLOWS
In this section, we provide the Lipschitz constants for various conditional normalizing flow architec- tures; i.e. such that (f c1) - f22 c2)l s ||c1 - c2 ll for all z, C2.
Remark 1. The conditional inverse autoregressive flow (IAF; (Kingma et al. 2016)) has Lipschitz constant L < 1. (L., + L.,,,, where the outer product is over autoregressive blocks and the inner product is over steps within each autoregressive block. L,, and L,, are the Lipschitz constants of the neural networks yielding the m, and st values for each autoregressive step (these can be easily bounded for most neural networks; e.g. by the product of the L2 matrix norms of the weight matrices; (Miyato al. 2018)).
14

---

Published as a conference paper at ICLR 2025
A neural spline flow (Durkan et al. 2019) has Lipschitz constant L S 2s, where 8 is an upper bound on the slope s = (8k+1 (Tk+1 2x)) between adjacent knot points of the spline (this can be constrained by construction by ensuring a minimum spline bin width).
Remark 2.
Remark Continuous normalizing flows (Chen al., 2018) have Lipschitz constant L S er where At is the duration of integration and L. is the Lipschitz constant of the neural network defining the vector field of the flow.
Remark 4 (from (Verine et al. 2023 ). Normalizing flows based on invertible residual networks, such i-ResNet (Behrmann al. 2019) and Residual Flow (Chen al. 2019), have Lipschitz constant L S. (1 (ggh where m is the number of residual blocks and Lg < is the Lipschitz constant of the residual block g(z).
Remark (from (Verine et al. 2023)). Normalizing flows based on Glow (Kingma Dhariwal
2018) have Lipschitz constant L. S 1I [Well2 where the product is over the weight matrices W; of the convolution blocks.
B PROOF OF LEMMA 1
Proof. The optimal maximum likelihood estimator is mixture of delta functions (=) ED 5(= //N Since assume that the data points well-separated, the optimal L-Lipschitz maximum likelihood estimator replaces the scaled delta 0(=)/N with the L-Lipschitz function (2) that a) is non-negative, b) integrates to 1/N, and c) maximizes the value 8(0). Con- straint (c) will active (otherwise we recover the scaled delta function), so we know that å will have the form ma L||=|]). Normalizing to 1/N yields
((z) = max (0, a -|=1)
a (d + 1)L4T(3 +1) 1/((+1)
d//2N
Substituting one data point for another (subject to the assumption on data sparsity) changes the optimal estimator by swapping the corresponding mixture component; i.c. exchanging 8(z - 2(2) for 8(2 22)). The W/2 distance between the resulting mixtures is the same as the W22 distance between the changed components. Each component has probability mass 1/N and they are distance [| 2(2) apart, completing the proof.
C PROOF OF LEMMA 2
Proof. Let D: be random dataset created by sampling D N times with replacement. In the proof of Lemma D we show that the optimal L-Lipschitz maximum likelihood estimator given D; ΣLED, 8(2 - 2()). This gives ensemble model
K
Σ 8(z - 200) ]00
Jensemble (2) = Σ
/K
(8)
1=1
LEED.
Since each D; is sampled independently with replacement, we combine the nested sums
Gensemble (2)
Σ 8(=-2(0)KK
where D is single dataset of NK points sampled with replacement from D. As K - the empirical distribution D approaches that of D, so this sum reduces almost surely to:
lim (ensemble
Σ
8(2-20 - (0)
(10)
K +00
15

---

Published as a conference paper at ICLR 2025
D PROOF OF THEOREM
Proof. The W2 metric is defined infemum over couplings ",s in order to provide upper bound suffices to propose a coupling between the nominal and target posteriors, qo (z, 0x) and qs(2,c") Recall that the normalizing flow ya has base distribution yo and flow map fé. where [a c) is assumed to be L-L-ipshiit in the second argument. Consider the joint distribution 721,, #2) defined by 20 (0(2). 21 (zo. 0k) and 22 fz c²). By construction, the marginals of y in each argument are q 2, Ox) and (z c²). respectively, and so is a valid coupling.
This provides the bound
1/2
W: (x))))((;""))
E ||=1
=211?
121:2207
< 1. 1|ce - 0xx
-0x113]42
- L||e*11
E DETAILS ON BENCHMARK PROBLEMS
This section provides additional details for the three types of inverse problem studied in our paper. All problems are implemented using the Pyro probabilistic programming framework (Bingham et al.) 2019).
E.1 SEISMIC WAVEFORM INVERSION
An illustration of the SWI problem is given in Fig. 9 We implement the SWI problem using the Deepwave library (Richardson 2023). We use latent parameters E ER X representing the subsurface density profile (with spatial resolution fr = 10 and ny = 10), context y E TRer representing the source signal, and observations
R representing the signal measured each receiver, where 71, 1, n4, 9, nt 100 are the number of sources, receivers, and timesteps, respectively. Before solving the elastic wave PDE, the density profile is upsampled to 100 30. The observations are corrupted with additive isotropic Gaussian noise. The parameters of this problem are summarized in Table
Serrcce
Receivers
Weasurements x10
Source
100
Time (ms)
Figure 9: (Left) An illustration of the SWI problem and (right) the receiver measurements (blue)
given a source signal (black).
E.2
UAV CONTROL
We model the nonlinear attitude dynamics of the UAV combination of an unknown linear mapping from the current and desired states to angular rates, then nonlinear mapping from angular rates to updated UAV orientation. The state q (Ø, 0, vi] includes the roll, pitch, and yaw angles of the UAV,
16

---

Published as a conference paper at ICLR 2025
Table 4: Summary of parameters for the SWI problem.
Dimension
Latent parameters Density profile (10 10)
100
Context
Observation æ Seismic waveform (100 timesteps at receivers)
900
and denotes the commanded orientation. We model the angular rates of the UAV
p q
Aq K( + d+
(11)
where A, K, and d are unknown feedforward, feedback, and bias dynamics, and is Gaussian process noise. The state derivative is related to w by
dt
4
(g)w
(12)
(0) sin(Ø)
(an()) cos(O) (o)
J (q)
0
cos(o) ()
(13)
0
(on( cos
cos(o)/ cos
We apply a first-order time discretization to yield the one-step stochastic dynamics
a++1 = Q: 8,J (Q) + K(g 9) + 7)
and observed states are additionally corrupted by Gaussian noise. A summary of the parameters for this problem given in Table
An example trajectory, including both nominal and anomalous segments, for the UAV dataset are shown in Fig. TO, In this case, the anomaly is relatively easy detect; the challenge is understanding how the aircraft's s flight dynamics change during the failure so that recovery controller can be designed to handle this case.
Pitch (commanded)
Pitch (measured) Anomaly label
M
0
-10
-30
20
40
60
80
100
120
Figure 10: Example trajectory that includes an elevator failure, including both nominal and anomalous segments.
17

---

Published as a conference paper at ICLR 2025
Table 5: Summary of parameters for the UAV problem.
Dimension
Latent parameters &
Feedforward matrix A (3 x 3)
9
Feedback matrix K (3 3)
9
Bias term d
3
Context y Current state
3
Desired orientation
3
Observation x Next state 3
E.3 AIR TRAFFIC NETWORK
The input to our air traffic model is a list of scheduled flights, each specifying an origin and destination airport and a scheduled departure and arrival time. The latent state includes the mean travel time between each origin/destination pair, the mean service time at each airport (which affects both arriving and departing aircraft and models taxi, deicing. and ATC delays). the mean turnaround time each airport (the minimum time that must elapse before an arriving aircraft may depart), the baseline cancellation rate at each airport, and the initial number of aircraft at each airport. A summary of these parameters are given in Table 7 So that the benchmarks in Section 4 can be run in reasonable time, we restrict the ATC problem used for benchmarking to the four busiest airports and do not model cancellations, but we use the ten busiest airports and do include cancellations in case study in Section 5
The model steps through the scheduled flights in 15 minute increments. In each increment, it checks for the flights that are scheduled to depart from each airport. Each of these flights receives a certain probability of cancellation given by
P(canceled) = - - -1- (10-
available aircraft
(14)
# departing flights in this block
where Pe is the baseline cancellation rate for the origin airport and is the sigmoid function, the probability of cancellation is P. when there are more available aircraft than scheduled departures and approaches 1 as the number of available aircraft decreases. Cancellations are sampled from relaxed Bernoulli distribution with this cancellation probability and a straight- thhrouh gradient estimator. If a flight is canceled, is marked such and the observation for that flight will just be cancel led and will not include actual departure and arrival times. If the flight is not canceled, then it is moved to the runway queue if there are enough aireraft available; otherwise, it delayed until the next time block.
Both departing and arriving flights are served using single M/M/1 queue for each airport, with service times drawn from an exponential distribution with the mean specified according to each airport's mean service time. Once airborne, departing flights are assigned random flight time from a Gaussian with mean given by the mean travel time for each route and fixed variance. Once this travel time has elapsed, they enter the runway queue at the destination airport. Once an aircraft has landed, does not become available to serve new flights until the minimum turnaround time has elapsed (which is sampled from Gaussian with mean given by the mean turnaround time for each airport). Observations for non-cancei con-canceled flights include the simulated arrival and departure times, plus some fixed-varianee Gaussian noise.
E.4
ToY 2D PROBLEM
The data for the 2D toy problem is generated by uniformly sampling nominal data:
0 2(0, T)
x ~ N (cos 0 - 0.5, 0.1)
y (siin 0 0.25, 0.1)
18

---

Published as a conference paper at ICLR 2025
Table 6: International Air Transport Association (IATA) codes and full names of the ten busiest
airports in the Southwest network.
DEN
Denver International Airport
DAL
Dallas Love Field Airport
MDW
Chicago Midway International Airport
PHX
Phoenix Sky Harbor International Airport
HOU
William P. Hobby Airport
LAS
McCarran International Airport
MCO
Orlando International Airport
BNA
Nashville International Airport
BWI
Baltimore/Washingteee International Thurgood Marshall Airport
OAK
Oakland International Airport
Table 7: Summary of parameters for the ATC problem. Pairpon indicates the number of airports in the
model. Yofightss indicates the total number of scheduled flights. indicates parameters that are only included the case study.
Dimension
Top (Section 4 Top- 10 (Section
Latent parameters z Log. of turnaround time each airport (mean minimum delay between arrival and departure) Log. service time at each airport (mean delay between pushback and takcoff) Log. of mean travel times between each airport Log. of initial aircraft reserves at each airport Log. of baseline cancellation probability each airport
Rainoor
10
Veairport
16
100
nairport
nairport
Context y Scheduled arrival time af each flight Scheduled departure time of each flight
Fligghts
44-102
405-497
Milgght
44-102
405-497
Observation
Actual arrival time cach flight Actual departure time of each flight Whether each flight was cancelled
Flfights
44-102
405-497
Thlligtt
44-102
405-497
Tilligtt
44-102
405-497
and anomaly data
~ U(T, 2x)
[ccos 0 0.5, 0.1)
y ~ (ssin 0.75,
Since this problem is meant as an easy-to-visualia test for whether method can learn a posterior distribution with complex shape, set [z, y] as the latent parameters and assume they are observed directly (with the addition of Gaussian noise), rather than treating as the latent parameter (which would lead very easy-to-fi fit posterior).
F IMPLEMENTATION DETAILS
We implement CALNF using neural spline flows (NSF) the underlying normalizing flow (Durkan ct al. 2019). We note that CALNE is agnostic to the underlying flow architecture; we also tried using masked autoregressive flows (Huang al. (018). which trained faster but had slightly worse performance, and continuous normalizing flows (Chen et al.) 2018). which trained much more slowly.
We implement the KL regularization baseline using neural spline flows with KL regularization penalty between the learned anomaly and nominal posteriors. We implement an RNODE-derivel method that includes only the W2 regularization term, not the Froebenius regularization term
19

---

Published as a conference paper at ICLR 2025
(which is used only to speed training and inference, not to regularize the learned posterior; (Finlay et al. ).
We extend our method to anomaly detection by defining score function the ELBO of given observation, approximated using 10 samples from the learned posterior.
All methods were implemented in Pytorch using the Zuko library for normalizing flows Zuk, 2024). The neural spline flows used 3 stacked transforms, and all flows used two hidden layers of 64 units each with ReLU activation (except for the continuous flows the 2D problem, which use two hidden layers of 128 units each). All flows were trained using the Adam optimizer with the learning rate 10-3 (except
the UAV problem, which used learning rate of 10-2) and gradient clipping. CALNF used K- = 5 on all problems, All methods were trained on either single NVIDIA GeForce RTX 2080 Ti GPU or g xlarge AWS instance, with 200, 500, 500, and 150 epochs for the 2D, SWI, UAV, and ATC problems, respectively. The image generation examples were trained for 1000 epochs (MNIST) and 500 epochs (CIFAR-10), Non-image benchmarks all take less than 2 hours to train, and image benchmarks take approximately 20 hours to train. We estimate cloud compute costs for this entire project (including preliminary experiments) at less than 200 USD.
Code examples, including scripts for reproducing the results in Tables | and g and notebooks containing our data analysis for Section 5 are included the attached supplementary material. An open-source version of our algorithm and air traffic network simulator is available at https 7/github. ctm/ hwwp///wwww/aaaaoeaeeo BBac//aaayairrr
G FURTHER EMPIRICAL RESULTS
Table 8: ELBO (nats/dim) on held-out target data for ablations of CALNF. The first is our proposed method, the second fixes c, the third excludes the nominal data during training, and the fourth does not subsample the target data. scaleed by x x10-3
2D
SWI
UAV
ATC!
CALNF
-0.90
46.3 46.340.2
6.95
-0.01
w/O C¹
-0.96
46. 46.2
7.86+1.
-2.02+0.1
W/O Do
-1.1210.2
110.4
-9.22=10
-2.0310.2
w/o DE
-1.0310.2
43.912.8
-3.65=11
-2.05
Fig. shows the sensitivity of our method to different values of K on the SWI benchmark. We
find that there is slight trend towards better performance as increases, and that including the calibration step (rather than using fixed e²) improves performance at all levels of K.
46400
Test set ELBO (nats/dim)
46200
46000
45800
Calibration False
True
45600
Number of subsamples
Figure 11: The ELBO on a held-out test set for the anomaly posterior learned using CALNF on the SWI example using varying number of subsamples.
Table 9 includes the training times for our method and each baseline. We can also compare these methods theoretically in terms of the number of evaluations of the joint likelihood p(x, 2; y) in Eq. (4)
20

---

Published as a conference paper at ICLR 2025
Table 9: Training time (minutes) on benchmark problems, Mean and standard deviation reported over four random seeds.
2D
SWI
UAV
ATC
KL-regularized (8 0.01)
0.25
6.48
31.17
84.29+1.32
Kl-regularized 0.1)
1.26
6.53 0.18
31.20
84.3940..8
KL-regularized = 1.0)
1.34
6.58_0.18
31.24:20.1
84.0994.778
W2-regularized (B = 0.01)
8.7020.43
8.50=0.25
23.93
9-1.4411.36
Wz-regularized (B 0.1)
8.5920.49
8.27=0.42
23.54
93.9841.55
We-regularized (3 1.0)
8,4040.50
8.40=u.38
23.96+40.44
93.99+1.22
Ensemble
1.58
14. 14.25
76.24
147.91 46.81
CaINF (ours)
2.35
19.74
77.74
174.49
The KI. and Wz-regularized methods require one evaluation per training step, the ensemble method requires K evaluations, and our method requires + 1. The inference times are identical except for the Wz-regularized method, which is slower due to its use of neural ODEs for the flow map).
Fig. 12 show the performance of CALNF the 2D benchmark as the number of target data points is decreased. When the size of the target dataset is at least 10% the size of the nominal dataset, high value. Between 2% and 5%, CALNF's performance begins to deteriorate, and below 2% performance drops off sharply. Higher values of / improve performance extremely data-sparse cases, decrease performance when many target data points are available. This example uses 1000 nominal data points; the results in our main paper use / = 1.0 (constant across all experiments) and a target dataset 2% the size of the nominal dataset for this problem.
CALNF's test-set ELBO saturates
we see that larger values of perform better when the failure dataset very small, but smaller values of / perform better when the failure dataset is large. These results suggest that when plenty of information available (in the form of large training dataset), is beneficial to encourage diversity among the candidate posteriors through small regularization strength 3 however, when information limited, we can achieve better performance by encouraging similarity between candidate posteriors to reduce overfitting.
From results in Fig. 12
0.0
0.5
-1.0
(nats/clmm
-1.5
-2.0
-2.5
8
-3.5
0.1
-4.0
10
10
10"
(M target data points) / (# nominal data points)
Figure 12: Performance of CALNF under extreme data scarcity on the 2D benchmark, with varying self-regularization strengths 3.
21

---

Published as a conference paper at ICLR 2025
Table 10: ELBO (nats/dim) on held-out anomaly data of the ensemble baseline trained without nominal data. The second and third rows are replicated from Table T for ease of comparison.
2D (nats/dim) f
SWI (nats/dim)
UAV (nats/dim) t
ATC (nats/dim)
Ensemble (w/o nominal data) Ensemble (w/ nominal data) CaINF (ours)
-1.33
45.93
5.81
-2.07
-0.84
46. 20.42
6.6540.98
-2.2320.06
0.90
46.440.26
7.55+0.60
-2.1120.13
Table TO includes additional results comparing the performance of the ensemble method both with and without simultaneous training the nominal dataset, with the performance of our method from Table included for ease of reference.
H ADDITIONAL RESULTS ON SOUTHWEST AIRLINES CASE STUDY
A timeline of the 2022 Southwest Airlines scheduling crisis is shown in Fig. 13
1322: 13/24
60%
20sh Ahutt
2023 20-23
1222-12-66
2039 SPo77
2022 28
2022 -g
DEN NDW BMA
BKI
CAL
LAS
PHX
MC0
HOU
DAC
Figure 13: (Left) Timeline of cancellations during the 2022 Southwest Airlines scheduling crisis.
(Right) Cancellations at the 10 busiest airports during the first four days of the disruption.
22


Reviews:
[{"review_id": "ybZ68bTtWh", "reviewer": "Reviewer_MKpx", "summary": "In this work, the authors propose an algorithm for rare-event modeling and failure detection using normalizing flows. The motivation stems from the observation that rare events are significantly low in number, making traditional approaches like prior regularization or simple bootstrapping ineffective. To address this challenge, the authors develop an optimization pipeline that trains a normalizing flow with a built-in calibration mechanism, which identifies the optimal latent factors associated with a target (failure) dataset. The paper is supported by extensive theoretical analyses, experiments, and an interesting case study, all demonstrating the efficacy of the proposed approach.", "questions": "* How would the performance change if the calibration procedure is conducted jointly with the training of the normalizing flow? Would this make the optimization process more challenging?\n* Figure 4 needs to be referenced in the text of the paper for clarity.\n* Can this framework be extended to problems where the calibration label is continuous-valued? Understanding this extension would be useful, as many scientific applications involve real-valued outputs or measurements.\n* Could the authors elaborate on the anomaly rejection and few-shot inference experiment details? A more comprehensive explanation of these procedures would enhance the clarity and depth of the paper.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper addresses an important and intriguing problem in modeling rare events or failures with generative models, which often struggle to generalize when trained on limited data. It is mathematically well-supported and includes relevant case studies and experiments on standard benchmarks", "weaknesses": "Refer to Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ObmwvbJni9", "reviewer": "Reviewer_Wb4h", "summary": "This paper aims to address rare event modeling problems where failure data is in a very limited amount. Specifically, the authors propose a generative modeling framework called Calibrated Normalizing Flows (CALNF) to learn a shared representation from both normal and rare failure data. Furthermore, they leverage calibration and self-regularization processes to avoid overfitting. They provide empirical results on benchmarks and real-world cases.", "questions": "See Weaknesses for the major concerns. Other questions:\n1. Is the calibrated normalizing flows model the optimal solution? Can the proposed model generalize to other generative models such as GAN and Diffusion models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is intuitive and easy to understand.\n2. The paper is well-organized and easy to follow.\n3. The qualitative visualizations on benchmark datasets are impressive.\n4. The case study on the 2022 Southwest Airlines scheduling crisis is persuasive.", "weaknesses": "1. The authors assume that failure events are not entirely disjoint from normal events, however, this might not hold in cases where failures are out-of-distribution (OoD) data. This means the failures can have drastically different characteristics from the normal ones. Can the proposed method still represent these rare events accurately in this case?\n2. Even with self-regularization, the model’s performance might still be sensitive to the selection of the number of subsamples $K$ and regularization weights $\\beta$. What method did you use to select $K$ and $\\beta$? Was it tuned manually, or is there a heuristic that works well across different failure datasets?\n3. As discussed by the authors, the proposed method introduces computation overhead. Have you considered how to extend the proposed method to high-dimensional data such as images?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VF9nyhkYpB", "reviewer": "Reviewer_QvkQ", "summary": "This paper presents Calibrated Normalizing Flows (CALNF), a method for modeling rare failures in data-limited settings by learning robust posterior distributions through self-regularization. CALNF combines nominal and limited failure data to capture event distributions effectively, demonstrated on benchmarks like UAV control and air traffic disruptions, including an analysis of the 2022 Southwest Airlines crisis.", "questions": "1. Could you provide more theoretical insights or empirical evidence on how CALNF handles data sparsity when transferring to new or unseen data points? For applications in autonomy and robotics, understanding the model's generalization capability could be essential.\n\n2. How sensitive is CALNF to the choice of regularization hyperparameter β and the number of subsamples K? While some sensitivity analyses are included, it would be beneficial to understand how practitioners might adapt these parameters to different problem settings with minimal fine-tuning.\n\n3. Given the added training cost of CALNF due to subsampling, how practical is this method for real-time applications in autonomy or robotics where immediate inference might be required? Would you recommend any adjustments for faster, low-latency performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The CALNF framework addresses an important challenge in failure modeling by using normalizing flows for posterior learning from limited data. This approach is innovative in combining data regularization with low-dimensional embedding optimization, which could be valuable in real-world failure analysis in autonomous systems.\n\n2. The paper tests its methodology on practical datasets, including autonomous UAV control and air traffic disruptions. This inclusion strengthens the paper’s relevance for the conference theme, as it presents an approach applicable to practical, high-impact fields.\n\n3. The paper compares CALNF against well-established baselines, such as KL and Wasserstein regularized methods, and ensemble methods. These comparisons and detailed analyses provide a fair view of CALNF’s effectiveness, especially in data-limited scenarios", "weaknesses": "1. The dual steps of the method—learning a low-dimensional embedding and calibrating a label space—add significant complexity, which may impact reproducibility and practical use, particularly in robotics where model simplicity and speed are often crucial. Additional clarifications or simplifications could improve accessibility to a broader audience \n\n2. Although the authors acknowledge this in the Limitations section, the increased training time due to subsampling may limit CALNF's applicability to settings requiring rapid inference, such as real-time planning and decision-making in robotics. The authors could consider discussing strategies to mitigate this cost \n\n3. While CALNF aims to model rare failures effectively, it does not provide direct risk estimates or probability bounds for these events. Adding a framework for estimating risk or failure probabilities could enhance the utility of this approach for preemptive failure detection and mitigation \n\n4. The paper provides some analysis around sensitivity and regularization but lacks a comprehensive discussion of how well the approach generalizes to new data. For applications in robotics and autonomous systems, where new conditions are frequently encountered, more robust guarantees on generalization could significantly strengthen the work", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTv5SUu1WX", "reviewer": "Reviewer_Qbv4", "summary": "In this work, the authors propose a calibrated normalizing flow method to address the rare-event modeling problem in data-constrained settings. Experimental validation demonstrates improved performance compared to the prior regularization and bootstrapping ensemble baselines.", "questions": "**Q1:** Equation 5 and the explanation between lines 175-177 appear to present conflicting information. If minimizing the total loss leads to maximizing the KL divergence between candidate posteriors (due to the subtraction sign), wouldn’t this be against the statement that the divergence between candidate posteriors learned for each subsample should be small?\n\n**Q2:** Could you please elaborate on why the candidate posteriors learned for each subsample should be kept small? Typically, ensemble methods enhance predictive performance by increasing the diversity among ensemble members.\n\n**Q3:** Is there a typo in the Equation 3 where the $q_{\\phi_t}$ in the DL part should be $q_{\\phi}$? Should $c^*$ be in the loss $L_{cal}$ (line 178) rather than $c$?\n\n**Q4:** Could you please elaborate on why the ensemble method can not infer the correct shape (lines 314-316) in Figure 3? The results of the ensemble and the CalNF look a bit similar.\n\n**Q5:** In lines 112-114, some drawbacks of the ensemble method (e.g., data sensitivity issue in deep models), would the proposed method counter these issues?\n\n**Q6:** Regarding the statements in lines 129-131, could you please provide some supportive references or explanations to illustrate the practical issues even in the case of selecting an optimal $\\beta$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written, and the proposed method is presented clearly. \n\nExperimental settings and results are described in detail, and limitations related to training costs are analyzed.\n \nExtensive experimental validations across various settings demonstrate the effectiveness of the proposed method.", "weaknesses": "A subsection that clearly outlines a comprehensive literature review of recent research achievements in related work is missing.\n\nIn the primary experimental validation, the most relevant comparisons are made between prior regularization techniques (based on studies from 2020/2021) and the bootstrapping approach (from 2019). Thus, claiming that the proposed method achieves state-of-the-art performance in both the abstract and introduction (lines 63–64) seems not very convincing.\n\nIn Section 4.2, the ensemble size is set to $K=5$. Additionally, an ablation study examines both the training time and the sensitivity of CalNF to varying values of $K$ (These results are appreciated). As increasing ensemble size generally improves performance, it remains uncertain whether the proposed method maintains its superiority over ensemble baselines (as shown in Table 1) as $K$ increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FlZWGLRPmi", "reviewer": "Reviewer_mQHM", "summary": "The paper introduces Calibrated Normalizing Flows (CALNF), a novel approach to rare event modeling in data-constrained scenarios. CALNF builds on normalizing flows to learn an approximate posterior over latent variables $z$, using a blend of ensemble techniques and self-regularization. In essence, the method operates by jointly training on the complete target dataset, its subsets, and nominal data. A conditional normalizing flow is applied across both the target, nominal and subset posteriors, followed by a calibration step that optimally combines these learned posteriors to approximate the overall target distribution. By jointly learning across both subsets and the full dataset, CALNF captures the target distribution effectively while minimizing overfitting, making it more robust in settings with limited data.\n\nWhile the paper presents an innovative approach to rare event modeling, the marginal performance gains over the ensemble baseline do in my opinion not justify the increased computational complexity, leading to a weak reject. For a more detailed justification, please see the comments below.", "questions": "I recommend that the authors further clarify and refine the benefits of CALNF compared to the Ensemble method, particularly addressing cases where the performance gains justify the additional complexity.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Addresses an important problem:** The paper tackles the challenge of modeling rare events with limited data which is a relevant issue in many safety-critical applications.\n\n**Theoretical Foundation:** The paper includes a theoretical analysis, such as bounding the Wasserstein distance between nominal and target posteriors, adding robustness to the method.\n\n**Clear Presentation:** The method and its applications are well-presented, making the paper easy to read.", "weaknesses": "**Marginal performance gains vs computational complexity**  As the authors note, Table 1 contains the main empirical results, comparing performance on four benchmark datasets between the proposed method and simpler ablated versions, including the Ensemble approach. However, the Ensemble method performs nearly as well as the proposed CALNF, with results that are statistically indistinguishable from CALNF in three out of four cases. This similarity in performance raises questions about whether the added complexity of CALNF is justified. In particular, (i) the Ensemble approach has fewer hyperparameters (only the number of ensemble components, with no regularization parameter), (ii) does not rely on nominal data, and (iii) allows parallelized training across ensemble members.\n\nWhile the issue of additional hyperparameters is mitigated to some extent by the authors' ablation study in the appendix, the other limitations are more concerning. CALNF requires joint training on nominal and target data to estimate the posterior over the target distribution, adding significant complexity relative to the Ensemble method. This dependency on both data types also limits CALNF’s suitability for online or real-time applications, where training with nominal data might not be feasible, further reducing its practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)