Title: Learning from Uncertain Data: From Possible Worlds to Possible Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: data uncertainty

robustness verification

predictive multiplicity

abstract interpretation

linear regression
Abstract: We introduce an efficient method for learning linear models from uncertain data, where uncertainty is represented as a set of possible variations in the data, leading to predictive multiplicity. Our approach leverages abstract interpretation and zonotopes, a type of convex polytope, to compactly represent these dataset variations, enabling the symbolic execution of gradient descent on all possible worlds simultaneously. We develop techniques to ensure that this process converges to a fixed point and derive closed-form solutions for this fixed point. Our method provides sound over-approximations of all possible optimal models and viable prediction ranges. We demonstrate the effectiveness of our approach through theoretical and empirical analysis, highlighting its potential to reason about model and prediction uncertainty due to data quality issues in training data.

Learning from Uncertain Data: From Possible Worlds to Possible Models

Abstract

We introduce an efficient method for learning linear models from uncertain data, where uncertainty is represented as a set of possible variations in the data, leading to predictive multiplicity. Our approach leverages abstract interpretation and zonotopes, a type of convex polytope, to compactly represent these dataset variations, enabling the symbolic execution of gradient descent on all possible worlds simultaneously. We develop techniques to ensure that this process converges to a fixed point and derive closed-form solutions for this fixed point. Our method provides sound over-approximations of all possible optimal models and viable prediction ranges. We demonstrate the effectiveness of our approach through theoretical and empirical analysis, highlighting its potential to reason about model and prediction uncertainty due to data quality issues in training data.

1 Introduction

This paper addresses the challenges of learning from uncertain datasets by employing the framework of possible world semantics , a well-established concept in AI and database theory [59, 23, 41, 28]. In this approach, uncertainty in a dataset D is conceptualized through a collection of possible datasets { D 1 , D 2 , . . . } , each representing a potential state of the real world, reflecting variations due to missing entries, errors, inconsistencies, and biases. Given this framework and a learning algorithm, our objective is to construct a set of models { f 1 , f 2 , . . . } , where each model f i is trained on a corresponding potential dataset D i . This method, that we implement in a system called ZORRO ( ZO notope-based R obustness Analysis for R egression with O ver-approximations), allows for a thorough evaluation of how data uncertainties affect the robustness, reliability, and fairness of models in predictive modeling and statistical inference, particularly in scenarios where the ground truth is unidentifiable, necessitating consideration of all possible dataset variations.

While the framework of possible world semantics is essential for modeling dataset uncertainties, it poses significant challenges due to the potentially infinite number of scenarios each dataset might represent. Exploring every possibility and training a model for each is impractical. The concept of model multiplicity , which highlights situations where models with similar accuracy differ in individual predictions, has gained traction [7, 42], yet it primarily focuses on competing models without addressing the full range of dataset variations. Similarly, dataset multiplicity introduced in [47] recognizes data variations due to uncertainty but [47] only proposed a solution for linear models with label errors. Our approach expands these ideas by using possible world semantics to systematically manage uncertainty across all features and labels, thus creating a comprehensive framework for evaluating model robustness amid data uncertainties.

To address the challenges of learning from uncertain datasets, we employ the method of abstract interpretation [13]. Utilizing zonotopes-a type of convex polytope well-suited for compactly representing high-dimensional data spaces [74, 4]-we over-approximate the set of all possible dataset variations. This framework allows for the simultaneous symbolic execution of gradient descent

across all possible datasets, compactly over-approximating all possible optimal model parameters as a zonotope. We demonstrate that for linear regression with ℓ 2 regularization (Ridge), our method admits a non-trivial closed-form solution. The zonotope representation of model parameters enables efficient inference, facilitating reasoning about the range of possible predictions or of specific parameters.

We introduce an abstract gradient descent algorithm for learning linear regression models from uncertain data. This method over-approximates data variations using zonotopes and symbolically executes gradient descent on all possible datasets concurrently. We define and prove the existence of a fixed point that soundly over-approximates all potential models.

Symbolic execution generates intractable polynomial zonotopes for gradients due to non-linear terms and monomial growth that is exponential in the number of iterations. We use linearization and order reduction to compactly over-approximate these polynomial zonotopes using linear zonotopes at each step, introducing an efficient version of abstract gradient descent.

The efficient version, however, does not guarantee a fixed point for arbitrary order reduction techniques. To address this, we develop advanced order reduction techniques that ensure fixed points and provide a non-trivial closed-form solution for these fixed points in ridge regression.

We implement our approach in a system called ZORRO and use it to evaluate the impact of data uncertainty on linear regression models. Our empirical results and analytical solutions validate the effectiveness of our approach, demonstrating its efficacy in computing prediction ranges and verifying robustness.

Related Work. Predictive multiplicity has shown that a single dataset can produce multiple optimally fitted models due to variations in training processes [8, 11, 44, 66, 69, 9, 15, 71, 42] or modifications to training parameters such as random seeds, data ordering, and hyperparameters. For predictive multiplicity due to missing data, Khosravi et al.[34] address the issue using a probabilistic method that computes expected predictions for all possible imputations. Our approach can be seen as an extreme case of multiple imputation [62, 61], where we consider all possible data variations rather than just a few plausible scenarios. Meyer et al. [47] recently introduced the concept of dataset multiplicity, using possible world semantics to model how uncertain, biased, or noisy training data can lead to predictive multiplicity. However, their focus is on uncertainty in training labels, and they use interval arithmetic for over-approximation of prediction intervals for linear regression. In contrast, our approach handles arbitrary uncertainty in features and labels during both training and testing using zonotope-based learning for over-approximation of prediction ranges and model parameters. We show that interval arithmetic fails to provide tight prediction ranges even for uncertainty in labels.

Our work is broadly related to robust model learning, which ensures robustness against data quality issues such as attacks [31, 76, 70, 60, 55, 52, 29]. Distributional robustness [6, 53, 65] studies model reliability against varying data distributions, while robust statistics [19, 18] examines model performance under outliers or data errors. Our approach provides exact provable robustness guarantees by exploring the entire range of models under extreme dataset variations, which is crucial for individual-level predictions and reasoning about the robustness of specific parameters.

Our work is also related to robustness certification, which certifies ML models' robustness against data perturbations and uncertainties [27, 68, 50]. These efforts mainly focus on test-time robustness, validating predictions for inputs in the vicinity of a test sample. In contrast, we address training-time robustness, considering the effects of possible datasets on training models. Closest to our approach is the work by Meyer et al. [46] for decision trees and Karlas et al. [33] for nearest neighbor classifiers. We use zonotopes to over-approximate prediction ranges for linear regression, generating robustness certificates. While zonotopes have been used for test-time robustness [49, 20, 22, 51], our work is the first to apply zonotopes for training-time robustness for an iterative learning algorithm.

Approaches for uncertainty quantification (UQ) aim to understand the range of outcomes a model may produce using Bayesian methods, ensembling, conformal prediction, and bootstrapping [54, 43, 35, 17, 16]. UQ focuses on epistemic and aleatoric uncertainty, stemming from insufficient data, noisy data, or uncertainty about the model parameters, and does not account for uncertainty due to systematic data quality issues, such as non-random data errors or missing values [25], which induce a multiplicity of possible datasets. In this case, UQ methods might underestimate the uncertainty as they rely on critical assumptions. Bayesian methods, for instance, require correctly specified priors to accurately model uncertainty, often failing under conditions with unknown or erroneous

priors [73, 67, 72], while conformal prediction (CP) assumes data exchangeability-a condition that breaks down when data errors are systematic [21, 75]. In contrast, our approach addresses this distinct challenge by computing sound over-approximations that guarantee complete coverage of potential predictions across all variations of the dataset. This sound coverage is essential in high-stakes settings such as evaluating the robustness of predictive models for medical use.

2 Notation, Problem Formulation and Background

Denote a training dataset D = ( X , y ) with X = [ x 1 · · · x n ] T ∈ R n × d as the matrix of features, and y = [ y 1 · · · y n ] T ∈ R n as the corresponding ground truth labels. Let f ( x ; w ) be a model parameterized by w ∈ R p that maps an input data point to a label. A learning algorithm A maps a training dataset D to the parameters of the trained model, w ∗ = A ( D ) . Given a test dataset X test ∈ R n × d , for any test sample x from X test, the function f computes a prediction ˆ y = f ( x ; w ∗ ) .

2.1 Learning Possible Models from Possible Worlds

We use possible world semantics to represent the uncertainty in a dataset D .

Definition 2.1 (Possible Datasets) . Given an uncertain dataset D , the uncertainty in D can be represented by a set of possible datasets D ⊙ : D ⊙ = { D 1 , D 2 , . . . } .

Each dataset D i ∈ D ⊙ is a "possible world", i.e., a hypothetical variation of the dataset D that could potentially exist in the real world based on our knowledge about the uncertainty in D .

Example 2.2. Consider an e-commerce dataset D where some product price is missing, meaning the exact price is unknown. Using possible world semantics, we represent this uncertainty with a set of possible datasets D ⊙ , each containing a possible clean price, which could be obtained from prices of the same items on the market.

In App. D, we discuss construction methods for common data quality issues. Our goal is to efficiently construct the set of all possible models from uncertain data and understand their behavior in making predictions.

Definition 2.3 (Possible Models and Prediction Range) . Given a set of possible datasets D ⊙ associated with an uncertain dataset D , the possible models, denoted f ⊙ , are obtained by applying the learning algorithm A to each training dataset D i within D ⊙ to obtain the set of all possible optimal model parameters w ⊙∗ , i.e.,

For a test data point x , the viable prediction range V ( x ) is defined as the interval between the least upper bound and the greatest lower bound of the outputs produced by all models in f ⊙ , i.e.,

This prediction range quantifies the minimum and maximum predictions that can be expected for x , highlighting the variability in model outputs due to differences in the training data. Our framework supports uncertainty in training and test data. We discuss test data uncertainty in App. F.3.

2.2 Sound Approximation of Possible Models with Abstract Interpretation

The set of all possible datasets associated with uncertain data can be intractable. We use abstract interpretation [12] to over-approximate sets of elements of a concrete domain D (the training data and model weights) with elements from an abstract domain D ♯ . Specifically, we use the abstract domain of zonotopes, a type of convex polytope, to over-approximate the possible datasets D ⊙ using a zonotope D ♯ that has a compact symbolic representation. Instead of applying the learning algorithm A to each possible dataset to

!

⊙

"

#

#

!

#

"

#

#

⊙

#

"

#

#

#

$

!

"

!

#

!

$

Abstract Learning

Concrete Learning

Dataset

Weights

Possible

Datasets

Possible

Weights

Abstract

Abstract compute all possible optimal model parameters w ⊙∗ , we develop an abstract learning algorithm A ♯ that operates directly on the abstract domain of zonotopes. Given D ♯ , A ♯ generates a zonotope w ♯ = A ♯ ( D ♯ ) that over-approximates w ⊙∗ (demonstrated in the graph on the right). Intuitively, this represents the symbolic execution of the learning algorithm across all possible datasets simultaneously.

Definition 2.4 (Abstract Domain) . Let D be a concrete domain. An abstract domain for D is a set D ♯ paired with two functions:

Abstraction α : P ( D ) → D ♯

Concretization γ : D ♯ →P ( D )

which satisfy the following condition for any subset S ⊆ D : γ ( α ( S )) ⊇ S . Two abstract elements d 1 and d 2 are equivalent, written as d 1 ≃ ♯ d 2 , if γ ( d 1 ) = γ ( d 2 ) .

Def. 2.4 ensures that the abstract element α ( S ) associated with a set S through application of the abstraction function α encodes an over-approximation of S . We will use an abstract element D ♯ = α ( D ⊙ ) to over-approximate the possible worlds of an uncertain training dataset D ⊙ . We discuss abstraction functions α for specific types of training data uncertainty in App. D.

Definition 2.5 (Abstract Transformer) . Consider a function F : D 1 → D 2 on concrete domains D 1 and D 2 . An abstract transformer F ♯ : D 1 ♯ → D 2 ♯ over-approximates F in the abstract domain:

An abstract transformer is exact (does not loose precision) if ∀ d ∈ D ♯ : γ ( F ♯ ( d ) ) = F ( γ ( d )) .

Importantly, (exact) abstract transformers compose (see App. B.2, Prop. B.1) and, thus, we can construct an abstract transformer for complex functions from simpler parts.

To over-approximate the set of possible model parameters w ⊙∗ , we will develop an abstract transformer A ♯ for the learning algorithm A to get γ ( A ♯ ( D ♯ ) ) ⊇ w ⊙∗ .

Symbolic Abstract Domains and Zonotopes. We consider a symbolic abstract domain Ψ of vectors and matrices (marked with · ♯ ) with elements that are polynomials ψ over variables E = { ϵ i } . The concretization of a polynomial ψ is the result of evaluating ψ on all assignments e : E → [ -1 , 1] , encoded as vectors [ -1 , 1] | E | : γ ( ψ ) = { ψ ( e ) | e ∈ [ -1 , 1] | E | } . We lift concretization to vectors and matrices through point-wise application. Such an object z ♯ is typically referred to as a polynomial zonotope or zonotope if all symbolic expressions are linear (see App. C.2). The concretization of z ♯ is: γ ( z ♯ ) = { z ♯ ( e ) | e ∈ [ -1 , 1] | E | } .

3 Exact Abstract Transformers for Learning Linear Models

Given an uncertain training dataset D ⊙ , we aim to over-approximate the set of possible optimal linear models w ⊙∗ = { w ∗ 1 , w ∗ 2 , . . . } , where w ∗ i ∈ R p represents the optimal parameters of a linear model trained on D i ∈ D ⊙ . These optimal parameters are the fixed point of the sequence { w k i } ∞ k =0 generated by: w k +1 i = Φ( w k i ) where the operator Φ : R p → R p captures one step of gradient descent, i.e., Φ( w ) = w -η ∇ L ( w ) , for a learning rate η and a loss function L ( w ) [57].

In the abstract domain, we use the zonotope representations D ♯ and w ♯ to abstract the possible datasets D ⊙ and the set of possible model weights w ⊙∗ . While it is theoretically possible to compute symbolic expressions for the standard closed form solution for linear regression, this can result in large expressions that contain fractions with polynomial numerators and denominators and, thus, computing prediction intervals based on such expressions is computationally infeasible (see App. L). Instead, we over-approximate the optimal parameters using an abstract operator Φ ♯ exact : w ♯ → w ♯ that generates a sequence of abstract elements { w ♯ k } ∞ k =0 :

where the abstract operator Φ ♯ exact given by Φ ♯ exact ( w ♯ ) = w ♯ -η ∇ L ( w ♯ ) captures one gradient descent step in the abstract domain. Specifically, for any loss function L whose gradient ∇ L consists of linear or polynomial expressions, such as the mean squared error (MSE) loss, Φ ♯ exact is an exact abstract transformer. This follows from the existence of exact abstract transformers for addition and multiplication over polynomial zonotopes [37] and the fact that abstract transformers compose (see App. E, Prop. B.1 and Prop. E.1).

Proposition 3.1. The abstract gradient descent operator Φ ♯ exact is an exact abstract transformer for the concrete gradient descent operator Φ . Formally, for any abstract w ♯ ,

FP

11

15

30

199

Unfortunately, the sequence { w ♯ k } ∞ k =0 does not have a fixed point as the highest-order symbolic terms in w ♯ k are multiplied with symbolic terms in the gradient. Thus, w ♯ k +1 = Φ ♯ exact ( w ♯ k ) has terms of higher orders than any term in w ♯ k . To identify when the abstract model weights represent all fixed points in the concrete domain, we define abstract fixed points using concretization.

Definition 3.2 (Fixed Point of Abstract Gradient Descent) . An abstract model weight w ♯ ∗ is a fixed point for an abstract transformer F ♯ for the gradient descent operator Φ if,

Here, γ ( · , · ) denotes the joint concretization of w ♯ ∗ and D ♯ (see App. C.2 for more details).

Any abstract fixed point w ♯ ∗ over-approximates all possible optimal model weights w ⊙∗ .

Proposition 3.3. Consider an abstract transformer F ♯ for the gradient descent operator Φ . Let w ♯ ∗ be a fixed point according to Def. 3.2. Then it holds that γ ( w ♯ ∗ ) ⊇ w ⊙∗ .

As we demonstrate in App. F.3, w ♯ ∗ can also be used to compute prediction ranges. In general, an abstract fixed point is not guaranteed to exist for every abstract transformer for Φ . While Φ ♯ exact as defined above has a fixed point (see Prop. F.1 in the appendix), two significant barriers remain. First, the representation size of the polynomial zonotope w ♯ i for abstract model parameters grows exponentially in i , the number of the steps of abstract gradient descent. Additionally, testing for convergence is challenging, as checking for containment is already NP-hard for linear zonotopes [39].

4 Efficient Sound Approximation for Learning Linear Models

In this section, we present an efficient abstract gradient descent method for ridge regression that guarantees a fixed point and addresses the exponential growth of generated abstract model parameters. The core idea is to use the linearization and order reduction techniques introduced in the following to deal with intractable polynomial zonotopes. We develop a specific order reduction technique that admits a fixed point and enables us to find a closed-form solution for the fixed point.

4.1 Gradient Descent With Order Reduction

A linearization operator L maps a polynomial zonotope z ♯ to a linear zonotope ℓ ♯ that 11 15 30 199 FP over-approximates z ♯ : γ ( L ( z ♯ ) ) ⊇ γ ( z ♯ ) . An order reduction operator R takes a linear zonotope ℓ ♯ as input and returns another linear zonotope ℓ ♯ of reduced order (representation size) such that: γ ( R ( ℓ ♯ ) ) ⊇ γ ( ℓ ♯ ) . Specific linearization and order reduction operators are discussed in App. G. The graph on the right shows the progression of the zonotope of parameters toward a fixed point from a random initialization by applying 200 iterations of the following abstract gradient descent operator Φ ♯ . The number inside each zonotope indicates its corresponding iteration.

After 200 iterations the resulting zonotope is close to a fixed point shown in red and computed using techniques we introduce in the following. This operator first linearizes the gradient zonotope using L , subtracts it from the current abstract model parameters w ♯ , and then reduces the order of the resulting zonotope using R :

This operator is an abstract transformer for the gradient descent operator (Prop. H.1). While each

36

80

100

199

FP

step can be computed efficiently by bounding the order of the resulting zonotope, Φ ♯ may not always converge to a fixed point. The graph on the left illustrates a real example where, despite the existence of a zonotope containing all optimal possible parameters (shown again in red), abstract gradient descent with linearization and order reduction keeps diverging, generating larger and larger zonotopes due to over-approximation error. In the following, we develop an order reduction operator that ensures abstract gradient descent with linearization converges to a fixed point for linear regression with ℓ 2 regularization. Additionally, we derive a closed-form solution for this fixed point. The red dotted zonotope in the graphs represents the fixed point generated by our method.

FP

199

100

80

36

Decomposition of Gradients. A core idea in our approach is to decompose the abstract dataset into real and symbolic parts and use an order reduction operator that allows for the decomposition of the abstract gradient descent operator into components processing the real and symbolic parts in a specific evaluation order. Specifically, we observe that the components of the abstract training dataset D ♯ = ( X ♯ , y ♯ ) can be decomposed into a sum of a real part (without error symbols) and a symbolic part (containing only symbolic terms). Furthermore, the abstract model weight w ♯ produced by Φ ♯ can be decomposed as shown below. w R is the real part of the zonotope and does not contain any error symbols, w ♯ D contains only symbolic terms with error symbols that also occur in D ♯ , and w ♯ N contains only symbolic terms introduced by linearization and order reduction, hence does not contain any error symbols from D ♯ since these methods introduce fresh symbols.

The key observation that enables our approach is that one step of abstract gradient descent can now be decomposed into three components:

where

w R w ♯ D w ♯ N Φ R ( w R ) Φ ♯ D ( w R , w ♯ D ) Φ ♯ N ( w R , w ♯ D , w ♯ N ) Φ R ( w R ) , the real part updater , only relies on and updates w R , the real part of the abstract weight, and coincides with the abstract gradient operator Φ on the real part of abstract weights. Φ ♯ D ( w R , w ♯ D ) , the symbolic data-dependent updater , which is a function of w ♯ D itself and w R , updates w ♯ D based on some linear terms and consists of symbols that come from the abstract dataset D ♯ and does not include any non-linear terms; hence applying this does not generate higherorder terms, so the output is a linear zonotope, and no linearization is needed. Additionally, no order reduction is needed since the number of error symbols remains constant. Φ ♯ N ( w R , w ♯ D , w ♯ N ) , the symbolic data-independent updater , is a function of w ♯ N itself, w ♯ D , and w R . It consists of the non-linear terms in the gradient, hence performs linearization to over-approximate higher-order terms and performs order reduction to reduce the number of error symbols, which would otherwise grow exponentially with the number of iterations. Therefore, it generates the updated w ♯ N as a linear zonotope that does not share any symbol with D ♯ , i.e., only consists of fresh symbols introduced by linearization and order reduction. See Appendix I for formal statements and proofs for this section.

Proposition 4.1. Any abstract model weights w ♯ ∗ = w ∗ R + w ♯ ∗ D + w ♯ ∗ N is a fixed point for the abstract gradient descent operator Φ ♯ if the following conditions are satisfied:

Algorithm 1: Abstract Learning

Input: abstract dataset D ♯ , learning rate η , regularization coeff. λ , transformation A w ∗ R ← closedFormReal ( D ♯ , λ ) w ♯ ∗ D ← closedFormSymb ( D ♯ , λ, w ∗ R ) Ξ ← genNonDataEq ( D ♯ , λ, η, w ∗ R , w ♯ ∗ D ) w ♯ ∗ N ← solveFixedPointEq (Ξ , λ, η ) w ♯ ∗ ← w ∗ R + w ♯ ∗ D + w ♯ ∗ N return w ♯ ∗

We summarize the fixed point construction process in Alg. 1. The algorithm takes as input an abstract dataset D ♯ , a learning rate η , a regularization coefficient λ , and a transformation matrix A . The construction order is: first w ∗ R , then w ♯ ∗ D , and finally w ♯ ∗ N , each step building on the previous results. Closed-form solutions for w ∗ R and w ♯ ∗ D exist, as detailed in Lem. I.4. The fixed point of Φ R matches the fixed point of gradient descent on the real part of the abstract data, independent of other parts. Given w ∗ R , w ♯ ∗ D is obtained

by solving a system of linear equations from Eq. (3), ensuring zonotope containment by equalizing the coefficients of the same error symbols, which is a sufficient condition for zonotope equivalence. This involves solving a system of m equations, where m is the number of error symbols in D ♯ .

To construct w ♯ ∗ N in Eq. (3), which involves zonotope equivalence, we encounter a challenge because the RHS and LHS have different error symbols, making symbolic enforcement of equivalence inapplicable. We exploit the properties of the order reduction operator to create a system of linear equations whose solution yields a fixed point for Φ ♯ N . Re-

Projection	A

Interval

Hull

Interval

Hull

Inverse

Projection	A

-1

Linearization

Transformation-based

Interval	Hull

Polynomial

Zonotope

call that this component uses linearization to handle non-linear terms and applies order reduction to efficiently manage the order of the resulting linear zonotope. As shown on the right side, the order reduction operator in Φ ♯ N utilizes a transformation matrix A to project the higher-order linear zonotope into a new space and then over-approximates it using an interval hull-a zonotope that is a box enclosing the input higher-order zonotope in the projected space-and finally projects this box back into the original space using A -1 . The core idea of transformation-based order reduction [38] is that interval hulls provide a better over-approximation in the projected space, where the shape of the input zonotope is closer to a box than in the original space, as shown in the graph on the right.

Φ

!

#

𝑥𝑥𝑥𝑥, 𝑥𝑥𝑥𝑥𝑥, 𝑥𝑥𝑥𝑥𝑥 ≃

#

𝑥𝑥𝑥𝑥𝑥

𝑤

!

∗

(known)

𝑤

#

#∗

(known)

𝑤

%

#∗

(unknown)

𝑘

!

𝑘

"

𝑘

!

𝑘

"

At a fixed point, the input and output interval hulls of Φ ♯ N should be equivalent in the projected space. This equivalence can be effectively enforced by checking for equal edge lengths along each dimension, which is sufficient for two interval hulls to be equivalent. This translates to a system of k equations, where k is the number of model parameters, hence can be done efficiently. We show that this system of equations is guaranteed to have a solution. We are now ready to state our main technical result.

Theorem 4.2 (Correctness of Alg. 1) . Given a set of possible training datasets D ⊙ associated with uncertain data D , and an appropriate abstraction function α in the zonotope abstract domain, and given a regularization coefficient λ , Alg. 1, when provided with D ♯ = α ( D ⊙ ) as input, computes abstract model parameters w ♯ ∗ such that:

where w ⊙∗ denotes the set of all optimal model parameters corresponding to D ⊙ for linear regression with ℓ 2 regularization with regularization coefficient λ .

5 Experiments

We implement ZORRO using SymPy [45], a Python library for symbolic computations and evaluate the system on two key applications: (1) computing prediction ranges and robustness certification for linear models trained on uncertain data, and (2) robustness of model weights for causal inference using linear models as a case study. We also measured the performance of ZORRO under varying conditions, including varying the degree of training data uncertainty. All our experiments are performed on a single machine with an Apple M1 chip, 8 cores, and 16 GB RAM. Experiments are repeated 5 times with different random seeds, and we report the mean (error bars denote 3 σ ). The code is shared at https://github.com/lodino/Zorro.

Datasets, Baselines, and Metrics. For robustness verification we use regression tasks: for MPG [58] (392 instances) we predict fuel consumption based on car features (cylinders, horsepower, weight); for Insurance [30] (1338 instances) we predict medical insurance charges based on demographics (age, gender, BMI), habits (smoking), and geographical features. We use a 80:20 train-test split and inject random errors to the training data varying (i) the Uncertain Data Percentage , the percentage of instances that have uncertain features / labels, and (ii) the Uncertainty Radius , the difference between the minimum and maximum possible value of an uncertain feature expressed as a fraction of the feature's domain. There is no direct baseline capturing prediction ranges for learning linear regression models from uncertain data. Most existing works in robustness certification focuses on test-time robustness (cf. Sec. 1). The exception is [47], which only supports uncertainty in labels using interval arithmetic. We compare ZORRO against this approach for robustness certification, referred to as MEYER in the following, only for training label uncertainty. For robustness verification, a prediction is robust if the size of the prediction interval is smaller than a given threshold. We use

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Ratio

Uncertainty Radius: 5 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 10 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 15 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 20 %

Uncertain Data Percentage (%)

ZORRO

Meyer et al.

(a) Robustness verification with uncertain labels (MPG data).

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Ratio

Uncertainty Radius: 4 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 6 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 8 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 10 %

Uncertain Data Percentage (%)

ZORRO

Meyer et al.

(b) Robustness verification with uncertain labels (Insurance data).

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Ratio

Uncertainty Radius: 6 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 8 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 10 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 12 %

Uncertain Data Percentage (%)

ZORRO

(c) Robustness verification with uncertain features on MPG data.

the robustness ratio which is the fraction of the test data receiving robust predictions as a metric in all robustness verification experiments. The robustness threshold is set to 5% of the label range for the MPG data, and 0.8% of the label range for the Insurance data. Additionally, we assess the worst-case test loss using certain test data and uncertain model weights trained from uncertain training data.

5.1 Robustness Verification

Prediction Robustness (Uncertain Labels). Fig. 1(a)(b) compares ZORRO with the baseline MEYER [47] on a setting where only training labels suffer from uncertainty. We vary the uncertainty radius and uncertain data percentage. As both systems provide sound over-approximations of prediction ranges, they may underestimate a model's robustness. As shown, ZORRO consistently certifies significantly higher robustness ratios than MEYER. This is due to the fact that MEYER uses interval-arithmetic which ignores the correlation between model weights in different dimensions and between the training labels and the weights, leading to overly conservative prediction ranges. Specifically, for higher uncertainty radius values, MEYER fails to certify robustness for most of the data while ZORRO still can certify robustness for 100% of the instances.

Prediction Robustness (Uncertain Features). We also evaluate the impact of training feature uncertainty (not supported by MEYER). Specifically, we introduce uncertainty into the vehicle weight column for the MPG dataset. As shown in Fig. 1(c), uncertainty in the features results in relatively less robust predictions compared to uncertain labels for a similar uncertainty radius (Fig. 1(a)). This is primarily because uncertain features result in more high-order terms in the closed form solution than uncertain labels which in turn leads to larger over-approximation errors during linearization.

In all experiments the standard deviation of the robustness ratio, calculated by repeating experiments with different random seeds, is large when the average robustness ratio is close to 0.5. This is because when the uncertain data percentage is low, the model will be robust no matter which training instances are selected to be uncertain. Likewise, when the uncertain data percentage is high, then most predictions will be uncertain no matter which training data points are uncertain.

Parameter Robustness. Next, we apply ZORRO for robustness certification of parameters in linear regression models, crucial for statistical estimation and causal analysis. We compare the ground truth coefficients for a treatment variable with the results obtained by ZORRO and through KNN imputation on a dataset with injected missing data. Fig. 2 shows the treatment variable coefficient and

0.50

0.55

Treatment Effect

(a) Consistent Positive Effects

-0.03

0.00

0.03

Intercept

-0.05

0.00

0.05

Treatment Effect

(b) Contradictory Effects

0.00

0.05

-0.50

-0.45

Treatment Effect

(c) Consistent Negative Effects

-0.03

0.00

0.03

ZORRO

GT

KNN

0.00

0.04

0.08

0.12

0.16

0.20

0.24

0.06

0.12

0.19

0.25

0.31

0.38

0.44

GT

Zorro

loss

uncertainty radius

(a) Varying uncertainty radius

0.07

0.08

0.09

0.10

0.11

0.12

0.13

5

6

7

8

9

10

11

12

13

GT

Zorro

loss

dimension

(b) Varying data dimension

the regression model's intercept. The intercept captures the baseline level of the outcome variable when all predictors are zero, highlighting how baseline values can shift under uncertainty. While the model trained after KNN imputation sometimes correctly identifies the directionality of the treatment effect, this is not always the case as shown in Fig. 2(b). This highlights the needs for techniques like ZORRO which guarantee that the true treatment effect is within certain bounds.

5.2 Solution Quality with Varying Uncertainty and Hyperparameters

We evaluate ZORRO's effectiveness by testing the tightness of the over-approximation and the accuracy of possible models. Specifically, we examine how the over-approximation quality and worst-case loss are influenced by the level of data uncertainty and the regularization coefficient.

Varying Data Uncertainty. To evaluate how specific characteristics of the data affect the effectiveness of ZORRO, we injected errors into real datasets, varying uncertain data percentage and uncertainty range. We compare ZORRO with the ground truth range of the loss computed by enumerating all possible worlds (GT). The results shown in Fig. 3(a) demonstrate that ZORRO tightly over-approximates the ground truth loss range, especially for smaller uncertainty radius values. As uncertainty increases, the over-approximation gap widens due to the increased coefficient of higherorder terms in the gradient, which are linearized, leading to higher linearization errors. As shown in Fig. 3(b), the tightness of ZORRO's over-approximation is not affected by the dimension of the data.

Effect of Regularization. We investigate the impact of the regularization coefficient on the robustness of predictions and the worst-case loss of possible models. Following a similar approach to Sec. 5.1, we introduce uncertainty in both features and labels in the MPG dataset and use zonotopes with varying levels of uncertainty to over-approximate the training data uncertainty. The results, shown in Fig. 4, indicate that a higher regularization coefficient leads to more robust predictions, as regularization tends to "compress" all possible model weights towards the origin. Interestingly, the worst-case loss shows that λ = 0 is not optimal across all scenarios, especially when the fraction uncertain instances is high. Instead, a small, positive λ (e.g., 0.02 or 0.025) generally yields the best worst-case losses. Combining these results, the optimal regularization coefficient should enhance robustness (i.e., a higher robustness ratio) while maintaining an acceptable worst-case loss. Therefore, the regularization coefficient should be tuned based on a validation dataset to achieve a small range of accurate possible models.

6 Conclusions, limitations and broader impacts

We introduce an approach for propagating uncertainty through model training and inference for linear models. Given an abstract uncertain training dataset that over-approximates the possible worlds of a training dataset, we develop abstract interpretation techniques to over-approximate the set of possible models and inference results for this set of models using zonotopes. This is challenging, as we need to compute fixed points of gradient descent in the abstract domain. Our main technical

0.00

0.02

0.04

0.06

2.20

2.40

2.60

Worst-case Loss

Uncertain Data: 8%

0.00

0.02

0.04

0.06

2.20

2.40

2.60

Uncertain Data: 10%

0.00

0.02

0.04

0.06

2.50

3.00

Uncertain Data: 12%

0.00

0.02

0.04

0.06

3.00

4.00

Uncertain Data: 14%

0.98

1.00

0.8

1.0

0.0

0.5

1.0

0.0

0.5

1.0

Robustness Ratio

contribution is the development of closed-form solutions for such fixed points that can be solved efficiently. Our techniques efficiently over-approximate models and inference for several use cases, including robustness verification, uncertainty management in causal reasoning, and improving the interpretability and reliability of predictions and inferences. This framework can be particularly valuable in critical applications where data quality and robustness are paramount. While we propose an effective method for abstract learning of linear models, extending our approach to more complex models is challenging. Non-linear models, such as neural networks, would require more advanced linearization and order reduction techniques, as well as parallelization, to manage the increased complexity of the involved symbolic operations. In addition, adapting our method to a broader classes of models through efficiently approximating the fixed points remains a challenging and promising future direction.

7 Acknowledgment

This research was supported by NSF awards IIS-2340124, IIS-2420691, and IIS-2420577, as well as by NIH grant U54HG012510. The views, opinions, and findings presented are those of the authors and do not necessarily represent those of the NSF or NIH.

References

Amr Alanwar, Anne Koch, Frank Allgöwer, and Karl Henrik Johansson. Data-driven reachability analysis using matrix zonotopes. In Learning for Dynamics and Control , pages 163-175. PMLR, 2021.

Matthias Althoff. Reachability analysis and its application to the safety assessment of autonomous cars . PhD thesis, Technische Universität München, 2010.

Matthias Althoff. Reachability analysis of nonlinear systems using conservative polynomialization and non-convex sets. In Proceedings of the 16th international conference on Hybrid systems: computation and control , pages 173-182, 2013.

Matthias Althoff and Bruce H Krogh. Zonotope bundles for the efficient computation of reachable sets. In 2011 50th IEEE conference on decision and control and European control conference , pages 6814-6821. IEEE, 2011.

Stanley Bak, Sergiy Bogomolov, Brandon Hencey, Niklas Kochdumper, Ethan Lew, and Kostiantyn Potomkin. Reachability of koopman linearized systems using random fourier feature observables and polynomial zonotope refinement. In International Conference on Computer Aided Verification , pages 490-510. Springer, 2022.

Aharon Ben-Tal, Dick den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Advanced Risk & Portfolio Management® Research Paper Series , 2011.

Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concerns, and solutions. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 850-863, 2022.

Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, et al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems , 3:747-769, 2021.

Leo Breiman. Heuristics of instability and stabilization in model selection. The annals of statistics , 24(6):2350-2383, 1996.

Christophe Combastel. Functional sets with typed symbols: Mixed zonotopes and polynotopes for hybrid nonlinear reachability and filtering. Automatica , 143:110457, 2022.

A Feder Cooper, Yucheng Lu, Jessica Forde, and Christopher M De Sa. Hyperparameter optimization is deceiving us, and how to stop it. Advances in Neural Information Processing Systems , 34:3081-3095, 2021.

P. Cousot. Abstract interpretation. CSUR , 28(2):324-328, 1996.

Patrick Cousot and Radhia Cousot. Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Proceedings of ACM SIGACT-SIGPLAN symposium on Principles of programming languages , pages 238-252, 1977.

Patrick Cousot and Nicolas Halbwachs. Automatic discovery of linear restraints among variables of a program. In Proceedings of the 5th ACM SIGACT-SIGPLAN symposium on Principles of programming languages , pages 84-96, 1978.

Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research , 23(226):1-61, 2022.

Fabricio Olivetti de Franca and Gabriel Kronberger. Prediction intervals and confidence regions for symbolic regression models based on likelihood profiles, 2022.

Nicolas Dewolf, Bernard De Baets, and Willem Waegeman. Valid prediction intervals for regression problems. Artificial Intelligence Review , 56(1):577-613, April 2022.

Ilias Diakonikolas, Gautam Kamath, Daniel M. Kane, Jerry Li, Ankur Moitra, and Alistair Stewart. Robustness meets algorithms. Commun. ACM , 64(5):107-115, apr 2021.

Ilias Diakonikolas and Daniel M. Kane. Recent advances in algorithmic high-dimensional robust statistics. CoRR , abs/1911.05911, 2019.

Tianyu Du, Shouling Ji, Lujia Shen, Yao Zhang, Jinfeng Li, Jie Shi, Chengfang Fang, Jianwei Yin, Raheem Beyah, and Ting Wang. Cert-rnn: Towards certifying the robustness of recurrent neural networks. CCS , 21(2021):15-19, 2021.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA , 10(2):455-482, 2021.

Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2018.

Matthew L Ginsberg. Counterfactuals. Artificial intelligence , 30(1):35-79, 1986.

Eric Goubault, Tristan Le Gall, and Sylvie Putot. An accurate join for zonotopes, preserving affine input/output relations. In Jan Midtgaard and Matthew Might, editors, Proceedings of International Workshop on Numerical and Symbolic Abstract Domains , volume 287, pages 65-76, 2012.

Cornelia Gruber, Patrick Oliver Schenk, Malte Schierholz, Frauke Kreuter, and Göran Kauermann. Sources of Uncertainty in Machine Learning - A Statisticians' View. arXiv , 2023.

Arthur E. Hoerl and Robert W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics , 42(1):80-86, 2000.

Chengqiang Huang, Zheng Hu, Xiaowei Huang, and Ke Pei. Statistical certification of acceptable robustness for neural networks. In Artificial Neural Networks and Machine Learning , pages 79-90. Springer, 2021.

Tomasz Imieli´ nski and Witold Lipski Jr. Incomplete information in relational databases. Journal of the ACM (JACM) , 31(4):761-791, 1984.

Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE symposium on security and privacy (SP) , pages 19-35. IEEE, 2018.

Arun Jangir and Willian Oliveira. Healthcare insurance, 2023. This dataset is licensed under CC0: Public Domain.

Jinyuan Jia, Xiaoyu Cao, and Neil Zhenqiang Gong. Intrinsic certified robustness of bagging against data poisoning attacks. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 7961-7969, 2021.

Matt Jordan, Jonathan Hayase, Alex Dimakis, and Sewoong Oh. zonotope domains for lagrangian neural network verification. In NIPS , 2022.

Bojan Karlas, Peng Li, Renzhi Wu, Nezihe Merve Gürel, Xu Chu, Wentao Wu, and Ce Zhang. Nearest neighbor classifiers over incomplete information: From certain answers to certain predictions. Proc. VLDB Endow. , 14(3):255-267, 2020.

Pasha Khosravi, Yitao Liang, YooJung Choi, and Guy Van den Broeck. What to Expect of Classifiers? Reasoning about Logistic Regression with Missing Features. arXiv , 2019.

Byol Kim, Chen Xu, and Rina Foygel Barber. Predictive inference is free with the jackknife+-afterbootstrap. In International Conference on Neural Information Processing Systems . Curran Associates Inc., 2020.

Niklas Kochdumper and Matthias Althoff. Sparse polynomial zonotopes: A novel set representation for reachability analysis. IEEE Transactions on Automatic Control , 66(9):4043-4058, 2020.

Niklas Kochdumper and Matthias Althoff. Constrained polynomial zonotopes. Acta Informatica , 60(3):279316, 2023.

Anna-Kathrin Kopetzki, Bastian Schürmann, and Matthias Althoff. Methods for order reduction of zonotopes. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC) , pages 5626-5633. IEEE, 2017.

Adrian Kulmburg and Matthias Althoff. On the co-np-completeness of the zonotope containment problem. Eur. J. Control , 62:84-91, 2021.

Sungyoon Lee, Hoki Kim, and Jaewook Lee. Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

David Lewis. Counterfactuals and comparative possibility. In IFS: Conditionals, Belief, Decision, Chance and Time , pages 57-85. Springer, 1973.

Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In International Conference on Machine Learning , pages 6765-6774. PMLR, 2020.

Peter McCullagh, Vladimir Vovk, Ilia Nouretdinov, Dmitry Devetyarov, and Alex Gammerman. Conditional prediction intervals for linear regression. In International Conference on Machine Learning and Applications , pages 131-138, 2009.

Johannes Mehrer, Courtney J Spoerer, Nikolaus Kriegeskorte, and Tim C Kietzmann. Individual differences among deep neural network models. Nature communications , 11(1):5725, 2020.

Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondˇ rej ˇ Certík, Sergey B Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, et al. Sympy: symbolic computing in python. PeerJ Computer Science , 3:e103, 2017.

Anna P. Meyer, Aws Albarghouthi, and Loris D'Antoni. Certifying data-bias robustness in linear regression. CoRR , abs/2206.03575, 2022.

Anna P Meyer, Aws Albarghouthi, and Loris D'Antoni. The dataset multiplicity problem: How unreliable data impacts predictions. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , pages 193-204, 2023.

Antoine Miné. The octagon abstract domain. Higher-order and symbolic computation , 19:31-100, 2006.

Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. In International Conference on Machine Learning , pages 3578-3586. PMLR, 2018.

Matthew Mirman, Alexander Hägele, Pavol Bielik, Timon Gehr, and Martin Vechev. Robustness certification with generative models. In ACM SIGPLAN International Conference on Programming Language Design and Implementation , pages 1141-1154, 2021.

Mark Niklas Müller, Marc Fischer, Robin Staab, and Martin Vechev. Abstract interpretation of fixpoint iterators with applications to neural networks. Proceedings of the ACM on Programming Languages , 7(PLDI):786-810, 2023.

Nicolas Müller, Daniel Kowatsch, and Konstantin Böttinger. Data poisoning attacks on regression learning and corresponding defenses. In Pacific Rim International Symposium on Dependable Computing (PRDC) , pages 80-89. IEEE, 2020.

Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

David J. Olive. Prediction intervals for regression models. Computational Statistics & Data Analysis , 51(6):3115-3122, 2007.

Andrea Paudice, Luis Muñoz-González, and Emil C Lupu. Label sanitization against label flipping poisoning attacks. In ECML PKDD Workshops , pages 5-15. Springer, 2019.

Robert J Plemmons. M-matrix characterizations. i-nonsingular m-matrices. Linear Algebra and its applications , 18(2):175-188, 1977.

Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics , 4(5):1-17, 1964.

R. Quinlan. Auto MPG. UCI Machine Learning Repository, 1993. DOI: https://doi.org/10.24432/C5859H. This dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Raymond Reiter. A theory of diagnosis from first principles. Artificial intelligence , 32(1):57-95, 1987.

Elan Rosenfeld, Ezra Winston, Pradeep Ravikumar, and Zico Kolter. Certified robustness to label-flipping attacks via randomized smoothing. In International Conference on Machine Learning , pages 8230-8241. PMLR, 2020.

Donald B Rubin. Multiple imputations in sample surveys-a phenomenological bayesian approach to nonresponse. In Proceedings of the survey research methods section of the American Statistical Association , volume 1, pages 20-34. American Statistical Association Alexandria, VA, USA, 1978.

Donald B Rubin. Multiple imputation after 18+ years. Journal of the American statistical Association , 91(434):473-489, 1996.

Anian Ruoss, Mislav Balunovic, Marc Fischer, and Martin Vechev. Learning certified individually fair representations. In Advances in Neural Information Processing Systems , pages 7584-7596, 2020.

Christian Schilling, Marcelo Forets, and Sebastián Guadalupe. Verification of neural-network control systems by integrating taylor models and zonotopes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8169-8177, 2022.

Soroosh Shafieezadeh-Abadeh, Peyman Mohajerin Esfahani, and Daniel Kuhn. Distributionally robust logistic regression. In Neural Information Processing Systems , 2015.

Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross J Anderson. Manipulating sgd with data ordering attacks. Advances in Neural Information Processing Systems , 34:18021-18032, 2021.

Daniele Silvestro and Tobias Andermann. Prior choice affects ability of bayesian neural networks to identify unknowns. arXiv preprint arXiv:2005.04987 , 2020.

Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev. Fast and effective robustness certification. Advances in neural information processing systems , 31, 2018.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems , 25, 2012.

Jacob Steinhardt, Pang Wei Koh, and Percy Liang. Certified defenses for data poisoning attacks. In NeurIPS , 2017.

Damien Teney, Maxime Peyrard, and Ehsan Abbasnejad. Predicting is not understanding: Recognizing and addressing underspecification in machine learning. In European Conference on Computer Vision , pages 458-476. Springer, 2022.

Ba-Hien Tran, Simone Rossi, Dimitrios Milios, and Maurizio Filippone. All you need is a good functional prior for bayesian deep learning. Journal of Machine Learning Research , 23(74):1-56, 2022.

Rens van de Schoot, Sarah Depaoli, Ruth King, Bianca Kramer, Kaspar Märtens, Mahlet G Tadesse, Marina Vannucci, Andrew Gelman, Duco Veen, Joukje Willemsen, et al. Bayesian statistics and modelling. Nature Reviews Methods Primers , 1(1):1, 2021.

Marianne Southall Winslett. Reasoning about action using a possible models approach . Number 1427-1432. Department of Computer Science, University of Illinois at Urbana-Champaign, 1988.

Margaux Zaffran, Aymeric Dieuleveut, Julie Josse, and Yaniv Romano. Conformal prediction with missing values. In International Conference on Machine Learning , pages 40578-40604. PMLR, 2023.

Xuezhou Zhang, Xiaojin Zhu, and Stephen Wright. Training set debugging using trusted items. In AAAI , 2018.

A Linear Regression and Ridge Regression

In this section, we review standard loss functions for linear regression, specifically mean squared error (MSE) and the loss function used in ridge regression. can be trained via either the closed-form solution or gradient descent. Specifically, the closed-form solution requires computing the inversion of the covariance matrix, while the gradient descent only involves matrix addition and multiplication. Since the linear regression model is convex, gradient descent is guaranteed to converge to the global optimum.

Suppose we have a training data D = ( X , y ) with n i.i.d. samples , where the feature matrix X = [ x 1 · · · x n ] T ∈ R n × d , and the labels y = [ y 1 , · · · , y n ] T ∈ R n . Given an input x and the model weight w , the prediction of the linear regression model ˆ y = w T x (the bias term can be integrated into w , corresponding to an added column in X with constant 1's).

Loss Functions The mean squared error (MSE) loss on D is defined as shown below.

In practice, regularization terms, e.g., based on the l p -norm of the model parameters, are often added to the original MSE loss to prevent overfitting by penalizing large weights. Using l 2 -regularization with a regularization coefficient λ which determines the strength of regularization is often called ridge regression . The loss function for ridge regression is:

Gradient Descent for Linear Regression Due to the convexity of linear models, the locally optimal point, which can be obtained by gradient descent with an appropriate learning rate η , is globally optimal. In gradient descent, the model weights w are iteratively updated (with some learning rate η ) towards the reverse direction of the gradient ∂L ( X , y , w ) ∂ w . Thus, for ridge regression:

Thus, one step of gradient descent Φ is:

Closed-form Solution for Linear Regression The convex nature of linear models ensures that the optimal weight w = w ∗ , which minimizes the loss L ( X , y , w ) , can be computed by establishing ∂L ( X , y , w ) ∂ w = 0 :

B Background on Abstract Interpretation

Abstract interpretation [13] is a technique for over-approximating the results of computations over a set of inputs. This is achieved by associating sets of elements of a concrete domain D with elements from an abstract domain D ♯ . In this context, various abstract domains are employed: interval domains represent variables as ranges of possible values [13], octagon domains allow for constraints between

pairs of variables within a specific bound [48], and polynomial zonotopes which represent convex polytopes using polynomial constraints [14]. Abstract interpretation, while originally designed for static program analysis such as strictness analysis, has also found applications in wide range of other domains including reachability analysis [1, 5, 2, 10, 36, 4], robustness verification for neural networks [32, 64], learning robust models by providing bounds on the loss for a set of inputs [51, 63, 49, 22], and many others.

We now state some important, but well-known, facts about abstract transformers that we utilize in our derivations.

B.1 Abstract Transformers

In Def. 2.5, we presented the standard definition of abstract transformers as functions over abstract domains that over-approximate the application of functions in the concrete domains to sets of elements. To clarify the connection between abstract transformers and possible world semantics observe that both take a concrete function F : D → D ′ and lift it to sets of inputs S ∈ P ( D ) through point-wise applications:

Thus, abstract interpretation is a natural fit for over-approximating PWS by over-approximating sets of possible worlds using abstract elements and then over-approximates computations with PWS for a function F using abstract transformers for such a function.

B.2 Abstract Transformers Compose

Importantly, abstract transformers compose. This enables us to decompose a complex computation into simpler operations and build an abstract transformer for the computation by composing abstract transformations for these simpler operations.

Proposition B.1 (Abstract Transformers Compose) . Consider (exact) abstract transformers f ♯ and g ♯ for functions f and g , then g ♯ ◦ f ♯ is an (exact) abstract transformer for g ◦ f .

C Symbolic and Standard Representation of Zonotopes

We now provide a more detailed account of the correspondence between the symbolic and standard geometric representation of zonotopes and polynomial zonotopes and discuss why matrices and sets of symbolic matrices as used in our abstract domain can equivalently be thought of as (polynomial) zonotopes.

C.1 Symbolic vs. Geometric Representation

We defined zonotopes z ♯ as vectors Ψ d where Ψ denotes polynomials over variables (called error symbols) E . The concretization of such a symbolic representation of a zonotope is the set of vectors in R d that can be derived from z ♯ by assigning values from [ -1 , 1] to each variable ϵ ∈ E . We encode such variable assignments as vectors [ -1 , 1] | E | .

Definition C.1 (Polynomial zonotopes - Symbolic Representation) . A d -dimensional polynomial zonotope is a vector z ♯ ∈ Ψ d [10]. Let m = | E | . The concretization of z ♯ is defined as:

A common measure of the representation size of a zonotope is its order. The order of a d -dimensional (polynomial) z ♯ in symbolic representation is the total number of distinct monomials in z ♯ divided by d :

where # M ( ψ ) denotes the cardinality of the set of distinct monomials in polynomial ψ .

A more common way to represent d -dimensional zonotopes is by fixing a set S of monomials over E and writing the zonotope as a central point c ∈ R d and a sum of generator vectors g i ∈ R d multiplied with the monomials from S . The generator vector g i assigns a coefficient to monomial S [ i ] for each of the d dimensions.

Definition C.2 (Polynomial zonotopes - Geometric Representation) . The geometric representation of a d -dimensional polynomial zonotope is a sum of a center point c ∈ R d and the monomials over error symbols in E from a set S multiplied by coefficients encoded in set of generator vectors g i ∈ R d :

For the geometric representation, the order of a d -dimensional (polynomial) z ♯ over monomials S is defined as:

Note that the order of the symbolic and geometric representation of a zonotope are the same. As an example consider the 4-dimensional polynomial zonotope z ♯ shown in symbolic representation (left) and standard representation (right).

C.2 Symbolic Matrices and Sets of Symbolic Matrices

We use matrices over symbolic expressions and sets of heterogeneous abstract matrices to represent the state of a computation and allow matrices from such a set to share variables to encode relationships between the elements of such matrices. The semantics we associate with such a set is that of a joint concretization . For S ♯ = { M i ♯ } , we define:

zonotopes encode convex sets of points in R d . We can think of a symbolic matrix M ♯ ∈ Ψ n × m as a n · m -dimensional polynomial zonotope. Similarly, a set of symbolic matrices S ♯ = { M i ♯ ∈ R n i × m i } can be thought of as a l -dimensional polynomial zonotope R l where l = ∑ i n i · m i . If every symbolic expression in a matrix or set of matrices is a linear combination of error symbols (an affine form), then such objects can equivalently be represented as zonotopes.

For instance, below on the left we show a zonotope matrix W ♯ (all expressions are linear) and two possible worlds in its concretization (for assignments [ -1 , -1] and [0 , 0 . 5] ).

C.3 Abstract Training Data and Model Weights

Using possible world semantics to compute all possible model weights w ⊙∗ given a set of possible training datasets D ⊙ , there will be a natural correspondence between a model weight w ∈ w ⊙∗ and the dataset D ∈ D ⊙ from which is was derived. When training in the abstract domain such correlations can be preserved by sharing error symbols between the abstract training data D ♯ and corresponding model weights w ♯ . When such sharing occurs, then it is critical to reason about the joint concretization when determining whether an abstract fixed point as been achieved. Testing equivalence (equal concretization) of w ♯ alone can lead to false positives, as w ♯ 1 ≃ ♯ w ♯ 2 does not in general imply ( D ♯ , w ♯ 1 ) ≃ ♯ ( D ♯ , w ♯ 2 ) . This is due to the fact that a particular concrete model weight w ∈ γ ( w ♯ 1 ) = γ ( w ♯ 2 ) may be associated with different datasets in ( D ♯ , w ♯ 1 ) and D ♯ , w ♯ 2 ) because of shared error symbols between D ♯ and the abstract model weights. Thus,

concretization equivalence between model weights only does not imply equivalent results after application of a gradient decent step.

To further illustrate this consider, two pairs of abstract model weights w ♯ 1 and w ♯ 2 .

The only difference between w ♯ 1 and w ♯ 2 is that w ♯ 1 shares a variable ( ϵ 1 ) with D ♯ while w ♯ 2 does not. Observe that γ ( w ♯ 1 ) = γ ( w ♯ 2 ) . However, applying a step of gradient decent to w ♯ 1 and w ♯ 2 may lead to abstract models that are not equivalent. For sake of this example, consider a hypothetical gradient operator Φ ♯ dummy that subtracts the data from the current model weight.

Note that Φ ♯ dummy ( w ♯ 1 ) ̸≃ ♯ Φ ♯ dummy ( w ♯ 2 ) . Thus, even if two abstract model weights have equal concretization this does not guarantee that a fixed point has been reached when also taking the data into account. This is important as otherwise it is not possible that the abstract model weights contain all possible optimal model weights w ⊙∗ . Instead, we need to consider the joint concretization of model weight and data, s.t. if γ ( S ♯ 1 ) = γ ( S ♯ 2 ) then concretization equivalence is guaranteed to hold for all subsequent iterations.

D Examples Of Training Data Uncertainty and Abstraction

In this section, we discuss several causes of training (and test) data uncertainty, how to encode them as possible worlds, and abstraction functions for approximating such uncertainty in our symbolic model.

D.1 Measurement Uncertainty

Sensors typically have some measurement uncertainty. If the measurement error ϵ is know, e.g., provided by the instrument manufacturer or estimated through repeated measurements and calibration, then for a set of sensor readings used in training D = { ( x i , y i ) } , then each possible world in D ⊙ is derived from D by replacing values x i [ j ] with values in x i [ j ] ± ϵ .

D.2 Missing Values and Imputation

Consider a training dataset D where some of the features are missing for some of the datapoints in D . If each feature X i 's domain is an interval [ l i , u i ] and assuming that missing values can be represented as independent random variables, then the set of possible worlds of the training data D ⊙ are all training datasets that can be derived from D by replacing each missing value in a feature X i with a value from [ l i , u i ] .

We can use an abstraction function α missing that represents each missing value as an interval [ l i , u i ] . In the symbolic representation, this is encoded as the central point of the interval of an error symbol ϵ with a coefficient half of the interval's length. We associate a separate error symbol with each missing value. For D ∈ R n × m , we define α missing for i ∈ [1 , n ] and j ∈ [1 , m ] as:

For instance, consider the training dataset D ∈ R 3 × 2 shown below where ⊥ denotes a missing value.

Assuming that both features have a domain [0 , 10] , we get:

If the independence assumption on missing values holds, then γ ( α missing ( D )) = D ⊙ . If the independence assumption does not hold, then D ⊙ would be a subset of the worlds described above and α missing ( D ) is a still a valid abstraction function, albeit an over-approximating one:

Imputation If we make the stronger assumption the unknown ground truth value corresponding to a missing value is in a set of estimations { a 1 , . . . , a m } returned by a set of imputation methods, then we can use [ min ( { a i } ) , max ( { a i } )] instead of [ l i , u i ] .

Training Label Uncertainty The abstract transformer α missing can also be used if training data labels are missing.

E Abstract Transformers for Zonotopes

In this section we introduce (exact) abstract transformers for (polynomial) zonotopes that have been introduced in related work and discuss their computation complexity and the space requirements for the output zonotope z ♯ in terms of its order ORD ( z ♯ ) (see Sec. 2.2). Kochdumper et al. [37, Table 1] shows an overview of which operations are exact for linear and polynomial zonotopes (and other set representations). Relevant to for our purpose is that exact transformers exist for polynomial zonotopes for all operations used in the learning algorithms for linear models we consider.

Proposition E.1 (Exact Transformers for Polynomial Zonotopes) . There exist exact abstract transformers for scalar addition and multiplication as well as for matrix addition and multiplication for polynomial zonotopes [3]. There exist exact transformers for scalar addition and matrix addition for zonotopes. Abstract transformers for multiplication and matrix multiplication for zonotopes exist, but are not exact.

We present the details of these operations in the following.

E.1 Arithmetic Operations

Addition. Scalar and matrix addition are exact in both zonotopes and polynomial zonotopes. Given two matrix (polynomial) zonotopes V ♯ and W ♯ in Ψ n × m , their addition Z ♯ = V ♯ + W ♯ is defined by adding entries. For each i ∈ [1 , n ] and j ∈ [1 , m ] :

The order of Z ♯ is the sum of the orders of V ♯ and W ♯ :

Scalar multiplication. Multiplying a (polynomial) zonotope matrix W ♯ with a scalar c is exact using the abstract transformer · poly defined below. We simply multiply each entry in the matrix by c .For each i ∈ [1 , n ] and j ∈ [1 , m ] :

Multiplication with a scalar abstract value d ∈ Ψ is exact for polynomial zonotope matrices, but increases the order of the input zonotope:

Here we define the order of a scalar abstract value d ∈ Ψ to be the number of monomials in the representation of d . For a linear zonotope W , d · W ♯ ij is in general not linear as it contains higherorder terms. Thus, matrix multiplication for linear zonotopes requires application of linearization (see App. G):

Matrix multiplication. Matrix multiplication is defined using scalar multiplication and addition. As discussed above, addition is exact for both linear and polynomial zonotopes. However, scalar multiplication of symbolic expressions is only exact for polynomial zonotopes, but requires linearization and, thus, over-approximation, for linear zonotopes. That is, matrix multiplication is exact for polynomial zonotopes only. Consider two matrices V ♯ ∈ Ψ n × m and V ♯ ∈ Ψ m × k , then matrix multiplication is defined as usual, but using abstract transformers for scalar addition and multiplication. Each symbolic entry in the matrix V ♯ · W ♯ is a sum of m elements, each the multiplication of one entry of V ♯ with one entry of W ♯ . Thus,

For linear zonotopes, we have to again apply linearization to make sure that the output is a linear zonotope.

F Abstract Transformers for Gradient Descent

F.1 Abstract Fixed Points Over-Approximate Possible Fixed Points

Proof of Prop. 3.3. Initially, we will assume that w ♯ j is computed through repeated application of F ♯ . Let n be the smallest number such that w ♯ ∗ = w ♯ n = F ♯ ( w ♯ n -1 , D ♯ ) for iteration with abstract transformer F ♯ . As F ♯ is an abstract transformer for Φ and abstract transformers compose (Prop. B.1), we know that for every D i ∈ D ⊙ and n ∈ N , we have ( w n i , D i ) ∈ γ ( w ♯ n , D ♯ ) . To prove the claim it is sufficient to show that for every such D i we have ( w ∗ i , D i ) ∈ γ ( w ♯ n , D ♯ ) . WLOG consider some D i ∈ D ⊙ and ( w n i , D i ) ∈ γ ( w ♯ n , D ♯ ) . Let m ∈ N be the smallest number such that w ∗ i = w j m . If m ≤ n , then based on the fact that F ♯ is an abstract transformer (over-approximates Φ ), the result holds as ( w n i , D i ) ∈ γ ( w ♯ n , D ♯ ) . Now consider the case where m>n . We will show through induction that for all j ∈ [ n +1 , m ] , ( w j i , D i ) ∈ γ ( w ♯ n , D ♯ ) and, thus ( w ∗ i , D i ) ∈ γ ( w ♯ n , D ♯ ) .

Induction start . For j = n the result trivially holds based on the definition of abstract fixed points.

Induction step . Assume that ( w j i , D i ) ∈ γ ( w ♯ n , D ♯ ) for j ∈ [ n, m -1] , we have to show that this implies that ( w j +1 i , D i ) ∈ γ ( w ♯ n , D ♯ ) . By definition, we have

As F ♯ is an abstract transformer for Φ , we have, ( w j +1 i , D i ) ∈ γ ( w ♯ n +1 , D ♯ ) . Now based on the fact that w ♯ n is an abstract fixed point according to Def. 3.2 and, thus, γ ( w ♯ n , D ♯ ) ⊇ γ ( w ♯ n +1 , D ♯ ) , it follows that ( w j +1 i , D i ) ∈ γ ( w ♯ n , D ♯ ) .

So far we have demonstrated that a fixed point w ♯ ∗ that appears in the iteration sequence { w ♯ j } ∞ j =0 contains all optimal model weights w ⊙∗ . We now prove the stronger result that as long as w ♯ ∗ fulfills the condition of Def. 3.2, no matter it is the result of an iteration sequence using F ♯ or not, its concretization encloses w ⊙∗ . Consider one D i ∈ γ ( D ♯ ) and as above let w ∗ i denote its optimal model weight. We will demonstrate that ( w ∗ i , D i ) ∈ γ ( w ♯ ∗ , D ♯ ) . First note that given that gradient descent for linear models is convex, for any initial model weight w 0 i , the sequence { w j i } ∞ j =0 converges to w ∗ i . Specifically, let w denote the model weight such that ( w , D i ) ∈ γ ( w ♯ ∗ , D ♯ )

and let n i denote the smallest integer such that w n i i = w ∗ i for the sequence generated starting from w 0 i = w . However, now we can apply the same proof by induction shown above to demonstrate that ( w n i i , D i ) ∈ γ ( w ♯ ∗ , D ♯ ) . This concludes the proof.

F.2 Exact Abstract Transformer for Gradient Descent

In this section we show that the abstract transformer for gradient descent introduced in sec. 3 has a fixed point.

Proposition F.1. The abstract gradient descent operator Φ ♯ has a fixed point w ♯ ∗ .

Proof of Prop. F.1. The existence of a fixed point is implied by the fact that Φ ♯ is an exact abstract transformer of the concrete gradient descent operator Φ . Consider γ ( w ♯ 0 , D ♯ ) = { ( w 0 i , D i ) } . Let n i be the smallest integer such that w n i i = w ∗ i and let n = max i n i , i.e., at iteration n , the concrete model weights have converged in every possible world in the concretization of ( w ♯ 0 , D ♯ ) . As Φ ♯ is an exact abstract transformer, we can show by induction that γ ( w ♯ j , D ♯ ) = { ( w j i , D i ) } for any j . As all computations in the concretization have converged at n , we know that w n +1 i = Φ( w n i ) = w n i and, thus, { ( w n i , D i ) } = { ( w n +1 i , D i ) } . As Φ ♯ is exact, we get the desired result: γ ( w ♯ n , D ♯ ) = { ( w n i , D i ) } = { ( w n +1 i , D i ) } = γ ( w ♯ n +1 , D ♯ ) .

F.3 Prediction with Abstract Model Weight Fixed Points

We now discuss how to use the abstract model weights w ♯ ∗ returned by our abstract transformer for learning linear models during inference to over-approximate the possible set of predictions for a test data point. We start by discussing test data that is not uncertain and then extend the discussion to the case where the test data is also uncertain.

Deterministic Test Data For now let us assume that the test data is not uncertain. The following corollary then enables us to use abstract gradient descent for inference and for over-approximating the prediction ranges.

Corollary F.2. Let f w ( x ) denote the linear model for parameters w . Given an incomplete training dataset D associated with a set of possible worlds D ⊙ , the prediction range V ( x ) for a test data point x can be over-approximated by:

where w ♯ ∗ is the fixed point of the abstract gradient descent operator Φ ♯ applied to D ♯ = α ( D ⊙ ) , the abstract representation of D ⊙ in the zonotope domain.

The prediction returned by a linear model with parameters w for a data point x is w T x . As x does not contain any symbolic term this is the sum of linear terms multiplied by constants, i.e., the result is a 1-dimensional linear zonotope which is a linear expression of the form:

where c i ∈ R and ϵ i ∈ E . The minimum and maximum value of a 1-d linear zonotope can be determined efficiently as shown below:

Uncertain Test Data In Sec. 2 we modeled uncertainty in the training data as sets of possible worlds D ⊙ . As mentioned in that section, our techniques also supports uncertain test data, i.e., both the training and test data may be uncertain.

In the most general case, the training and test data may be correlated. 1 We model this as a set of possible worlds ( D ⊙ , X test ⊙ ) where each world is a pair of a training dataset D i and a test dataset X test i :

If training and test data are independent, then we can specify their worlds separately ( D ⊙ and X test ⊙ ) and assume the worlds of ( D , X test ) ⊙ to be their cross product. Uncertainty propagation for inference then requires us to compute the set of possible predictions:

For inference in the abstract domain we first have to select an appropriate abstraction function for the test data, e.g., using the same abstraction function α we use for the training data. The only difference to the case discussed above is that a test data point is now also an abstract element x ♯ and the prediction w ♯ x ♯ is a polynomial zonotope as it contains higher order terms that are the result of multiplying linear terms. To efficiently determine the minimum and maximum we can, e.g., employ linearization to map the polynomial zonotope into a linear one and apply the solution described above for finding the minimum and maximum of a linear zonotope.

G Linearization and Order Reduction Techniques

G.1 Linearization

The purpose of linearization is to over-approximate an input polynomial zonotope with a linear zonotope.

Definition G.1 (Linearization Operator L ) . A linearization operator L maps a polynomial zonotope z ♯ to a linear zonotope ℓ ♯ . It replaces high-order polynomial terms with new error symbols, ensuring that all expressions are linear while maintaining an over-approximation:

During inference, computing the prediction intervals using the linear zonotope representation can be done efficiently with linear programming. However, more importantly, in our construction of abstract fixed points we use a specific order reduction technique that requires prior linearization in each gradient descent step to enforce the existence of a fixed point. For a d -dimensional polynomial zonotope z ♯ with a set of monomials S :

we have its linearization:

where σ l denotes the set of indices of all linear terms in z ♯ . Here, the third part over-approximates each high-order term in z ♯ by replacing it with a new error symbol.

G.2 Order Reduction Operators

Order reduction operators are used to reduce the representation size of a zonotope. That is, an order reduction operator takes as input a linear zonotope ℓ ♯ and return a linear zonotope of smaller order

(smaller representation size) that over-approximates ℓ ♯ . For linear zonotopes this means that order reduction operators reduce the number of distinct error symbols that occur in a zonotope by merging error symbols.

Definition G.2 (Order Reduction Operator R ) . An order reduction operator R takes a linear zonotope ℓ ♯ as input and returns another linear zonotope of reduced order:

We now present details about two commonly adopted order reduction techniques: Interval Hull ( IH ) and transformation-based Interval Hull ( TIH ) in App. G. In this section, we use the geometric representation of zonotopes (see Def. C.2). Note that for a linear zonotope, each S [ i ] consists of a single error term ϵ i . Thus, we can write such a zonotope ℓ ♯ as:

IH merges a selected subset δ s of the error symbols of a zonotope and their corresponding generator vectors. For a d -dimensional zonotope ℓ ♯ :

The selected terms δ s are merged into a d -dimensional box described by d new error symbols { ϵ ′ 1 , · · · , ϵ ′ d } . We will drop δ s if all error symbols of the input zonotope are selected.

The error symbols getting merged ( δ s ) are often determined based on some heuristic, e.g., symbols with lowest coefficients. TIH R TIH first projects the zonotope to another space using an invertible linear transformation matrix A ∈ R d × d , then applies IH, and finally projects the resulting zonotope back with A -1 . IH is a special case of TIH where A = I .

Intuitively, the purpose of the linear transformation A is to project the zonotope to a space where its shape is closer to a box, to reduce the loss of precision brought by order reduction [2, 38]. One of the most widely used TIH is PCA-based order reduction, whose transformation is obtained from the PCA of the set of all generator vectors of the input zonotope [38, 51].

Order Reduction for PTIME Zonotope Training With linearization, the number of error symbols in the model weights zonotope w ♯ still grows exponentially with rate O ( p 2 ) , leading to exponential time complexity for gradient descent. We can overcome this challenge through order reduction , which enforces the maximum number of terms in the symbolic expressions of model weights zonotopes [38]. Note that for a linear zonotope, each S [ i ] consists of a single error term ϵ i . Thus, we can write such a zonotope ℓ ♯ as:

Order reduction R reduces the order (representation size) of a linear zonotope ℓ ♯ . This is achieved by merging error symbols in S , while ensuring that the result over-approximation the input zonotope, i.e.,

Two commonly adopted order reduction approaches [38] are Interval Hull ( IH ), denoted as R IH , and Transformation-based Interval Hull ( TIH ), denoted as R TIH [2, 38, 64]. Specifically, IH merges a set of error symbols δ s ⊆ { ϵ i } and their corresponding generator vectors (here δ k = { ϵ i } -δ s ):

The selected terms δ s are merged into a d -dimensional box described by d new error symbols { ϵ ′ 1 , · · · , ϵ ′ d } . The error symbols getting merged ( δ s ) are often determined based on some heuristic [38], e.g., the symbols with lowest coefficients. Similar to IH, TIH also merges error terms using IH, but in some projected space. TIH first projects the zonotope to another space using an invertible linear transformation matrix A ∈ R d × d , then conducts IH, and finally projects the resulting zonotope back into the original space with A -1 :

Intuitively, the linear transformation A aims to project the zonotope to a space where its shape is closer to a box, to reduce the loss of precision brought by order reduction [2, 38]. One of the most widely used TIH is PCA-based order reduction, whose transformation is obtained from the PCA of all generator vectors [38, 51].

H Efficient Abstract Gradient Descent with Order Reduction

As mentioned in sec. 4.1, to address the tractability issues with abstract gradient descent, we employ two key techniques: linearization and order reduction. We employ linearization at each step of gradient descent to ensure that the resulting abstract representation of model parameters remains a linear zonotope.

Given a linearization operator L and order reduction operator R , we construct an abstract gradient descent operator, Φ ♯ :

which ensures that the abstract representation size remains bounded while providing efficient overapproximation. This operator is an abstract transformer for the concrete gradient descent operator.

Proposition H.1. For any linearization L and order reduction R , the abstract gradient descent Φ ♯ is an abstract transformer for the concrete gradient descent operator Φ . Formally, for any abstract w ♯ ,

.

Proof. Given that abstract transformers compose (Prop. B.1), we can decompose Φ into separate steps and construct an abstract transformer for Φ by composing abstract transformers for the individual steps. We can inject identity functions anywhere into the computation without changing its result. Let ident represent an identify function of an appropriate type and F ♯ be the abstract operator resulting from injecting identity functions as shown below, then:

Since Φ ♯ exact is an abstract transformer for Φ , so is F ♯ . Now observe that both L and R are abstract transformers for ident as

Thus, Φ ♯ is an abstract transformer for Φ .

Note that Prop. 3.3 then implies that a fixed point w ♯ ∗ for Φ ♯ is an over-approximation of all possible model weights w ⊙∗ :

H.1 Abstract Gradient Descent With Order Reduction And Fixed Points

While Φ ♯ ensures that every step of gradient descent can be computed efficiently as we bound the order of the resulting zonotope in each step, this operator typically does not have a fixed point and even if it does, we still would have to solve an NP-hard problem [39] to detected that we have converged. The reason for the lack of a fixed point is that both linearization and order reduction results in over-approximation and the over-approximation error may grow in each iteration. Recall that we did show a real example of where this operator diverges in Sec. 4.1.

I An Efficient Approximate Abstract Transformer for Ridge Regression

In this section and the next section we present the proof of our main technical result: Thm. 4.2. Recall that Thm. 4.2 states that Alg. 1 computes abstract model parameters w ♯ ∗ that are a fixed point for the abstract transformer Φ ♯ using a closed form solution. Because Φ ♯ was shown to be an abstract transformer for gradient decent this then implies that w ♯ ∗ over-approximates all possible model weights w ⊙∗ . We start by presenting additional details of the decomposition we employ to force a fixed points, formally prove that the condition on w ♯ ∗ from Prop. 4.1 is a sufficient for w ♯ ∗ being a fixed point for Φ ♯ , and then develop a closed form solution that requires solving a system of linear equations. For w ♯ N , the parts of the abstract model weights that exclusively contains symbols that do not appear in the abstract training dataset D ♯ , the equation system only has a solution if the regularization coefficient λ is larger then or equal to a constant β that depends on D ♯ . Based on our experience and extensive experimental evaluation, we typically have β = 0 , i.e., the closed form solution exists for any regularization coefficient λ . Nonetheless, we present a technique for achieving any desired β ≥ 0 by splitting zonotopes into smaller parts with lower β , computing fixed points for each split individually, and merging the final result. These techniques will be presented in app. J.

I.1 Decomposing Fixed Point Equations For Abstract Gradient Descent

We now present additional details about our decomposition of the gradient from Sec. 4.1 for linear regression with ℓ 2 regularization (ridge regression). The loss function L for ridge regression an ℓ 2 penalty, is given by:

where λ is the regularization coefficient (see App. A for details).

Recall that we observed that an abstract model weight zonotope w ♯ can be decomposed into parts that can be dealt with separately in the sense that we will show that a sufficient condition for achieving a fixed point w ♯ ∗ is that each component has a fixed point. Given an abstract training dataset D ♯ , we use X ♯ and y ♯ to denote its feature matrix and labels. We decompose them into real (concrete) and symbolic components:

where X R ∈ R n × d and y R ∈ R n represent the real centers of zonotopes X ♯ and y ♯ , while X ♯ S ∈ Λ n × d and y ♯ S ∈ Λ n contain the symbolic terms.

Similarly, we decompose the abstract model weights at iteration i into real and symbolic components:

where w i R ∈ R d represents the real center, and w ♯ i S ∈ Λ d contains the symbolic terms. The symbolic terms are further decomposed into those containing data symbols ( w ♯ i D ), i.e., symbols that are shared with D ♯ , and those introduced via linearization and order reduction in linear abstract gradient descent ( w ♯ i N ), i.e., that do not appear in D ♯ .

Accordingly, we decompose the abstract gradient presented in Eq. (9) into several distinct components: real numbers ( G R ), linear symbolic expressions that share symbols with D ♯ ( G D L ), linear symbolic expressions that do not share symbols with D ♯ ( G N L ), and high-order symbolic expressions ( G H ). Using the loss function for ridge regression (see Eq. (5) in App. A):

where

In Eq. (10) we substitute definitions which are further expanded in Eq. (11) using new notation for rearranged parts of the gradient (Eq. (12) to (15)). Furthermore, in this step we also use the fact that linearization only affects higher order terms and G H is the only component of the gradient with non-linear terms. Now consider the relationship of Eq. (12) to (15) to

Enforcing Fixed Points With Parameterized Order Reduction. We now introduce an order reduction operator R A that is a specific type of TIH (see App. G) which only merges non-data symbols and is parameterized by its transformation matrix A , identify a sufficient condition for achieving an abstract fixed point for gradient descent with this order reduction operator, and demonstrate that the fixed point exists for any coefficient λ of l 2 -regularization that is larger than a data-dependent constant β . Furthermore, we show how such a fixed point w ♯ ∗ can be computed using closed form solutions for w R , w ♯ D and w ♯ N .

Recall from App. G that the TIH order reduction operator R TIH ( δ s , A ) is parameterized by δ s (the set of error symbols to merge) and a transformation matrix A . Consider the input to order reduction as shown in Eq. (10).We use E D to denote the error symbols that this input shares with the data (that occur in D ♯ ) and E N to denote those that only occur in the input to order reduction. Then we define R A = R TIH ( E N , A ) , i.e., we only merge symbols that are not shared with the data to ensure that the correspondence between model weights and possible worlds is preserved by letting w ♯ share symbols in linear expressions with D ♯ . The linear part that does not share symbols with D ♯ has O ( p 2 q ) monomials (where p = | E D | and q is the number of error symbols in w ♯ ). Now expanding the definition of abstract gradient descent with order reduction using R A as the order reduction operator (for some given transformation matrix A ), we get:

Eq. (18) shows the components of Φ ♯ ( L , R A ) ( w ♯ ) and related them to the notation Φ R ( w R ) , Φ ♯ D ( w R , w ♯ D ) , and Φ ♯ N ( w R , w ♯ D , w ♯ N ) we introduced in Eq. (2). Specifically, Φ R ( w R ) = ( w R -η G R ) does not contain any error symbols and, thus, corresponds to the real number part of the updated model weight. Similarly, Φ ♯ D ( w R , w ♯ D ) = ( w ♯ D -η G D L ) only contains linear symbolic expression with symbols that are from D ♯ . The remaining part, Φ ♯ N ( w R , w ♯ D , w ♯ N ) =

A ( R IH ( A -1 ( w ♯ N -η ( G N L + L ( G H )))) ) does not share symbols with D ♯ , as it consists of new symbols generated from order reduction.

I.2 Constructing Fixed Points For Abstract Gradient Descent

Having defined Φ ♯ ( L , R A ) which was denoted by Φ ♯ ( L , R ) in app. H, we are now ready to state the first part of our main technical result: under some mild assumptions on λ being larger than some datadependent constant β , Φ ♯ ( L , R A ) has a fixed point and, importantly, this fixed point can be computed efficiently. In practice, we often have β = 0 . However, if this is not the case, we demonstrate later that at the cost of reduced computational efficiency we can reduce β by splitting D ♯ into several zonotopes and solving the problem independently for each zonotope as we will details in App. J.

Theorem I.1 (Existence of Abstract Fixed Points) . Consider an abstract training dataset D ♯ . There exists a constant β specific to D ♯ such that Φ ♯ ( L , R A ) has a fixed point for any λ ≥ β .

In the remainder of this section, we prove Thm. I.1 by demonstrating how to construct such a fixed point efficiently. We start by identifying a sufficient condition for w ♯ ∗ to be a fixed point, then demonstrate that the problem of finding a w ♯ ∗ that fulfills this sufficient condition can be decomposed based on our decomposition w ♯ ∗ = w ∗ R + w ♯ ∗ D + w ♯ ∗ N presented earlier. Specifically, certain components are independent of other components suggesting an evaluation order where we find a fixed point for one component treating the previously computed fixed points for other components as constants. Then we proceed to prove closed form solutions for w ∗ R and for w ♯ ∗ D given w ∗ R . Finally, given w ∗ R and w ♯ ∗ D we construct a system of equations whose solution gives a fixed point for w ♯ ∗ N and demonstrate that this system of equations has a closed form solution for any λ ≥ β .

)

A Sufficient Condition for Abstract Fixed Points. Recall from Def. 3.2 that the fixed point w ♯ ∗ must satisfy γ ( w ♯ ∗ , D ♯ ) ⊇ γ ( Φ ♯ ( L , R A ) ( w ♯ ∗ ) , D ♯ ) . We now present a sufficient condition for this to hold. Intuitively this condition requires the invariance of different components of Φ ♯ ( L , R A ) ( w ♯ when doing joint concretization with D ♯ .

Proposition I.2. If an abstract model weight w ♯ ∗ = w ∗ R + w ♯ ∗ D + w ♯ ∗ N satisfies the following three conditions, then it is an abstract fixed point of the abstract gradient descent operator Φ ♯ ( L , R A ) :

Proof. Consider the definition of an abstract fixed point w ♯ ∗ ( Def. 3.2 applied to Φ ♯ ( L , R A ) ) :

This can certainly be achieved if

This requires equal joint concretization of the fixed point the pairs ( w ♯ ∗ , D ♯ ) and (Φ ♯ ( L , R A ) ( w ♯ ∗ ) , D ♯ ) . To understand the need for joint concretization here note that certain model weights w paired with certain datasets D in the joint concretization. If we would only consider equivalence of the abstract model weight, two zonotopes w ♯ and w ♯ ′ may be equivalent w ♯ ≃ ♯ w ♯ ′ , but pair a particular model weight w ∈ γ ( w ♯ ) = γ ( w ♯ ′ ) with different datasets D 1 ∈ γ ( D ♯ ) and D 2 ∈ γ ( D ♯ ) . Thus, it is possible that w ♯ ≃ ♯ Φ ♯ ( L , R A ) ( w ♯ ) holds but γ ( w ♯ ) = γ ( w ♯ ′ ) does not contain all model weights from w ⊙∗ .

First observe that requiring equality of abstract elements is a stricter condition than equivalence wrt. ≃ ♯ as

Also note that w ♯ D is the only component of w ♯ ∗ that contains symbols. Thus, requiring equality in Eq. (20) is sufficient for ensuring equivalent join concretization with D ♯ as long as Eq. (19) and Eq. (21) also hold. Furthermore, Eq. (19) and Eq. (20) imply:

Expanding w ♯ ∗ = w ∗ R + w ♯ ∗ D + w ♯ ∗ N and substituting into Eq. (18) we get

Thus, using Eq. (21) and the fact that w ♯ N does not share any symbols with D ♯ , we get the desired result:

Independence of Fixed Point Components. Recall that we observed that some components of the gradient depend only on some components of w ♯ ∗ . Specifically, G R only depends on w ∗ R and G D L only depends on w ∗ R and w ♯ ∗ D . This implies that we can find a fixed point by first computing w ∗ R , then finding w ♯ ∗ D treating w ∗ R as a constant, and finally determine w ♯ ∗ N treating both w ∗ R and w ♯ ∗ D as a constant.

Lemma I.3 (Independence of Fixed Point Components) . The following independence relationships hold:

Proof. The definition of G R is shown in Eq. (12). The only component of w ♯ ∗ that appears in the definition of G R is w ∗ R . Thus, the equations defining w ∗ R only depend on w ∗ R and are independent of w ♯ ∗ D and w ♯ ∗ N . Using a similar argument we can show that G D L does not depend on w ♯ ∗ N .

Closed Form Solutions for Real And Linear Data Components of Abstract Model Weights. Before explaining how to determine w ♯ ∗ N , utilizing the lemma above we derive closed form solutions for w ∗ R and w ♯ ∗ D .

Lemma I.4 (Closed Form Solutions for w ∗ R and w ♯ ∗ D ) . Given D ♯ , w ∗ R as defined below fulfills Eq. (19)

Given D ♯ and w ∗ R , w ♯ ∗ D as defined below fulfills Eq. (20) .

Proof. Fixed Point for w ∗ R . Substituting the definition of G R from Eq. (12) into Eq. (21):

The last step relies on the fact that X R T X R + λnI is invertible which is guaranteed for λ > 0 [26]. Note that, as expected since the fixed point equation for w ∗ R does not contain any symbolic expressions, the closed form for w ∗ R is equal to the well-known closed form for ridge regression. Fixed Point for w ♯ ∗ D . Note that given the independence of w ∗ R from w ♯ ∗ D , we can assume that w ∗ R has been determined using the closed form solution shown above. Thus, we can treat w ∗ R as a constant in the following derivation. We start by substituting the definition or Eq. (13) into Eq. (20).

Note that matrix inversion is applied to ( X R T X R + λnI ) which does not contain any symbolic terms and as discussed above this matrix is guaranteed to be invertible.

Analysis of the Fixed Point Equations for w ♯ ∗ N . Using the closed form solutions established in Lem. I.4, we can compute w ∗ R and w ♯ ∗ D fulfilling Eq. (19) and Eq. (20). In the remainder of this subsection we show how to compute a solution w ♯ ∗ N to Eq. (21) for a given λ ≥ β (regularization coefficient), i.e., we prove Thm. I.1 that postulated the existence of such fixed points w ♯ ∗ constructively.

To find a fixed point w ♯ ∗ N , we are searching for an abstract model weight that fulfills Eq. (21):

This implies that w ♯ ∗ N needs to be equivalent to the result of R A on some input zonotope w ♯ N -η ( G N L + L ( G H ) . Recall the definition of R A for a linear input zonotope ℓ ♯ :

Note that R A applies R IH to all error symbols in its input. Thus, R IH ( · ) is a d -dimensional box in the projected space into which the input ℓ ♯ is mapped into by A . Furthermore, as w ♯ N -η ( G N L + L ( G H ) does only contain symbolic terms, the interval hull (the box computed by R IH ) is centered at 0 .

Lemma I.5. Any solution w ♯ N of Eq. (26) is equivalent to the image of a box b ♯ centered at 0 produced by A -1 , i.e.,

Proof. IH does not change the center of a zonotope and A is a linear map. Now observe that all terms in Eq. (14) and (15) are symbolic (contain error symbols). Thus, the RHS of Eq. (26) that is the input to order reduction is a zonotope with center 0 .

A linear zonotope that is a box in d -dimensional space with center 0 is, up to equivalence, uniquely determined by the diameter of the box in each dimension. That is, if we use k i ≥ 0 to denote the diameter of the box in the i th dimension and k ∈ R d to denote the vector of these elements, then we can write any such zonotope b ♯ as:

Combining Eq. (26) with Eq. (27), we know that w ♯ N is a zonotope that is equivalent to the image of A -1 on a box b ♯ parameterized by k . We will only consider solutions of this form: 2

where a 1 , · · · , a d are the column vectors of A -1 . As the naming of error symbols is irrelevant, finding a solution to Eq. (26) now amounts to finding a vector k as the error symbols are now treated as fixed.

Now we can substitute the formula for w ♯ ∗ N from Eq. (28) into the RHS of Eq. (26) as shown below. Recall from app. G.2 that IH merges different error symbols by adding up the absolute values of their coefficients. For the i th dimension, each k j will appear with a certain coefficient. For some terms, the symbolic expression for the i th dimension may contain error symbols that do not have any k l in their coefficients. Thus, without considering for now the precise values for each coefficient we know that the box produced by IH in the RHS is of the form shown below where C ∈ R d × d is a matrix with non-negative entries c i,j ≥ 0 (at position ( i, j ) ), and c 0 ∈ R d is a vector with non-negative elements c i, 0 ≥ 0 (at dimension i ). Intuitively, c i,j is the coefficient of k j in the i th dimension and c i, 0 is the sum of the coefficients of error symbols that do not have any k j in their coefficients (e.g., the fresh symbols introduced by linearization). Note since IH does calculate coefficients of the error symbols in the output as a sum of absolute values, we know that all entries of C and c 0 are positive. Furthermore, recall that we assumed that k j ≥ 0 .

Substituting the LHS and RHS of Eq. (21) with Eq. (28) and (29), we get:

̸

Intuitively, the equivalence of boxes from Eq. (30) is equivalent to the equality of their diameters which yields Eq. (31). Therefore, any k ∈ R d satisfying Eq. (31) gives us a fixed point w ♯ ∗ N . In Eq. (32) we make explicit our assumption that k i ≥ 0 and finally in Eq. (33) we write Eq. (32) as a system of linear equations.

Existence of a Closed Form Solution for the Fixed Point Equations for w ♯ ∗ N What remains to be shown to prove Thm. I.1 is that Eq. (33) is guaranteed to have a solution. For that we will investigate the structure of C and c 0 . Recall that the matrix C and c 0 are obtained from IH order reduction on input (see Eq. (29))

w ♯ ∗ N and G N L share the same set of symbols, while they do not share any error symbols with L ( G H ) , as linearization always generates new error symbols. This implies that the terms of A ( w ♯ N -η G N L ) and -η A L ( G H ) will not cancel out. In other words, they contribute separately to the diameter k of the merged box produced by IH. Therefore, we will look into their contributions separately.

For A ( w ♯ N -η G N L ) , we substitute the definitions of G N L and w ♯ ∗ N to get:

̸

This contributes ( | 1 -2 ηλ -2 η n q i,i | k i + 2 η n ∑ d j =1 j = i | q i,j | k j ) to the diameter of the merged box along dimension i , where q i,j is the element in AX R T X R A -1 at position ( i, j ) . By choosing a small learning rate η to let 1 -2 ηλ -2 η n q i,i ≥ 0 the contribution is equal to ( (1 -2 ηλ -2 η n q i,i ) k i + 2 η n ∑ d j =1 j = i | q i,j | k j ) . From Eq. (34) it is obvious that this component has a shared term w ♯ ∗ N containing k and as w ♯ ∗ N is centered at 0 , we know that this component has only symbolic terms. This in turn implies that this component only contributes to the coefficient matrix C and does not contribute to the constant part c 0 of the system of linear equations.

̸

Now consider the other component

Note that the first part of the equation is the only part that contains w ♯ N which in turn contains the variables k for which we want to solve. As every element of this part of the equation contains some k i it only contributes to C . In contrast the second part of the equation does not contain any variable k i and, thus, is the only part of both components that contribute to c 0 . Let us use 2 η n C ′ k to denote the matrix of coefficients that is the sum of absolute values of coefficients of

2 n ( X R T X ♯ S + X ♯ S T X R + X ♯ S T X ♯ S ) w ♯ N that will be projected by A stemming from the first part of the equation in the expanded version of L ( G H ) . We use c ′ i,j ≥ 0 to denote the element of C ′ at ( i, j ) . As mentioned before this is the only part of this component that contains terms containing unknowns k i . We use c ′ 0 to denote the contribution from the second part of Eq. (37) towards the output of linearization L . Then based on the fact that the second part of this component is the only part of both components that contributes to c 0 , we know that the whole contribution of the component from Eq. (36) towards C + c 0 is:

In summary, the system of linear equations from Eq. (33) can be written as:

̸

̸

In the expanded system of linear equations, the diagonal elements of the coefficient matrix are ( λn + q i,i -c ′ i,i ) , while the off-diagonal elements are -( | q i,j | + c ′ i,j ) .

Lemma I.6. If for all i ∈ [1 , d ] ,

̸

then the system of linear equations from Eq. (39) has a solution.

̸

an M-matrix 3 . For any M-matrix M , the inverse M exists and is has all non-negative entries. Furthermore, for any system of linear equations represented as matrix equation Mx = b where x are the variables of the equation system, M -1 b is a solution. As our matrix I -C is an M-matrix, we know that its inverse exists and has only positive entries and we can compute a solution k as:

Proof. When this condition holds, the coefficient matrix of this system of linear equations is diagonally dominant. Also considering all diagonal elements are non-negative ( ( λn + q i,i -c ′ i,i ) ≥ ∑ d j =1 j = i ( | q i,j | + c ′ i,j ) ≥ 0 ) and all off-diagonal elements are non-positive, the coefficient matrix is -1

Given that all elements c i, 0 are positive and that ( I -C ) -1 is positive, the solution k for Eq. (39) is also positive.

The correctness of Thm. I.1 follows immediately based on the sufficient condition for the existence of solution stated in Lem. I.6, .

̸

Corollary I.7 (Proof of Thm. I.1) . Given an input abstract training dataset D ♯ , let β = 1 n max i ( ∑ d j =1 j = i ( | q i,j | + c ′ i,j ) + c ′ i,i -q i,i ) For any regularization coefficient λ ≥ β , we can compute an abstract fixed point w ♯ ∗ for the model weights according to Def. 3.2.

Proof. To find an abstract fixed point of the model weights

̸

that fulfills the sufficient condition for an abstract fixed point from Prop. I.2, we first use the closed form solutions from Lem. I.4 to compute w ∗ R and then using w ∗ R compute w ♯ ∗ D . These are guaranteed to fulfill the first two conditions of Prop. I.2. As the regularization coefficient λ ≥ β = 1 n max i ( ∑ d j =1 j = i ( | q i,j | + c ′ i,j ) + c ′ i,i -q i,i ) , the condition from Lem. I.6 holds. Thus, the linear system of equations for a w ♯ ∗ N that fulfills the last condition of Prop. I.2 has a solution that we can compute using any techniques for solving a system of linear equation. It follows that w ♯ ∗ = w ∗ R + w ♯ ∗ D + w ♯ ∗ N fulfills all conditions of Prop. I.2 and is an abstract fixed point according to Def. 3.2.

J Weakening the Requirements on Regularization

The existence of a fixed point w ♯ ∗ as postulated in Thm. I.1 only holds for sufficiently large regularization λ ≥ β where β depends on D ♯ . The majority of real world examples without multicolinearity in the real part of the training data that we have investigated have β = 0 . However, we still provide a technique to reduce β if the need should arise at the cost of increasing the runtime of our approach. Specifically, we will make use of splitting [2] which divides an input zonotope into multiple zonotopes of smaller expand. We apply splitting to D ♯ , then compute the fixed point for each zonotope in the result of splitting, and finally combine these fixed point using a join operation for zonotopes that over-approximates the union of the concretizations of two zonotopes (allows us to merge multiple zonotope such that the merged zonotope has a concretization that over-approximates the concretizations of all of the inputs). We will demonstrate that this approach enables us to reduce β to any value ≥ 0 . Intuitively, this works because we can reduce the extend of the input zonotope for the fixed point computation to the extend necessary to ensure β = 0 . Albeit in worst-case this can require a splitting operator that generates a number of zonotopes that is exponential in the dimension.

J.1 Splitting and Join for zonotopes

Intuitively, a splitting operator divides an input zonotope into two or more zonotopes such that the union of their concretization is the same as the concretization of the input zonotope.

Definition J.1 (Split Operator) . An operator S that maps a zonotope ℓ ♯ to a set of zonotopes { ℓ ♯ 1 , · · · , ℓ ♯ m } is for any input zonotope ℓ ♯ we have:

As an example of a split operator consider the binary split (2-split) at dimension i denoted by S 2 ,i that splits the input zonotope into two parts by scaling the i th generator g i by 1 2 and shifting the center of the zonotope by 1 2 g i ( -1 2 g i ):

̸

̸

As γ ( ℓ ♯ 1 ) ∪ γ ( ℓ ♯ 2 ) = γ ( ℓ ♯ ) , S 2 ,i ( · ) is a split operator according to Def. J.1. We can generalize 2-split to an ( m,i ) -split S m,i that divides the zonotope evenly into m parts along the i th dimension. We will call the composition of ( m,i ) -splits across all dimensions as a µ -split. Given µ = 1 m for m ∈ N as input, the effect of the µ -split S µ is the generators of each zonotope in the result of splitting are the generators of the input zonotope scaled by µ . Thus, µ -splitting allows us to downscale the generators of a zonotope. For a d -dimensional zonotope and µ = 1 m , S µ returns d m zonotopes.

Definition J.2 ( µ -split) . For d -dimensional zonotopes and µ = 1 m for m ∈ N , the µ -split S µ is defined as:

where the ( m,i ) -split S m,i is defined as shown below and the application of a split operator to a set of zonotopes is defined as applying the split operator to every element in the set.

̸

For example, S 2 (1 + ϵ 1 ) = { 0 . 5 + 0 . 5 ϵ 1 , 1 . 5 + 0 . 5 ϵ 1 } . It is easy to see that S µ is indeed a split according to Def. J.1.

Proposition J.3 ( S µ is a Split Operator) . S µ is a split operator for any µ = 1 m were m ∈ N .

Next we introduce join operators for zonotopes and then demonstrate that for any abstract transformer F ♯ for a function F , we can construct another abstract transformer for F by (i) splitting the input zonotope using some split operator S , (ii) applying F ♯ separately on each zonotope returned by the split, and (iii) merge the results using a join that we introduce next. 4

Definition J.4 (Join) . An operator ⊔ is a join for zonotopes if it over-approximates union of concretization. That is, for any set S ♯ of zonotopes:

Several join operators have been proposed for zonotopes, e.g., [24]. Next we establish that new abstract transformers can be constructed by wrapping existing transformers with split and join. This ensures that our approach of combining our abstract fixed point calculation with splitting yields valid fixed points.

Lemma J.5 (Abstract Transformers Are Sound on Splits) . Consider a zonotope z ♯ and let F ♯ be an abstract transformer for a function F . Furthermore, let ⊔ be a join operator, S be a split operator, and let S ( ℓ ♯ ) = { ℓ ♯ 1 , . . . , ℓ ♯ m } . Then, F S , ⊔ ♯ = ⊔ ◦ F ♯ ◦ S is an abstract transformer for F .

Proof. We have to show that

We have

for

J.2 Weakening Regularization Requirements

Equipped with the µ -split and a join operator, we are ready to analyze how the introduction of a µ -split can be used to achieve an arbitrarily small β . Given the input data D ♯ , the learning rate η ≥ β , the regularization coefficient λ , the transformation matrix A , we use Γ( D ♯ , η, λ, A ) to denote the function that computes the fixed point w ♯ ∗ S for these inputs using the process outlined in Corollary I.7. From Lem. J.5 follows that Γ S µ , ⊔ is an abstract transformer for gradient descent and, thus, according to Prop. 3.3 over-approximates w ⊙∗

Next, we show that given an abstract training dataset D ♯ and a desired regularization coefficient λ target , we can find a value of µ such that the abstract fixed point construction with µ -splitting computes a fixed point. Recall that Γ can construct a fixed point if λ ≥ β where β is a constant that depends on D ♯ . We will show that by choosing µ carefully, we can achieve β ≤ λ target for each zonotope in the result of the split and, thus, Γ will return a valid fixed point for each zonotope in the split result.

Lemma J.6. For any abstract training dataset D ♯ and desired regularization coefficient λ , there exists m ∈ N such that for µ = 1 m , Γ S µ , ⊔ returns a fixed point w ♯ ∗ .

Proof. Consider an abstract training dataset D ♯ = ( X ♯ , y ♯ ) with X ♯ = X R + X ♯ S . WLOGconsider a single abstract training dataset D ♯ i in the result of µ -split on D ♯ :

We will show that using the regularization coefficient λ , we can find µ such that the precondition λ ≥ β for Γ to compute a fixed point holds. Then applying Lem. J.5 we get the desired result.

First observe that since D ♯ i is in the result of µ -splitting, we know that it has the following shape where the symbolic component X ♯ S is scaled by µ and the real component X Ri of D ♯ i typically differs from X ♯ as µ -splitting changes the center of the zonotope.

Recall that 2 η n C ′ k denotes the matrix of coefficients that is the sum of absolute values of coefficients of 2 n A ( X R T X ♯ S + X ♯ S T X R + X ♯ S T X ♯ S ) w ♯ N . Plugging in the definition of X ♯ i from above, we get a matrix 2 η n C ′ k that is the sum of absolute values of coefficients of

Recall that c ′ i,j denotes entry of C ′ at row i and column j . Here, X ♯ S T X ♯ S has higher order than ( X R T X ♯ S + X ♯ S T X R ) , thus their terms will not cancel out. Therefore, we can decompose the contribution to c ′ i,j into the above two parts, denoted as

Given scalar µ , let i.e., c ′ µ denotes the largest entry c ′ i,j in D ♯ i .

Let i and j such that c ′ µ = c ′ i,j , we have:

where c ′ max is a constant that is equal to the maximum c ′ i,j before scaling X ♯ S through µ -splitting.

Next, we prove the lemma for a specific transformation A = V T during order reduction derived through singular value decomposition (SVD). Using SVD, we can decompose the covariance matrix X R T i X Ri of the real number part of features X Ri as shown below

For A = V T , and using the fact that V is a rotation in SVD which implies V -1 = V T , we get:

Thus, the matrix AX R T i X Ri A -1 = Σ is a diagonal matrix, meaning that all off-diagonal elements of it ( | q i,j | for all ( i = j ) in Corollary I.7) are zero. In addition, the diagonal element Σ[ i, i ] ( q i,i in Corollary I.7) is the i th eigenvalue of the covariance matrix X R T i X Ri , which must be positive when assuming no multicolinearity.

̸

To sum up, β from Corollary I.7 can be re-written as:

̸

̸

̸

where 1 n min i ( q i,i ) is a positive constant. Therefore, when setting µ = min i ( q i,i ) dc ′ max > 0 , we have β ≤ 1 n dµc ′ max -1 n min i ( q i,i ) = 0 .

K An Approximate Abstract Transformer for Ridge Regression

Alg. 2 is the detailed version of Alg. 1 we presented in the main paper. Like Alg. 1, Alg. 2 takes as input an abstract dataset D ♯ that over-approximates D ⊙ , a learning rate η , a regularization coefficient λ , and a transformation matrix A used for order reduction (e.g., the SVD-based transformation discussed in App. J.2). In Line 1, we first use function constructFPEquations to compute fixed points for w ∗ R and w ♯ D using the closed form solution from Lem. I.4 and construct the equation system Ξ for w ♯ ∗ N (Eq. (39)). Furthermore, this function also computes the threshold β on the regularization coefficient for this system of equations to have a solution. If β is smaller or equal to the desired regularization coefficient λ (Line 10), then we solve the linear equations Ξ (Line 11) and then returns the fixpoint w ♯ ∗ . Otherwise, we determine a sufficiently small splitting factor µ that ensures that β ≤ λ for each w ♯ i in the result of the split (Line 3), compute the fixed point for each such w ♯ i (Lines 6 and 8), and merge these fixed points using join to compute the final result (Line 9).

L Infeasibility of Symbolically Evaluating the Closed-form Solution for Ridge Regression

In this section, we demonstrate why evaluating the closed-form solution for linear regression with MSE is not feasible. It produces symbolic expressions with many variables and large symbolic terms (their representation size) that include the fractions with denominators and enumerators that are polynomial expressions. Apart from the size of these expressions, computing viable prediction intervals from such a symbolic representation of model weights is computationally hard. We use MSE loss for simplicity, but the same arguments also apply to ridge regression.

We illustrate the issues using a randomly-generated toy dataset with 10 samples and 3 uncertain data cells in the third column (corresponding to 3 error symbols ϵ 0 , ϵ 1 , ϵ 2 ) 5 :

Then, symbolically evaluating the closed-form solution for linear regression with MSE loss, we obtain the symbolic expression representing all possible model weights:

Next, we consider one test sample x t = [1 , -1 , 1] and apply this closed-form model weight expression to infer its prediction range V ♯ ( x t ) (cf. Corollary F.2). With merely 3 uncertain data points in the training data, the prediction of this test data point

already consists of fractions of complex polynomial expressions. Finding the viable prediction range for this data point, i.e. the minimum and maximum possible prediction for the point across all models in the concretization of the symbolic expression, using this expression is infeasible, as it is harder than finding an extrema for multivariate polynomials under linear constraints 6 , which is known to be NP-hard. In practice, expressions will be significantly larger as they will involve more error symbols. Assume each column has p uncertain cells, and q is the number of uncertain labels, the number of distinct monomials is O ( p d d 2 d q ) , where d is the number of dimensions, and the d in exponents mainly comes from the matrix inversion when computing the determinant.

In summary, the symbolic expressions obtained from symbolically evaluating the closed-form solution for linear regression are not suitable for representing the space of possible model weights.

M Additional Experiments

M.1 Robustness Verification Additional Results

For a better comparison with MEYER [47], we use heatmaps to visualize the robustness ratios averaged over 5 repeated experiments with different random seeds. In App. M.1, ZORRO exhibits much less yellow regions (representing higher uncertainty) compared to MEYER. Especially when the uncertainty is high (bottom-right part), there are many cases that ZORRO returns high robustness ratio but MEYER does not. Recall that both approach gives sound over-approaximation of prediction robustness, thus the robustness ratios returned by both are lower bounds of the ground truth prediction robustness ratio. Therefore, the aforementioned cases correspond to highly robust ground truth (no less than the result of ZORRO), where ZORRO gives tight over-approximations but MEYER does not.

M.2 Micro Benchmark

We use range of loss as the metric to measure the quality (tightness) of uncertain training result. range of loss for samples are calculated by max loss - min loss across all sampled results. Loss range for symbolic representations are calculated my measuring the range of the interval concretization of the loss function result. Notice that samplings return a subset of all possible result which has no soundness guarantees, as a result, produces an under-approximation of the loss range. The symbolic fixed point approach produces an over-approximated loss range. We take 1,000 (1k) and 10,000 (10k) samples from all possible worlds as under approximations to the concrete range when number of possible worlds are to large to compute the ground truth range.

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Percentage of Uncertain Data

0.05

0.1

0.15

0.2

0.25

100

100

100

97.2

92.5

80.8

67.5

60.2

49.8

40.5

100

98.2

89.0

64.5

45.0

26.5

17.2

11.8

7.2

4.8

100

88.5

57.5

32.0

18.0

6.8

3.5

1.2

0

0

97.5

68.2

35.2

15.5

6.0

1.0

0

0

0

0

94.8

49.8

20.8

5.8

1.5

0

0

0

0

0

Meyer et al.

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Percentage of Uncertain Data

0.05

0.1

0.15

0.2

0.25

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

98.8

100

100

100

100

100

100

99.2

97.7

93.5

87.0

100

100

100

100

100

98.2

93.5

87.2

79.0

66.5

ZORRO

0

20

40

60

80

100

Robustness Ratio (%)

(a) Mpg

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Percentage of Uncertain Data

0.02

0.04

0.06

0.08

100

100

100

100

100

100

99.9

99.1

96.3

87.5

100

100

100

99.1

86.0

62.0

36.3

21.7

14.3

9.7

100

100

95.9

64.7

27.5

14.3

8.0

4.0

1.9

0.4

100

99.0

61.9

23.1

9.3

4.0

0.9

0.4

0

0

Meyer et al.

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Percentage of Uncertain Data

0.02

0.04

0.06

0.08

100

100

100

100

100

100

100

100

100

100

100

100

100

100

100

99.9

99.6

98.4

94.9

88.3

100

100

100

99.9

99.3

93.8

83.7

66.7

46.0

28.3

100

100

99.9

98.0

86.6

65.3

39.6

21.1

10.7

2.7

ZORRO

0

20

40

60

80

100

Robustness Ratio (%)

(b) Insurance

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.01 0.02 0.03 0.04 0.06 0.07 0.08 0.09

0.1

0.12

1k

10k

Zorro

loss

uncertain ratio

(a) varying amount of uncertainty

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.06

0.12

0.19

0.25

0.31

0.38

0.44

1k

10k

Zorro

loss

uncertainty radius

(b) varying uncertainty radius

0.08

0.09

0.10

0.11

0.12

0.13

0.14

0.15

0.16

5

6

7

8

9

10

11

12

13

1k

10k

Zorro

loss

dimension

(c) varying dimension

Varying ratio of tuples contains uncertain data Figure 6a shows the loss range results while updating the ratio of uncertain rows from 0.01 to 0.12. ZORRO has a tight range considering all samples are under approximations of ground truth range especially in . As uncertainty increases, the over-approximation gap widens due to the increased coefficient of higher order terms in the gradient, which are linearized, leading to higher linearization errors.

Increasing uncertainty radius Figure 6b shows the result by increasing the radius of uncertain data (imputation results) by multiplier of 0.06 to 0.44 where 0.06 means the range for each uncertain data is increased by 6%. Similar to amount of uncertainty,

Varying dimensions Figure 6c Shows the result by change number of dimensions to the training data. Result indicates tightness of ZORRO 's over-approximation is not affected by the dimension of the data.

Figure 7 added sampling result to Figure 3 shows additionally that sampling result is an under approximation of the ground truth result.

M.3 Additional Experiments on Varying Regularization Coefficients

Fig. 8 shows the effect of varying regularization coefficients on the worst-case loss and prediction robustness ratio. To avoid much overlapping in the plots, we used one standard deviation as the error bar. Similar to the conclusions in Sec. 5.2, λ = 0 is often not the optimal regularization coefficient in terms of accuracy or robustness. In fact, a small, positive λ could result in higher accuracy and better robustness, and this optimal λ varies across different settings. In general, higher data uncertainty requires higher λ , which coincides with the intuition of using regularization in traditional settings of linear regressions, which aims to mitigate the effect of data noises or errors.

0.00

0.04

0.08

0.12

0.16

0.20

0.24

0.06

0.12

0.19

0.25

0.31

0.38

0.44

1k

GT

Zorro

loss

range multiplier

(a) varying range size

0.07

0.08

0.09

0.10

0.11

0.12

0.13

5

6

7

8

9

10

11

12

13

1k

GT

Zorro

loss

dimension

(b) varying data dimension

0.00

0.01

0.02

0.03

2.60

2.70

Worst-case Loss

Uncertain Data: 8%

0.00

0.01

0.02

0.03

2.70

2.80

2.90

Uncertain Data: 10%

0.00

0.01

0.02

0.03

3.00

3.10

3.20

Uncertain Data: 12%

0.00

0.01

0.02

0.03

3.20

3.40

Uncertain Data: 14%

0.85

0.90

0.95

0.6

0.7

0.8

0.2

0.3

0.4

0.05

0.10

0.15

Robustness Ratio

Figure 9: Robustness ratio (red y-axis) and worst-case test loss (green y-axis) vs. regularization coefficient λ (x-axis), with varying percentages of uncertain labels.

M.4 Comparing with Bayesian Regression

We ran empirical evaluations to demonstrate how data quality issues pose challenges for Bayesian linear regression (implemented with torchbnn [40]), making them inapplicable to our setting. Using the setting from the third plot in Fig 1c, where the uncertain data percentage is set to 10%, we tested Bayesian linear regression on different possible worlds using two methods: impute-and-predict and sampling from possible worlds. The results show that the prediction intervals generated by Bayesian methods do not cover the ground truth prediction, i.e., the prediction by the model trained on the ground truth training data. In contrast, our approach guarantees 100% coverage across all cases.

M.5 Varying Robustness Threshold

Different practical applications may differ in how much uncertainty they are willing to tolerate, thus leading to varying choices of robustness thresholds. To account for this in the experiments, we selected both a low threshold (0.5% for the insurance data) and a high threshold (5% for the MPG data). As presented in Fig 10, we also explored other thresholds, which did not impact the trends significantly: Zorro can consistently certify a larger fraction of the test data points than the baseline due to its use of the more expressive zonotope domain.

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Ratio

Robustness Threshold: 0.4 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Threshold: 0.8 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Threshold: 1.6 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Threshold: 2.4 %

Uncertain Data Percentage (%)

ZORRO

Meyer et al.

M.6 Varying Uncertain Feature

The variability of trained models will depend on the correlation that features with uncertainty have with the label, which then also affects our over-approximation of this set. As a rule of thumb, model variability will increase with the correlation between the uncertain feature and the labels, leading to less robust predictions. We conducted an additional experiment using the MPG dataset, focusing on the feature 'acceleration,' which has relatively low correlation with the label. Unlike the feature 'weight' (Fig. 1c), where the robustness drops when the uncertainty radius is 10%, the robustness for 'acceleration' starts to drop only when the uncertainty radius is increased to 16%. The result shown in Fig 11 indicates that features with lower predictive power have less impact on the robustness of a model compared to features that are highly correlated with the label.

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Ratio

Uncertainty Radius: 6 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 8 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 10 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 12 %

Uncertain Data Percentage (%)

ZORRO

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Robustness Ratio

Uncertainty Radius: 14 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 16 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 18 %

0.0

2.5

5.0

7.5

10.0

0.0

0.5

1.0

Uncertainty Radius: 20 %

Uncertain Data Percentage (%)

ZORRO

M.7 Runtime of ZORRO

We evaluated the runtimes for computing the closed-form solution on the MPG dataset, varying the numbers of uncertain data points, in Fig 12. With the same number of uncertainty data points, uncertain features lead to more complex computations of covariance matrices, and result in higher runtimes compared to label uncertainty, where the covariance matrix remains real-valued.

0.02

0.04

0.06

0.08

0.10

Uncertainty Percentage

2

4

6

Runtime (s)

(a) Uncertain feature

0.02

0.04

0.06

0.08

0.10

Uncertainty Percentage

1

2

3

4

Runtime (s)

(b) Uncertain label

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We state our contributions and scope in Sec. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our approach in Sec. 5 and app. I and J.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide intuitions in the main body of the paper (Sec. 3 and 4). We include the complete proofs in the appendix, and refer to them in the main body of the paper (App. F and H to J)

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Descriptions in the main body of the paper (Sec. 3 to 5), and the formulas and technical details in App. I and J are sufficient for reproducing our approach and experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We share the anonymize Github repository containing the data and code in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe the experiment settings in detail in Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeat experiments with different random seeds and show the mean and standard deviation in the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe the sufficient information on the computer resources in Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics and make sure the paper conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts of our technique in Sec. 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper has no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We credit all existing assests used in the paper, and include the license information in the reference.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide documentations for the released anonymized code (Sec. 5).

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new approach for uncertainty quantification of linear models. The proposed approach is interesting, but the empirical comparison to other means of uncertainty quantification is quite limited.\n\nSTRENGTHS:\n* The proposed approach for propagating the uncertainty about the correct linear model seems interesting.\n* The proposed approach empirically constructs tighter confidence sets than a prior work.\n\nWEAKNESSES:\nMy main concern is that this work is lacking a proper empirical comparison to most confidence set / uncertainty quantification methods such as Bayesian linear regression, conformal prediction, and likelihood ratios (e.g., [1]).\nFor example, the manuscript could benefit from a more concrete argument for why (and when) the proposed approach would be advantageous over probabilistic (Bayesian) inference.\nI believe, provided a more thorough empirical validation, this work could add value to a conference.\n\nIn my opinion, the paragraph starting from line 82 dismisses such approaches too easily. From what I can see, reference 26 does not provide conclusive evidence that noise models cannot estimate effects of \"data quality issues\" or that estimating \"uncertainty about the model parameters\" is inherently insufficient (which is in effect what the presented approach does by returning a set of candidate models).\n\n[1]: Emmenegger, Nicolas, Mojmir Mutny, and Andreas Krause. \"Likelihood ratio confidence sets for sequential decision making.\" Advances in Neural Information Processing Systems 36 (2024).\n\nTypos: lines 285, 331\n\nQUESTIONS:\n* The paragraph starting from line 74 appears to argue that one cares more about possible models (training-time) than possible predictions (test-time). Why is that? This is not apparent to me in most practical applications (including the experimental settings of this work).\n* The paragraph starting from line 24 argues that it is impractical to train a separate model for each of the potentially infinite scenarios a dataset might represent. Quite commonly this is addressed by biasing towards structural simplicity, e.g., selecting the model(s) with the \"shortest description length\" (see Occam's razor). I am missing a reference to this very common approach. Why is this approach not sufficient?\n* In Section 2.1 I am wondering where the set of datasets come from. What is the benefit of incorporating, e.g., missing values as suggested in App. D as opposed to Bayesian approaches? I believe it would help to give one practical motivating example in the main text.\n* In Appendix D I did not find the definition of $\\epsilon$. What is it? And why isn't the maximum entropy distribution used in this case of missing data? Also, I am wondering what the relevant concretizations would look like. Perhaps this section can be extended.\n* Why is not the size of the prediction set (which is common, e.g., in conformal prediction) used in place of the \"robustness ratio\"? Also, perhaps something like \"efficiency ratio\" captures the meaning of this ratio more accurately.\n* How are the robustness thresholds chosen?\n* Why are the abovementioned means of uncertainty quantification / confidence estimation not applicable in the experimental settings of the paper?\n* For Figure 3: How exactly were the errors injected? How was the multidimensional data generated?\n\nLIMITATIONS:\nAn extension of the proposed approach beyond linear models seems difficult.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method called ZORRO that allows for the over-approximation of the set of ridge regression models learned over a set of possible datasets. The authors use abstract interpretation, propose abstract gradient descent algorithms, and develop efficient techniques to learn the set of models. The empirical results illustrate that ZORRO is effective in robustness verification and other analyses related to linear regression.\n\nSTRENGTHS:\n1. The paper proposes a framework that can potentially be used for various methods of defining uncertainty in datasets.\n2. The method seems to be efficient in computing prediction ranges for the overapproximation of set of models learned from uncertain data and discusses how this set can be useful for causal inference applications.\n3. ZORRO's results remain consistent regardless of the number of dimensions for ridge regression. The variance of the robustness ratio and worst case loss decreases with larger regularization coefficients.\n4. The authors provide an extensive overview of related literature from related fields. Literature on the Rashomon Effect, which generalizes predictive multiplicity, might also be of additional interest.\n5. Overall, the paper is well-written, and the problem of learning from uncertain datasets is interesting.\n\nWEAKNESSES:\n1. As the authors acknowledge in the Limitation section, it is unclear how to scale the proposed approach to other, more complex hypothesis spaces or problems.\n2. Despite ZORRO being a more general approach, sampling seems to provide results closer to the ground truth even with 1k samples (Figure 7).\n\nMinor:\n1. Please specify that the numbers on figures on page 5 represent gradient descent iterations.\n2. Definition 2.1 and the surrounding text around it is very abstract. While Appendix D is helpful, examples become clearer after reading Section 2.2. Providing one sentence with specific examples of uncertain data around Definition 2.1 might be helpful.\n\nQUESTIONS:\n1. Could you please comment on why the robustness threshold was set to different values for the two datasets?\n2. For modeling uncertainty in features, the distribution was built around one feature. Does the choice of the feature, such as based on the correlation value between the feature and the label, affect the results\n3. How much time does it take to compute the set of possible linear models for the discussed datasets?\n\nLIMITATIONS:\nThe authors discuss the limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to represent the uncertainty from dataset variations using a type of convex polytope that can compactly represent high-dimensional spaces, as dataset multiplicity from imperfections like missing data may scale exponentially. The paper develops techniques to ensure the tractability of gradient descent and obtain a fixed solution for linear models in this regime. Over-approximations of possible model parameters are provided by performing suitable order reductions and linearizations to ensure that the gradient descent is tractable. Meyer et al. (2023) introduce the dataset multiplicity problem but only account for uncertainty in output labels. This paper also allows uncertainty in input features, thus providing a more general framework to represent and learn under the regime of dataset multiplicity. They also provide a SymPy implementation of their method and perform an experimental evaluation of their techniques.\n\nSTRENGTHS:\nThis work provides a general framework to represent dataset multiplicity in both features and labels, allowing us to represent the potentially exponential number of dataset variations with a suitable convex polytope. The paper carefully performs several order reductions and linearizations, overapproximating the dataset variations and fixed point model parameter variations. Although limited to linear models, this work addresses an important source of uncertainty stemming from dataset variations and non-trivially extends existing work in the area by incorporating uncertainty in input features in the dataset. \n\nThe SymPy implementation of the techniques introduced in the paper helps in evaluating this method extensively.\n\nWEAKNESSES:\nThe techniques introduced in the paper are only applicable to linear models.\n\nQUESTIONS:\nAre there scenarios where the overapproximation of dataset variations and corresponding model parameter variations results in too broad outcome intervals? I would like to understand the limitations of the proposed method apart from the linearity assumption. \n\nIn the experimental evaluation, the dataset uncertainty is varied by changing the uncertain data percentage and uncertainty radius. Did the authors perform any experiments with missing data as the source of uncertainty? (I imagine that you could represent missing data by using a high uncertainty radius). How was the robustness threshold set for the experiments?\n\nLIMITATIONS:\nAt the moment, the last section of the paper (Conclusions/broader impact/limitations) does not include any explicit limitations of their methods. It would be helpful to have a separate discussion on limitations in the main paper instead of appendix. Broader societal impacts may not be relevant for this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}