RS-Reg: Probabilistic and Robust Certified Regression through Randomized Smoothing
Authors: Aref Miri Rekavandi
Olga Ohrimenko
Benjamin I. P. Rubinstein
Venue: Accepted by TMLR ()

Abstract: Randomized smoothing has shown promising certified robustness against adversaries in classification tasks. Despite such success with only zeroth-order access to base models, randomized smoothing has not been extended to a general form of regression. By defining robustness in regression tasks flexibly through probabilities, we demonstrate how to establish upper bounds on input data point perturbation (using the $\ell_2$ norm) for a user-specified probability of observing valid outputs. Furthermore, we showcase the asymptotic property of a basic averaging function in scenarios where the regression model operates without any constraint. We then derive a certified upper bound of the input perturbations when dealing with a family of regression models where the outputs are bounded. Our simulations verify the validity of the theoretical results and reveal the advantages and limitations of simple smoothing functions, i.e., averaging, in regression tasks. The code is publicly available at \url{https://github.com/arekavandi/Certified_Robust_Regression}.

Published in Transactions on Machine Learning Research (04/2025)
RS-Reg: Probabilistic and Robust Certified Regression through Randomized Smoothing
Aref Miri Rekavandi
School of Computing and Information Systems The University of Melbourne
Olga Ohrimenko
aohrrmmenkoowmmimmme www.:meeew.eed........oaa..a
School of Computing and Information Systems The University of Melbourne
Benjamin I.P. Rubinstein
bergjamiin.uubbnssiiiiuuuuniiii edu
School of Computing and Information Systems The University of Melbourne.
Reviewed on OpenReview: /ttps://openrevieee net/ forum? id- 34-4eLllg.1522eeee/eerr
Abstract
Randomized smoothing has shown promising certified robustness against adversaries in classifiaatio tasks. Despite such success with only zeroth-order access to base models, randomized smoothing has not been extended to a general form of regression. By defining robustness in regression lasks flexibly through probabilities, we demonstrate how to establish upper bounds on input data point perturbation (using the l2 norm) for a user-specified probability of obscrving valid outputs. Furthermore, showcase the asymptotic property of a basic averaging function in scenarios where the regression model operates without any constraint. We then derive certified upper bound of the imput perturbations when dealing with a family of regression models where the outputs are bounded. Our simulations verify the validity of the theoretical results and reveal the advantages and limitations of simple smoothing functions, i.e., averaging, regression tasks. The code publicly available at htt.g/ttb...oeeeetttteaaatrrrrer Robustt__ooutttheee "obbust_Reeression
1 Introduction
The ongoing competition between attackers and defenders has long history in cybersecurity. Whenever attackers have gained an advantage, defenders have countered with innovative empirical techniques, fostering cycle of continuous evolution. In recent times, research into defenses has embraced the concept of certified robustness, seeking assurances that attackers will find it challenging to discover adversarial examples in the vicinity of test samples for deceiving Al systems. While these guarantees are constrained to a certain type of threat model (which may not align with the attacker's actual strategy), they offer precise and reliable bounds crucial for cybersecurity systems. Randomized Smoothing (RS) has emerged as a prevalent strategy certifying classification models, demonstrating remarkable scalability to large-scale models. In this approach, majority voting is performed samples drawn, as the estimated parameters tend to their true values, the likelihood of discovering any adversarial examples in the vicinity of the test query to deccive the model goes to zero. RS has been recently extended into other settings such as classifiers with discrete or variable-size inputs (Huang et al.) 2023 ) classifiers with semantic certificates (Fischer et al. 2020), and probabilistic certificates (Pautov et al., 2022b). However, the best of our knowledge, the framework has not yet been fully expanded into conventional regression problems, while many attacks have been demonstrated (Chiang al. 20200] Liu al. (022). To extend RS to regression, the adversarial robustness definition requires some adjustment. In this study, we
samples drawn around given test query. With the greater number of

---

Published in Transactions on Machine Learning Research (04/2025)
introduce probabilistic definition for robustness in regression models and subsequently derive performance guarantees, given mild conditions. This new framework allows defenders to pursue probabilistic certificates and strikes a balance between the precision and the amount of effort an attacker must exert to discover an adversarial example. We summarize the main technical contributions of the paper as follows:
We introduce variant of probabilistic certification for regression problems where the output variable is multivariate and continuous.
Using the new definition, we then derive a probabilistic certified upper bound on the imput perturbation for a base regression model.
We then demonstrate that, asymptotically, the output of RS with averaging follows normal distribution, allowing the probability of obtaining valid results from a smoothed regressor to be determined through an integral over a neighborhood of such a normal density.
Under mild conditions, we propose a lower bound the probability of observing valid outputs using the smoothed regressor. Then we find the upper bound the adversarial perturbation in the imput space which satisfies the user-defined probability of observing valid outputs utilizing the smoothed regressor.
The technical progression of the paper as follows: (1) We first examine the robustness of the base regression model, i.e., fx) and derive an upper bound on the &g norm of the adversarial perturbation which satisfies our definition of robust regression. (ii) Using an average function as the smoothing policy, we study the asymptotic behavior of the output n + co (where is the number of samples drawn from Gaussian used to perturb the input and then averaged to compute the output) and show that is normally distributed. (iii) We then use the results of the base regressor and derive a lower bound probability for regression models with bounded outputs. The results up to this point are valid only in the asymptotic regime, hence, we derive a different bound for the case where the output is bounded and the user considers a discount on the validity range of output variables. This new result is valid for finite-sample scenarios.
2 Related Works
RS has seen little application outside classification. Among the few works that extend the RS framework, Pautov et al. (2022a) proposed smoothed embeddings that extend RS to few-shot learning models that map inputs to normalized embeddings. The work considered the case where the lz norm of the embedding is set to one and showed that the smoothing function is Lipschitz in the f2 norm. This setup is much different than ours because we do not consider any constraint on the output except a bounded range, and outputs can be independent of each other with different scales. Furthermore, our framework is probabilistic. [aaman et al. (2019) addressed the certification for soft classifiers, where the output variables are continuous. However, they were scaled such that they satisfy Σp: to finally classify inputs, not to regress any continuous variable. Their results are not probabilistic in the sense that we present in this work. A recent study by Hammoudeh & Lowd (2023) in the context of poisoning attacks and certified ML has been performed for the first time in regression tasks to find guarantee the number of training instances that can be inserted into or deleted from a training set without violating the output constraints. The most related work to our study is by Chiang et al. (2020) where the object detection problem has been treated as a regression problem. The result relies on expanding the range of classification models which limits certification to regression models with Softmax activation function the last layers.
3 Preliminaries
Notation. In this paper, the base regression model parametrized with is defined such that y = fx(x) : Rd + R where d and & are the input and output dimensions, respectively. A(,, I²I) indicates a multivariate normal density with mean m and covariance I²I, where is the identity matrix. The & norm of vector x is denoted by ||x||2 and it is defined by ||||| (, xExx) where x; is the 4th element of the vector The
2

---

Published in Transactions on Machine Learning Research (04/2025)
neighborhood centered around variable E R' with radius with respect to a given dissimilarity function is denoted by N(z,e). e). This neighboring function defines region such that
N(z,c) = fz' E IR diss x') S ct,
(1)
where diss(:;)) in general, can be any metric or function that the user is interested in and in particular, diss, (or dissy) indicates dissimilarity in the input (or output) space (Miri Rekavandi et al., 2024). P(event) indicates the probability that event occurs and {z) indicates the expected value of the random variable
z. [t] indicates the set (1, .t} and ( is the standard Gaussian CDF. [-] rounds up to the nearest.
integer value. Indicator condition returns 1 only when the condition is True and otherwise is zero.
Randomized Smoothing. RS is among the certification techniques that can be used for large-scale (and arbitrary) models, given only black-box access to model evaluations, and has been used for the first time in seminal works Cohen et al. (2019): Lecuyer et al. (2019) for classification tasks. In particular, Cohen et al. (2019) showed that if the initial classifier fo(x) is robust under Gaussian noise, then the new classifier g(x) arg max.eY [( +)== e~ .N o21) is certifiably robust against an lg norm adversary with radius c = 2 ( (pA) Φ - (PE)) where PA and PB (s.t. PA PB) are the lower bound probability of major
class, and upper bound probability of runner-up class, respectively. Later studies (Yang et al. 2020 Kumar
et al. 2020) showed that for l, norm attacks (p >p>>7 2), these radii decrease as O(1/d2 suggesting that for other norms and dighndimensional data points, these certificate radii tend to zero showing lack of meaningful insights. RS has mainly relied on Gaussian smoothing, however, improvements have been observed using other
smoothing functions such as with Uniform (Lee et al. 2019 Laplacian (Teng et al. 2019 and non- non-Gaussian
(Zhang et al., 2020) smoothing to deal with general types of attacks. We refer readers to Li et al. 20223 for
more details on RS and certified robustness.
Threat Model. We assume attackers with full information about the regression model and its underlying process, and who are limited to perturbations of an input sample x within an fg-bounded norm. The attacker's ultimate goal is to generate sample x' (x, Ex) that generate an output with large deviation from the (x) where this deviation is likely beyond the user's tolerance threshold. Although this paper focuses on the la threat model, one can exlend the results to other norms such as é1, tp, and fxo, either in the input or feature space. An extension of the results presented in this paper for the lp attack in the input space can be found in Miri Rekavandi et al. (2024), where a similar problem is explored. Additionally, one could explore scenarios where perturbations are applied to the parameter space in convolutional kernels, such as in blurring or sharpening operators in the context. of image modality (Brückner & Lomuscio, 2024
4 Probabilistic Robustness Certification for Regression
Randomized smoothing and correspouding certificates for classifiers are tied to aggregated predictions by majority vote (Mohapatra al.) 2020), which does not apply to regression. We therefore start by defining a new notion of probabilistic certificate that is suitable for regression models.
Definition 1: (Probabilistic Robustness Certificate). Given an erample (æ, y), a (possibly) randomized
regression function g(z) : Rdd + R' is said to be certifiably robust with probability 0 < P S 1 in the rundomness of 9, with respect to the given impul and oulpul dissimilaritt functions with radii ezy Ev1 .. , y if Val E Nz (x)
min (gdss, S scn) KE[t]
The definition of certified robustness for the function g(x) differs from the definition of local Lipschilz function as Vx E N(x, <), llg(x) [ < Lxx)]|x - x'Il familiar from analysis. Our definition is probabilistic: it is acceptable to fall outside the predefined neighborhood, and our dissimilarity function is considered to be general not limited to a certain norm. On the other hand, while the neighborhood region in the input space is defined for all the dimensions simultaneously, in this definition output dimensions are analyzed separately using the neighboring function Ny (y) = II Ny Cy.)-
Remark 1:
3

---

Published in Transactions on Machine Learning Research (04/2025)
General
f2 norm
N₂ (x, Ex)
N(X, Ex)
Ny,,
Yisk
V122
X.
X
(x)
fx(x)
Ymn
yi-11t
N, ky,))
N(y. My.z)
The general schematic of probabilistic certified robustness in regression where input deviate from in any direction (bounded with respect to diss, used in (x))) and the desired output should be within range (with respect to dissy) with probability P where outputs are analyzed in 2 groups (left). A particular case where the dissimilarity functions are & norm and / = t/2 (right).
Figure 1:
Figure | (left) illustrates the schematic of certified robustness for regression. As underpinmed by Eq. 22 we are interested in finding the tightest 6z where all samples within the region defined around x, are allowed to push average only 1 1-2 P fraction of generated samples beyond the accepted region of the output. In general cases, outputs can be grouped in categories to perform the analysis, however, for simplicity within this paper we consider 1 = I.
4.1 Robustness of Base Regression Models
In this section, analyze the probabilistic robustness of an arbitrary model fx(),, that has not undergone randomized smoothing. In the next scction, we will introduce randomized smoothing for regression, and analyze its impact on robustness. See Appendix for proof of the following result.
Thcorem (Certification of General Models Against ly-Bounded Attack). Let fo(x): Rd (possibly) randomized base regressor and NN(0, Suppose for some ezample (æ, z),
R be
(f + y.) S 6 PA, E
(3)
where pa, is the lower bound on the probability of accepling prediction in the ith oulput variable. Then referring to Definition 1, fo(x e) is probabilistic certifiably robust at ezample (x, y), for a & 22-norm dissimilarity in the input, chosen probability P < min4e]4) PA,, output radii Eyr..... Eyg and imput radius
=
min o ( ( x-1(
(4)
s
Several observations can be made from Theorem 1.
General output dissimilarities. This result is general and it is valid for any output vicinity (either (.s or 6y) that the user is interested in since they indirectly affect the results through changes PA not in the closed-form expression for the certified radius.
Interplay between PA, 's and Theorem 1 is only valid when the user-chosen probability P PSPA, Vi [t] otherwise, it returns Gr I 0. This observation makes sense because if PA, < P. it means that without any adversarial perturbation, the predicted result is not already in favor of the user. Therefore, additional perturbation of the input does not lead to a better performance. One way to satisfy this condition to use smaller a (sacrificing the tightness of the certification bound).
Assumption free on base regressor. Similar the results for classification tasks (Cohen et al. 2019 Theorem 1 does not make any assumption about fo).) and only requires black-box access to compute PA
4

---

Published in Transactions on Machine Learning Research (04/2025)
Interplay between ca and other parameters. The certificate radius cz is large when (i) the noise level of the isotropic Gaussian smoothing (a) is large, (1i) Pa is large (ideal when PA, - meaning that the evaluated x is significantly stable), (iii) user is happy with smaller P values which gives a better margin to change the input, or finally (iv) having almost the same sensitivity in all output variables to push min.ett Crs towards larger values.
A Abstention. Theorem 1 has the flexibility to abstain from certifying some of the output variables (by excluding them from the [t] or using larger cys) and offer wider certified radius of the input.
Estimating PA is in practice. The result in Theorem 1 relies on the exact PA. while in reality, this value is not exact unless + oo (large sample regime). Here, we suggest using the confidence interval provided by Clopper & Pearson 1934). Our estimator of PA follows a Binomial distribution and it is known that for such a parameter the lower bound PA containing the true parameter p with confidence level - 2. satisfies the equality Σk=x pA) (1 PA)"=* 2 This estimation is exact and X is the number of successes observed in the sample set. containing samples. See more in Appendix
5 Randomized Smoothing for Regression
RS for classification tasks is defined as g(x) = arg m P( + = c), e o0 where includes all the class labels. As this smoothing technique integrates mass of votes made for each class, it is not feasible to be used for regression since regression problems deal with continuous output variables. Therefore, we use the averaging function i.c., g(x) = (ff (x + +)), e~ o
5.1 Certifying Randomized Smoothing: Asymptotic Case
We have explored the robustness of base fo(.) through Theorem 1. Tt is natural to ask: Can smoothing by averaging improve the robustness of the base regression model? If yes, how much improvement can be obtained and what is the behavior of averaging for a large sample regime (n - oo)? In contrast to the discrete cases, where majority voting can tolerate voles against the target. class they are not the majority, in the case of smoothing using averaging. intuition suggest that even a single outlier outcome can drastically shift the mean. This behavior known the zero breakdown point of the sample average in robust statistics (Rekavandi et al.) 2021 In response to these challenges, we first find the approximate probability of returning acceptable outcome using the average function in the case that - co. See Appendix C for a proof of the following result.
Theorem 2:
(Asymptotic Behaviour g(x) Against a Fixed Attack). Let fo(x) Rd + R' a (possibly) randomized base regressor and suppose for a given x and 8, outputs generated by fo + with e e~ ~N(0 o² I) are independent and identically distributed with unknown mean m E RR and unknown bounded covariance E E Bext. TJ the accepted region (sel) for each outpul larget variable is conver then for the user-defined, Ey,
00, (x
N Ey is given by
Φ (v 3 (ub -x (xxx))
- - E x (V (x (c00__((0)) (xx))))
k=1
DER2
where Ø(-) is the cumulative distribution function of a standard multivariate normal distribution, cp uses the lower bounds by for all E DD and the upper bounds us for all & D such that Rr denotes the class of all subsets of [4] with exaclly & elements, and (x) is given by
g
-
n > fx +
(6)
In the above, u, and 4 are upper and lower bounds on the accepted region in output, and E is a consistent coveriance estimator.
5

---

Published in Transactions on Machine Learning Research (04/2025)
Example 1: For clarity and better interoperability of the results in Theorem 2, let's consider the case
where / = 2, representing regression model with two output variables. Then the quantity in 5 reduces to o
(up B., (x 8))) - 2 ( "b-]" - -,, (x + (x)) - 2 PeVv ([ub,,42] - g, (x 8))) + Φ (VV 3 (l. -0--.. (x++ As can be observed, the first summation in (5) only takes care of signs and the
second summation constructs the terms based on lower/upper upper bounds formed by cD; to accurately compute
the probability without omissions or redundant calculations of any region.
Remark 2:
Based on results in Theorem 2, if m falls in the accepted region of the output, 00, with high probability gn (x + 8) will stay in the accepted region. This helps to increase the probability of output robustness from (robustness probability for base regression, i.c., = 1) to 5 (sufficiently small &, i.e., 0 < 1).
conclude
Remark 3: Theorem claims an interesting result for the large sample regime. However, both m and Σ are unknown and they are both functions of d that is unknown in practice. Therefore, how the result can be interpreted to determine the bound for the general form of d is ambiguous. In other words, if a user is interested in setting the value of Eq. (5) to certain value, there is no way to find the constraint on ö. Therefore, we need more assumptions to establish a connection between the input and output.
5.2 Certifying Randomized Smoothing: Asymptotic, Bounded Output Case
Remark 3 demonstrates that further assumptions are needed for practical certificates. One assumption that can involve 5 in the output probability is considering range for output variables, i.., component- wise box constraints 1 S (z) S u, V2 where I and u construct sufficiently large region. We argue that this assumption is relatively weak: it is common in learning theory to assume bounded outputs, outputs can be artificially bounded through clipping, and in most applications, users estimate continu- ous variables such as location in a local area, angle, income, etc. which are all bounded naturally. We consider the robustness property of the averaging function for this class of models. See Appendix D for a proof.
Theorem 3: (Certification of g(x) Against l2 Attack for Bounded Outputs). Let fo(x) Rd Re be a (possibly) randomized base regressor and suppose outputs generated by fa are independent and identically distributed with user-defined ey (equivalent to us and 4) to define the accepted region. Suppose for ~ o= v||0||12 I 6x, and an arbitrary value of p, as stated in 4 Vi E [4]
P(diss, + 5 e)i. vi) S ccyc 2 p.
(7)
Considering bounded output cases, i.e., 1< (z) I u. V2 subject to 1 < 4 < S u, and if for those samples
which are accepted by the user, we have [Bfa (x + 8+ e)} - fa( T, 0 < S min(fa 4, us - (x))
for the conver accepted region (set) and as n - oo, the following inequality holds Vi E [t]
Pidiss (9, 8),, v1) S c
([T((1
"y -o, 7)1,
11, [n
yo f,(2); + 1), if fo(m).
Why
4-4,
min (E(A]
[T[[n(11
fala) -- by
Jo(a), -- )1,
" ())
[m
La (a): h:
m (a),
1);
otherwise,
[x(
(8)
[z())
where Ia b) is the regularized incomplete beta function defined as In b) Ble.b f 1a-1(1 - 1)0-2d di and B(a, b) is the complete beta function.
Remark 4: As the accepted region bounds, i.e., us and In get tighter around fx(,,, the lower bound of Theorem 3 tends to O. In other words, the user does not tolcrate any variation in the output, therefore, the
Although this assumption may look strange, asymptotically and when the accepted region is symmetric, this assumption makes more sense because the range is small and the distribution in that range can be approximated by symmetric distribution. Then the best representative of the range the average itself, and for the worst-case scenario, we use fx(x) t...
6

---

Published in Transactions on Machine Learning Research (04/2025)
method does not guarantee anything after taking the average.
Remark 5: Fixing the number of samples and the lower/upper bounds of the output in Theorem 3, the lower bound of ((dissy(g,, y),) ) I cyi is monotonically increasing with p (therefore, the inverse exists, I.C., I (:: a. b) exists), and as p + 1, P (B. (x + y,) S cvj) - 1.
Based on the above results for a bounded output, one is interested in finding the upper bound on ||ö]|2 to ensure the average value valid with probability 1/2 S q S 1, it is suggested to first estimate P for E [t] by
(g: [i(11 wa (a(x))- )1, [n Wh -b,))(
U.- 1); if [x)) [xx)
[=(])
(x-1(())-
P.
(g: (g: [n(1 (x).
(x)) 1(x)) 1b )7, [' fo(x), 1b 1 + 1), otherwise,
in reverse process (as we know I ( (:: a) exists), and then apply Theorem to find cg for gn (x), i.e.,
= min o ④ (pa.) (p))). = selt]
(10)
5.3 Certifying Randomized Smoothing: Non- Asympsopio Case
To provide an analytical result for the finite sample scenario i.e., n 30, we introduce a framework we call Discounted Certificate. Although for the base regression model, we consider the l and us as the accepted region boundaries, for g,, (x) the user is asked to apply some discount factor 8 2 0 to make the accepted range wider. By doing so, we can now consider the worst-case scenario (putting all the accepted samples in the boundary instead of placement at fx(x T) and leverage this additional margin added by the user. In other words, the new discounted upper and lower bounds of the accepted region are u, + B - fa (x)] and L, - B|1 (x)). A side benefit of this approach is that we relax the Theorem assumption that [[fo(x+ + 8 + e)}: (x)x S r. See Appendix for proof.
Proposition 1: (Discounted Certification of g(x) against lz Attack for Bounded Outputs). Let fo Rd RR' be a (possibly) randomized base regressor and suppose outputs generated by fo(a) are independent and identically distribuled with a user-defined Ey (equivalent. to us and 4)) lo define the accepted region. Suppose for e~ N1,, o² 2I,, V||6 S ca, and an orbitrary value of p. as stated in (. Vi [t]:
P(dissy(fo(a 5 e)i, v.) S cei 2p.
(11)
Considering bounded output cases, i.e., 1 i (z) I u. Vz subject to I 1 4 S un S u and assuming the accepted region (set) for each output target variable to be conver, then given discount factor 3 2 0 such that
I 1 4, (z))(z) - 41 S un (z)) mal S u holds, then the following inequalily holds for Vi E [1]
Pt diss, (gn, (x 8)4, v.) S (1 3)
[1([m((1 114, fz(2)c 11, [r( 3 3/un, -fo(s)d
[[[((1 - Blun - So(e)dd u. my
) + 1), u: Ma;
|m fo(u)a)l
<
1b, (a())
min (E(d]
3 l,, (z)2a
4,
(12)
) In
+ 1), otherwise,
where 1,(ab b) is the regularized incomplete beta function defined as (p Bla,b) fot ta-1(11 1) dt and B(a,b) b) is the complete beta function.
As outlined in Proposition 1, the user is required to incorporate discount factor, represented by positive constant 3, into the acceptable output range to establish certificate for the worst-casc scenario. As a consistency check, when 3 + 0 in (1). the lower bound approaches zero. This implies that in the extreme case where all valid output samples are already at the boundary, a single invalid output can push the average beyond the validity threshold. Since & + O, climinates any discount or margin, there is no justification for the result, leading the lower bound on the probability to converge to zero.
A drawback of this approach is that if the user chooses not to apply such discount, certification be
7

---

Published in Transactions on Machine Learning Research (04/2025)
30
30
25
x 696997
20
Y. -0.87787879
20
Z 23. 59576
15
15
2 X2
Figure 2: The two-dimensional function used for the simulation.
provided for the result. While this may initially seem discouraging, we propose an alternative strategy. the user cannot extend the output region, we introduce an artificial (conservative) marginal zone by adjusting ey thereby reducing the acceptable region for estimating PA, for Vi E [t]. Although this adjustment may yield a weaker certificate compared to the one for (x), it ensures that the certification is not entirely lost and that any obtained certificates remain valid.
5.4 Adversarial Training
To obtaín improved tightness from randomized smoothing, is first required to have robust base regressor stated in Theorem 1. Cohen et al. (2019) argued that to increase the robustness of the base classifier retraining of the network is permitted one can use Gaussian noise augmentation and train the network. Although our proposed method is designed for the regression task, it can leverage the same concept. In other words, we can use x + e, e ~ (O0 o²I) to train the base regression model with the same ground truth y. the case of black-box access, other strategies such as denoisers attached the base regression e.g. what. has been proposed in Salman al. (2020): Carlini et al. (2023) can be utilized.
6 Experimental Results
In this section, we perform experiments empirically validate our theoretical results. For synthetic simulations, we present the results for an example function that demonstrates sharp variations in output.
We then apply the proposed methods images. All simulations and experiments were conducted using an Intel(R) Core(TM) 17-9750H CPU rumning at 2.60GHz (with a base clock speed of 2.59GHz) and 16GB of RAM.
camera re-localization task (Rekavandi et al. 2023) based on
Simulation study. We begin the simulation with an example base model f RR + R given by
f(x) 10 + (z2 2)² 15.
(13)
This function was investigated for the interval -1 x1, with Figure illustrating this sharp function for the defined range. As all the results in this paper are point-wisc certificates, we only selected the integer points in these ranges to derive the certificate radii. We set o 0.23, = 6 for the e output norm, U = 35, L= 0, t = 0, n = 10K, to ensure that the user-defined probability P = 80% is always less than PA As n is large, we used the estimated PA as the Pa and skipped the use of the Clopper-Pearson lower bound estimator (see Appendix B). We also selected
(1.5, 2) for the discounted certification algorithm. Figure 3 visualizes the theoretical certificate radii derived from Eq. 44 for f(x) (blue), inequality (8) for g(x) (red),
8

---

Published in Transactions on Machine Learning Research (04/2025)
5
fxx
Bounded-gio
Discounted g(x), Beta=
Discounted g(x), Beta=2
3
2
-1
X1
Figure 3: Theoretical certificates derived for f(x) (blue), gxx) for well-behaved base regression (red), and discounted certificates of gxx (black and green) where the user set P = 80% (right).
Bus Plut Each Cortay itet
Doe Fist bach te gla)
oifinee
3375
3330
3333
3.500
04/3
3310
10-
3425
-22
34300
Dertlyy
dil
4.00
Figure 4: Empirical probability of valid output in comparison with desired probability defined by the user (80%) for f(x) (top left), g(x) (top right), discounted g(x) (bottom).

---

Published in Transactions on Machine Learning Research (04/2025)
[(x), sigma=0.05
180
f(x), sigma=0.025
225
g(x). sigmam0,05
160
g(x), sigma=0.025
200
g(x), sigma=0.05
Dis. g(x), sigma=0.025
certified median error
140
f(x), sigma=0.05
175
g(x), sigma-0.05
120
Dis. g(x). sigma=0.05
certified mean
150
100
125
100
75
40
20
0.00
0.02
0.04
0.06
0.08
0.10
radius
radius
Figure Certified median (left) and mean (right) error DSAC as a function of r.
Attacked Image with sigma=0.05, r<0.1
Attacked Image with sigma=0.05, r<5.0
Attacked Image sigma=0.2,
Figure 6: Examples of adversarial images contaminated with noise. As shown the changes in the certified range and considered noise are not visible to the human vision system, unless they become significant in magnitude.
and inequality green).
12 for g(x) where the output validity range is discounted in two different rates (black and
The certified radius for f(x) is wider for the points that return smoother changes in the output and is small for sharper regions. Randomized smoothing in Theorem 3, offers a slightly better certificate compared to the certification of fx)), however, this true only for those points that satisfy the assumed condition. This improvement small mainly due to a wide range of output variability and sensitivity of averaging function to single largely deviated point. Finally, for discounted certification of g(x), considering better when larger better certificate (black) than any other approaches. This certificate becomes discount is applied (green for 3 = 2). Note that all of these certificates become smaller and smaller when the user-defined desired robustness probability increases (see Appendix F). Figure 4 visualizes the empirical probabilities obtained over 25 points evaluated for f(x) and gx)), respectively for 20 trials. As shown in Figure 4 (top left), the proposed radii for the evaluated points (in f(x) perfectly satisfy the user-defined probability and they are all uniformly above this threshold as expected from the results in Theorem shown, at some points, these empirical gaps are small which empirically shows the tightness of the bound. Figure 4 (top right), shows the results of Theorem 3, and as shown the probabilistie lower bound holds with great margin though the upper bound input radius is better than the one for f(x) (red circles in Figure Figure 4 (bottom) illustrates the lower bound and the empirical probabilities for the discounted certificate (3 = 1.5) scenario, and as shown all the empirical values are above the desired level. Although these certificates are sound, they are conservative as expected, since we consider a worst-case scenario in the lower bound derivation.
1.5, attained
As
Camera re-localization task. Camera re-localization from a single RGB image is an important task in many applications such as robotic positioning systems, SLAMM, autonomous driving, etc. Given image
10

---

Published in Transactions on Machine Learning Research (04/2025)
in the input, the output of such a system is multivariate continuous variable and the task is to regress 6 parameters denoted by P. 3 for position and 3 for orientation. Defense against adversarial perturbation for such a system is vital because any wrong predictions in such systemss may cause irreparable consequences, e.g., in autonomous driving systems attacks may cause severe accidents with fatalitics. DSAC* (Brachmann & Rother 2022) is a popular technique and adopted this paper for robust analysis. Although in the literature, together with median error, the accuracies were reported for some user-defined thresholds, c-g-: 0.05% for outdoor scenes, we report the result in terms of median and mean error. The certified median error at radius r in general form is defined as the median of errors computed for each test image follows:
Ck = diss, + +)) +,>,, K, Sr
where K is an arbitrarily large value and pushes the output error to a larger range if the evaluated radius is beyond the estimated certificate bound. In our simulation, we only consider the pesition-related parameters in p and use the reduced form
ek = =1ggxx + + p2) + Kr V||8112 Sr
with K = 150em.. For learning of PA using Clopper- Pearson (a - 0.5), we used 200 samples and then we used = 10 for each radius to examine models in the Cambridge Great Court scene in the Cambridge Landmarks dataset (Kendall et al. 2015) using the DSAC* pre-trained model. For the Great Court Scene, out of 760 test images, 120 randomly selected images were used (due to the similarity of the images and to reduce the required runtime) to report the certified error rale defined above. We used a threshold of ey = 5m for defining the accepted region in the output, with (U 85 and 15) with 3 2 and 80%. we investigate the range of r E [0, 0.1] for the scene where the image dimension was 480 854 pixels. Figure (left) illustrates the certified median error rate using two different smoothing noise variations a increases, the error rate also increases, but the certified radius increases. Similar to the classification task (where certified accuracy is used), there is a jump in error rate for r 2 Eg and this jump occurs in larger radii when a increases. The simple averaging function is beneficial in lerms of error rate but less beneficial in terms of offered radii compared to the base regression model. This is mainly due to the large upper /lower bound on the output. Figure 3 (right) on the other hand shows the certified mean error of the pose estimation as function of r, to better illustrate the growth of the error. It can be observed for Ez, the mean error is growing smoothly and when it reaches this threshold, significant increase occurs in the error due to activation of the penalty term (+K). However, shown for both small and large r values, the smoothed functions offer better error rates. Examples of attacked images are shown in Figure 0
(0.025, 0.05 Generally,
7 Conclusion
In this paper, we investigated certified robustness against adversarial perturbation in regression tasks. We analyzed the robustness of a base regression model without any smoothing, using only black-box access to the model. We then proposed a new variant. of certification where the outputs are bounded. Subject to the user's flexibility, a discounted certificate approach was proposed for bounded outputs where the result is valid for a finite sample regime. The results were validated using simulation and camera localization, demonstrating effective bounds. A promising direction is to investigate other smoothing functions dedicated to continnous outputs and derivation of probabilistic certified guarantees for unbounded outputs. Although our definition of robustness is general, we only considered the case of & attacks results for other threat models would be valuable. In our results, outputs are analyzed individually, while in some applications it might be possible to analyze them by groupings for a joint analysis.
Broader Impact Statement
Adversarial examples demonstrate the vulnerability of many machine learning models to manipulation in contested environments. This paper considers defenses (via randomized smoothing) and robustness quantification, which are important approaches to improving resistance to attacks. As such, we believe this work has potential for positive societal benefit.
11

---

Published in Transactions on Machine Learning Research (04/2025)
Acknowledgement
This work was supported in part by the Department of Industry, Science, and Resources, Australia under AUSMURI CATCH, and the Australian Research Council under Discovery Project DP220102269.
References
Eric Brachmann and Carsten Rother. Visual camera re-localization from rgb and rgb-d images using dsac. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5847 5865, 2022.
Benedikt Brückner and Alessio Lomuscio. Verification of neural networks against convolutional perturbations via parameterised kernels. arXiv preprint 2Xiv:441.00450, 2024.
Nicholas Carlini, Florian Tramer, Krishnamurthy Dj Dvijotham, Leslie Rice, Mingjie Sun, and J Zico Kolter. (certified!!) adversarial robustness for free! International Conference Learning Representations, 2023.
Ping-ych Chiang, Michael Curry, Ahmed Abdelkader, Aounon Kumar, John Dickerson, and Tom Goldstein. Detection as regression: Certified object detection with median smoothing. Advances in Neural Information Processing Systems, 33:1275- 1286, 2020.
Charles Clopper and Egon Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404 413, 1934.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference machine learning, pp. 1310 1320. PMLR, 2019.
Mare Fischer, Maximilian Baader, and Martin Vechev. Certified defense to image transformations via randomized smoothing. Advances in Neural information processing systems, 118404-8411) 2020.
Zayd Hammoudeh and Daniel Lowd. Reducing certified regression certified classification for general poisoning attacks. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 484 523. IEEE, 2023.
Zhuoqun Huang, Neil G Marchant, Keane Lucas, Lujo Bauer, Olga Ohrimenko, and Benjamin IP Rubinstein. Rs-del: Edit distance robustness certificates for sequence classifiers via randomized deletion. In Thirby- seventh Conference on Neural Information Processing Systems, 2023.
Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization, In Proceedings of the IEEE international conference on computer vision, pp. 2938 2946, 2015.
Aounon Kumar, Alexander Levine, Tom Goldstein, and Soheil Feizi. Curse of dimensionality on randomized smoothing for cortifiable robustness. In International Conference on Machine Learning, pp. 5458-5467. PMLR, 2020.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP), pp. 656-672. IEEE, 2019.
Guang- He Lee, Yang Yuan, Shiyu Chang, and Tommi Jaakkola. Tight certificates of adversarial robustness for randomly smoothed classifiers. Advances in Newral Informalion Processing Systems, 32, 2019.
Linyi Li, Tao Xie, and Bo Li. Sok: Certified robustness for deep neural networks. In 2023 IEEE symposium on security and privacy (SP), pp. 1289-1310. IEEE, 2023.
Jiang Liu, Alexander Levine, Chun Pong Lau, Rama Chellappa, and Soheil Feizi. Segment and complete: Defending object detectors against adversarial patch attacks with robust patch detection. In Proceedings of the IFFF/CVF Conference on Computer Vision and Patlern Recognition, pp. 14973-14982, 2022.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Aref Miri Rekavandi, Farhad Farokhi, Olga Ohrimenko, and Benjamin Rubinstein. Certified adversarial robustness via randomized a-smoothing for regression models. Advances in Neural Information Processing Systems, 37:134127- 134150, 2024.
Jeet Mohapatra, Ching-Yun Ko, Tsui-Wei Wei Weng, Pin-Yu Chen, Sijia Liu, and Luca Daniel. Higher-order certification for randomized smoothing. Advances in Neural Information Processing Systems, 33:4501-4511, 2020.
Mikhail Pautov, Olesya Kuznetsova, Nurislam Tursynbek, Aleksandr Petiushko, and Ivan Oscledets. Smoothod embeddings for certified few-shot learning. Advonces in Neural Information Processing Systems, 35:24367- 24379, 2022a.
Mikhail Pautov, Nurislam Tursynbek, Marina Munkhoeva, Nikita Muravev, Aleksandr Petiushko, and Ivan Oscledets. Cc-cert: probabilistic approach cortify general robustness of neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence volume 36, pp. 7975 7983, 2022b.
Aref Miri Rekavandi, Abd-Krim Seghouane, and Robin J Evans. Robust subspace detectors based on a- divergence with application to detection in imaging. IEEE Transactions on Image Processing, 30:3017-5031, 2021.
Aref Miri Rekavandi, Farid Boussaid, Abd-Krim Seghouane, and Mohammed Bennamoun. B-pose: Bayesian deep network for camera 6-dof pose estimation from rgb images. IEEE Robotics and Automation Letters, 2023.
Hadi Salman, Jerry Li, llya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers Advances in Newral Information Processing Systems, 32, 2019.
Hadi Salman, Mingjie Sun, Greg Yang, Ashish Kapoor, and Zico Kolter. Denoised smoothing: A provable defense for pretrained classifiers. Advances in Neural Information Processing Systems, 33:21945 21957, 2020.
Jiaye Teng, Guang- He Lee, and Yang Yuan. ell adversarial robustness cortificates: a randomized smoothing approach. 2019.
Greg Yang, Tony Duan, Edward Hu, Hadi Salman, Iyya Razenshtoyn, and Jerry Li. Randomized smoothing of all shapes and sizes. International Conference on Machine Learning, pp. 10693 10705. PMLR, 2020.
Dinghuai Zhang, Mao Ye, Chengyue Gong, Zhanxing Zhu, and Qiang Liu. Black-box certification with randomized smoothing: A functional optimization based framework. Advances in Neural Information Processing Systems, 33:2316 33:2316-2326, 2020.
13

---

Published in Transactions on Machine Learning Research (04/2025)
A Proof of Base Regression Certification
We repeat the thcorem's statement here for convenience, followed by its proof.
Theorem 1: (Certification General Models Against E2- Bounddd Attack). Let fo(x) R2 - R be a (possibly) randomized base regressor and ~ N(0, 0² I). Suppose for some example (z, y),
P(diss, (fa(s y,) S Cyp) 2 PA, V
(14)
where PA; is the lower bound the probability of accepting prediction in the 2th output variable. Then referring to Definition 1, fo e) is probabilistic certifiably robust at ezample (a, y) for a E--norm dissimilarity in the impul, chosen probability P S minge]e] PA,, oulpul radii e......, and impul radius
Cx
min o (pA) Φ (P))
(15)
We need the following classical result to prove our theorem.
Lemma 1: (Cohen et al. 2019 (Neyman-Pearson for Two Gaussians with Different Means and the Same
Variances). Let X ~ ox and Q ~ ~N(x 8, I and let h Rd + {0.1) be a deterministic or random function, then if = {z E ga _8 z S B} for some and P(h(X) = 13 2 P(X E S}, then =
Proof of Theorem 1. The proof of this theorem analogous to the proof of Theorem of Cohen et al. (2019). since divide the output space into two regions: acceptable and rejectable zones (like a binary classifica tion task). However, instead of having two classes to compete, the probability score of the regression prediction should beat the user-defined probability level P. In this setup, predict target variables simultaneously, Without loss of generality, we derive the maximum tolerable deviation in the input for each output variable. Then, based on the definition of the robust regression model, we find the variable with the worst guaranlee to find a value valid for all outputs. For 7th target variable, let us define the accepted deviation in the output space (by user) as the deviations such that (f,s, + y.) I Ezg. We denote the probability of this event with P Pddiss, (f (fx e),, I E Now let us define random variables X x + e and Q = x 6 + e, where (O, TT). 8 represents the adversarial perturbation in the input. Based the assumption made in the theorem statement, we know that P Pddiss, (fo(X); y:) such that
S E } 2 PA and the goal is to derive bound for ô
Pldiss, (fa(Q)) (Q),, S 9, 2
(16)
Defining a half-space = {z 1 (z x) S / |0 (PA,)) we can show that P(X E A} = PA, therefore, one can say
1 P(X A). Now we apply Lemma 1 with h(z) = Laiss, (fo(x) and conclude
P Pldiss, (fo
(17)
(fo(), S c 2 P(Q A).
(18)
For PP(Q& E A} we have
P(Q A}
P(OT(Q (Q x) S =
'(pA,)
PlooT P(ooTN((0.1) N(O. < 01 11(pA) (10112)
P (PN(O,, < <-- --' ((AA)) |0112
|6||
$ (PA,)
14

---

Published in Transactions on Machine Learning Research (04/2025)
Lower Bound of P_A with 1=10 and X=9
Bound with n=50 and
Actual
0.8
0.7
0.5
0.3
0.2
0.0
0.5
0.6
0.7
0.8
0.9
1.0
0.6
Confidence
Lower Bound of P A with 1000 and X=785
Lower Bound 10000 and X=7995
0.8
0.7
0.7
0.5
2.A. Lower
0.4
0.4
2.A.
0.3
0.2
0.1
Actual P
0.0
0.5
0.6
0.7
0.8
0.9
1.0
Confidence
Confidence
Figure 7: The Clopper-Pearson Pearson estimation of PA in different sample sizes and desired confidence levels. Notably, since this is probabilistic lower bound estimator, as shown in the top-left figure, the actual pa might also be smaller than the proposed when the confidence level is low. However, this is an unlikely event in larger confidence levels and the proposed lower bound is reliable for subsequent analysis. Values of X are randomly drawn from the actual distribution.
Therefore,
S k.p 2 $ x (00000'0))) (D-( |
If decides to set lower bound of PPdiiss, (fo (Q)) y,) S [. } to the corresponding ó should satisfy
|8||2 S ( (pA,) x (P)).
(19)
This ||ō]| indicates the maximum permitted perturbation in the input which guarantees with probability P, the 7th output is valid to the user. Since we have i output variables, each one with its own permitted perturbation ranges, we estimate t different permitted 6, and finally, give certificate that all these predictions are certifiable with probability P. In other words, we select the worst estimated value given by
Eg = min (001(((-))) (PA,) x --((P)).
(20)
sE(d]
This completes the proof.
15

---

Published in Transactions on Machine Learning Research (04/2025)
II N (v. = f S expl
m) p) - - m)}
P(g (x 5)
ds
(23)
((T)) (de((2)
i=1
#(vnVV -VVni->( (us 8n (x + 8)))
Σ (-1)+- Σ
-8n (x +x++8) $ (V =cD
DER,
B Estimating PA via Clopper-Pearson
In this section, mumerically show how the Clopper-Pearson (Clopper & Pearson 1934) interval proposal, ie., solution of Σi=x (2) = Pa)"~*=
for given a, and X can offer confidence interval, ie., [a. 1] in finite sample regime with confidence level 1 - in containing the actual parameter P'A- We consider the cases where (10, 50, 1000, 10000} and Pa set to TO For each case, we binomial distribution to draw the number of successful events (X), and then use the Clopper-Pearson to estimate PA for various confidence levels. Figure 7 illustrates the lower bounds (in bluc) for all these cases and as shown for higher confidence levels, e.g. 0.95, the offered lower bound is uniformly smaller than the actual PA (shown in red dashed). For larger n values, the gap between P'A and PA tends to zero, meaning that for larger n, we can simply use the maximum likelihood estimate PA as the PA (as we do in the simulation section). Using this concept, within this paper, the regression certification comes in the following format: " "With confidence ;, the given model at point (æ o, v), V||ō||2 Ez is certifiably robust with probability P."
C Proof of Asymptotic Certification of Averaging Function
We repeat the theorem's 's statement here for convenience, followed by its proof.
Thcorem (Asymptotic Behaviour of g(x) Against Fixed Attack). Let fa(x:) Rdd RR be (possibly) randomized base regressor and suppose for a given and outputs generated by fo( 8 e), with e~ ~~(00, of bounded covariance Σ E Rext_ If the accepted region (set) for each output target variable is conver then for the user-defined Ey. as n + 00, (x 8) E No cy.) is given by
independent and identically distributed with unknown mean m E R' and unknown
Vn
[ub gn (x (xxx+)0))
E(-)) Σ + (Vn : ((x+))))
(21)
DERa
where (D-) is the cumulative distribution function of a standard multivariate normal distribution, cp uses the lower bounds b. for all i E 'D and the upper bounds us for all & 'D such that R2 denotes the class of all subsels of [t] with exactly h elements, and 9n (x) is given by
8, (x)
Σ fo(x + e.).
(22)
the above, us and 4n are upper and lower bounds on the accepted region in output, and E is a consistent covariance estimator.
Proof. For clarity of the proof, refer the reader to Figure which illustrates the setup for the case (t = d= 2). Since t-dimensional outputs are iid by assumption, based the Central Limit Theorem, we have
VT(g,, 5) m) no000 M, E).
(24)
16

---

Published in Transactions on Machine Learning Research (04/2025)
y/
b1
Uhi
Mb2
N(X, Ex)
(x)
X
12
fo())
N(v.cy)
11
Figure 8: Schematic of the asymptotic behavior of averaging function in output domain for d = t 2. As shown, the Bn will be Gaussian distributed as + The accepted region is shown in green colour.
1
Ui
]
fx
Figure The general schematic of tth output variable when upper and lower bounds of the variable are denoted by u; and I and upper and lower bounds of the acceptable region are shown by Ub and lb-- The accepted region is shown in green colour.
In other words, as n > 00, gn (x + 5) ~ (m, E) which shows the average value concentrates around the expected value of outputs and as n grows, the covariance gets smaller and smaller. Since for a deep net work, m and Σ are unknown and they are both functions of 8, by Weak Law of Large Numbers we may replace them with their sample mean and sample covariance estimates any other consistent estimators. Now, to estimate the probability of gn 5) to be accepted by the user, need to use (23) which is based the inclusion-exclusion principle. As is shown this probability function (not lower bound) is a function of estimated covariance, upper bound and lower bound of the accepted region denoted by us and l,, and 6. This completes the proof.
D Proof of Asymptotic Certification of Averaging Function for Bounded Outputs
We repeat the theorem's statement here for convenience, followed by its proof.
Theorem 3: (Certification of g(x) Against & Attack for Bounded Outputs). Let fo(x) : RR be deterministic or random base regressor and suppose outputs generated by fo(z) are independent and identically distributed with user-defined cy (equivalent to us and lo) to define the accepted region. Suppose for NN, o2 V|[ō|12 < Exy and an arbitrary value of p, as stated in (4 of the main manuscript), Vi E [t]:
(idis (f 8 + e)s, 4:) S c 2p.
(25)
Considering bounded output cases, i.e., I (z) S u. Vz subject to 1 1 4 S S u, and if for those samples which are accepted by the user, we have [ x + ö + e)}; ( <T. 0 O & min(fo(æ) 4. u. fo(x)), for the convex accepted region (set) and as n + oo, the following inequality holds E [t]
(Idissy(nn.
8) v,) 5 C 2
"y _o(m), )1, [n [p([n(1-
Mb: f0(2).- + 1),
if (2), otherwise,
2
4 4-4,
min rE(d]
(z)
(m)
--)
(26)
[4[[n[1
fa(an) - by
fo(0), )1, [n
[
4.
1
+1)
[z)
-
17

---

Published in Transactions on Machine Learning Research (04/2025)
where Ip(a,b) is the regularized incomplete beta function defined Ip(a,b) B(a.) tet (1-1-11 dt and B(a, b) is the complete beta function.
Proof. Considering the worst-case scenario find the lower bound on the probability of estimating valid En suggests considering the output generated by fx 8 + e); follows a Bernoulli distribution where the outcome is located with probability p fx(x), +T (successful event and true + oo) and with probability u,-ub
+T
1,=,, - p at u, (unsuccessful event) if -x=)=(x):-7 2 -x):--7-1i (see Figure 9). otherwise, at fo(x); - r and 4, Without loss of generality let us assume for 1thh output, the former condition satisfied. Now by defining the random variable O the number of valid outputs, have
Pidiss, (g.( (x + y.) S E
Pldiss, (g. (x + 8)),)))) S
n1 (10))
000 "4 (x))e-r
y=
[n[] "by (N),
(p) p)-
P=0
[a 98, Gg60
11 (1 (1 - (p)
v=0
= [n(n((1 Wb. fx(x);
Ub. - -x(()) -T + 1).
11; fo(x);
fo(x);
This is a lower bound for the probability of one of the outputs being in the accepted region if u.-ub; 1,-1, 1x) For the other condition, we only replace us and u, with 4, and 1, and take care of signs. To derive the same lower bound for the entire output variables, one can take a minimum over all these probability values to complete the proof.
E Proof of Noy-Asympo Certification of Averaging Function for Bounded Outputs
We repeat the theorem's 's statement here for convenience, followed by its proof.
Proposition 1: (Discounted Certification of g(x) against &2 Attack for Bounded Outputs). Let fo( Rd + R' be a (possibly) randomized base regressor and suppose outputs generated by fo(z) are independent and identically distributed with a user-defined Ey (equivalent. to us and b)) to define the accepted region. Suppose for e N~100 o² V| S ea, and an arbitrary value of p. as stated in (4) vi e [t]:
Pidiss, (fo( + 5 E) w:) S Sc,
(27)
Considering bounded output cases, i.e., i fz) S u, Vz subject to 1 1 46 S u and assuming the accepted region (set) for each output target variable to be conver, then given a discount factor 2 such that 1 4 ()ff(z) 41 S us + fz()) uel I u holds, then the following inequality holds for Vi [t]
P (g, 8)s, v.) < (1 3)
(TTn(11
Alub f2(x)el 17, [r
3 Me; [o(
) + 1), + 1);
of M0 fo(m)a)
14, fo (x))al
min *e[d
814, us nh, (()2
)7. 3 l,, (2)il
us we,
4, -1
(28)
[e([n(1-
otherwise,
4
18

---

Published in Transactions on Machine Learning Research (04/2025)
4
bp
Ub;
fx(x)
Bl:b, (x)
Wb; + B ub. - (x)i
Figure 10: The upper and lower bounds of ith output variable are denoted by u, and 4, and the upper and lower bounds of the acceptable region for the base regression model are shown by green colour between us and l,,- In the discounted certificate case, the user is asked to extend this acceptable region utilizing new bounds and the discount factor 3 as shown in the figure.
1 where Ip(a, b) is the regularized incomplete beta function defined as Ip(a,b) = B(a.)) So t0 -2) dt and B(a,b) is the complete beta function.
Proof. The proof is similar to the proof of Theorem 3 where the lower and upper bounds of the accepted region are replaced with the discounted bounds. Considering the worst-case scenario to find the lower bound on the probability of estimating valid 8n suggests considering the output generated by fø(x + d + e); follows a Bernoulli distribution where the outcome is located with probability at us (successful event) and
with probability 1 1- P at u, (unsuccessful event) [u, (x)):l I1, (x)= (see Figure 100, otherwise, at ba -1
and 4. Without loss of generality let us assume for 5th output, the former condition is satisfied. Now by defining the random variable as the number of valid outpuls, we have
P Pidiss, (g. (x + y)), S (1 + })
P Pdiss (B, (x + 5).. S (1 1
( (1-)(1-p)) p)"-
v
8 Blus f((x)
vmn
u. U6;
Alm, (6())cc
[n[
U3 4b
(p)" (1- pp)
m=0
3lu; Cg(K)s
(1 p)" (p)
D=0
3 Ub: (x(x)
3 8lub (x)x
[[[n(11
}1, [m
17 + 1).
u
18b,
u, ub;
This is a lower bound for the probability of one of the outputs being in the accepted region if | 1b ff(x)d For the other condition, we only replace us and u, with 4 and I and take care of signs. To derive the same lower bound for the entire output variables, one can take minimum over all these probability values to complete the proof.
us,
by
F Estimated Radius vs P
Figure | illustrates the input upper bound on the maximum certifiable perturbation as a function of the user-defined probability. If the user asks for larger portion of valid outputs, the balball around the basc prediction gets smaller.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Sigma=0.23 PA=0.99
Radius
0.3
0.2
0.0
0.5
0.6
0.7
0.8
0.9
1.0
Figure 11: Adversarial upper bound vs user defined probability.
20


Reviews:
[{"review_id": "6DsfChil89", "reviewer": "Reviewer_avtB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength is that the results are solid and well-supported, so that I don't have substantive complaints.\n\nThe main weakness is that the paper is a little terse and hard to follow sometimes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is an extension of randomized smoothing to regression tasks. The general idea of random smoothing is that by convolving a predictor with some noise, it can be made smoother. If sufficiently smooth, it will be difficult for an adversary to flip the outputs by perturbing the function. \n\nIn the most traditional model, the task is classification, the noise is Gaussian, the adversary is assumed to be bounded to perturb inputs within an l2 ball, and the resulting smoothness is Lipschitz in an l2 sense. There are various elaborations on this framework considering different geometries for the adversary and therefore different types of noise, as well as attacks such as patch attacks that can’t be characterized as lying within some metric ball. The most related work, which the authors point out, is by Chiang et al., which reduces bound-box object detection to regression, and then applies a median-based smoothing approach. The results here are quite different.\n\nThe authors here focus on regression tasks (possibly multi-dimensional), and they consider a slightly different threat model — the adversary still perturbs within an l2 ball, but now the requirement for robustness of the outputs is if they do move outside some distance, this happens with a low-enough probability.\n\nThe authors give a randomized smoothing approach that works under this threat model. Some technical differences arise from the classification case, because the exact location of the true output may vary (unlike in the classification case where the goal of the adversary is just to flip a class). The authors consider the very reasonable special case of bounded outputs (where the user wants even tighter bounds than are known a priori) to deal with this problem. In experiments, they show that one can adversarially-train a classifier and then apply smoothing to get practically effective results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the explanation of the discounting result (Proposition 1) would be helpful, as I had a hard time understanding this part of the paper and what the discount is supposed to mean about the user's threat model. It also couldn't hurt to actually write out the proof (even though almost the same as another proof) in the appendices.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y7ch3le8EQ", "reviewer": "Reviewer_ovzF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this paper lie in the rigorous theoretical analyses of how randomized smoothing can be applied to regression tasks. However, a major issue of this paper is its large text overlap with a NeurIPS 2024 paper [1] (not cited anywhere in the manuscript), including problem setup, certification framework, and experimental results. In my opinion, [1] provides a more general certification method for regression models, covering both bounded and unbounded output settings. As a result, I do not see how this paper contributes to the field with reference to [1], despite the fact that [1] cited this paper as a preliminary baseline. \n\nThe experimental results of this paper are fairly limited. In the related work section, you mentioned a few existing works (Pautov et al., 2022a; Salman et al., 2019; Chiang et al., 2020) that extend randomized smoothing outside classification, but they impose different constraints on the regression outputs or use different techniques. It is not clear why the authors do not conduct comparison studies with these related methods. Figure 2 and Figure 3 are of low resolution, which also takes up much space than necessary. I feel the experimental figures presented are difficult to parse, and there is a lack of clear explanations about the experimental settings (for general regression tasks and for the specific tasks considered in the experiments). In addition, I found it difficult to understand the introduced terminologies from place to place, especially the mathematical notations and the technical assumptions introduced in Section 4. \n\n[1] Certified Adversarial Robustness via Randomized α-Smoothing for Regression Models, Rekavandi et al., NeurIPS 2024, https://openreview.net/pdf/2c1928be34a76219c66d9466b72ff8cde2c15a4e.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a randomized smoothing-based framework for regression models. In particular, it proposes a definition of a probabilistic robustness certificate for regression tasks (Definition 1) and a series of theoretical analysis results for deriving the robustness certificate under asymptotic and non-asymptotic (bounded) output cases. Empirical evaluations on a simulation dataset and a camera re-location task are provided.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Explain how your work contributes to the field with reference to the NeurIPS work of [1].\n\n2. Highlight more on the technical challenges you encountered in proving the theoretical results in Section 3. Given the seminal randomized smoothing work of Cohen et al. (2019), the technical contributions of this work are not clear, especially given the several assumptions imposed in the technical theorems.\n\n3. Did you compare your robustness certificate with empirical attacks developed by Chiang et al. (2020) and Liu et al. (2022)? How tight is your robustness certificate?\n\n4. Improve the presentation of Section 3 by providing clearer explanations of the mathematical notations and detailed discussions of the assumptions.\n\n5. Conduct more comprehensive experimental results to benchmark the proposed method's performance and compare it to related methods.", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A433ZllyQa", "reviewer": "Reviewer_ESMF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper formally studies the extension of RS to regression tasks, a critical and understudied (in this context) area of machine learning.\n2. The approach of using averaging instead of just randomization for certifying robustness in this way seems novel and may be useful in settings where a deterministic classifier is desirable.\n3. The authors verify their results via a thoroughly analyzed set of experiments in both a synthetic data setting and a more practical task from computer vision. They also promise to provide code to reproduce these results.\n\n## Weaknesses\n1. While Theorem 1 is fairly clear, Theorems 2 and 3 are not stated in a way that allows for clear insights into the dependence of the bounds on the underlying problem; it seems the focus is only on enabling the bounds’ computation (which is useful, but less clear). While the remarks following these theorems help somewhat, it is hard to understand how the probability depends on the allowed perturbation in a clear, functional way. Corollaries about special cases following the theorems would be useful to the reader.\n2. Theorems 2 and 3 (as well as Proposition 1) make use of a confusing asymptotic setting where the limit as n approaches infinity of a quantity that depends on n is yet another quantity that depends on n. It is reasonable to say that the limit of an (n-dependent) probability is the limit of another (n-dependent) probability, but it is not clear what it means when the limit of an (n-dependent) probability is an n-dependent function that becomes 1 when n goes to infinity. From the proof I believe the issue is direct substitution of the n-dependent CLT expression rather than working with limits.\n3. The non-asymptotic averaging seems crucial for any practical implementation of this method, but it is not thoroughly discussed and requires additional assumptions on top of the (already possibly unrealistic, given the application to softmax object detection) boundedness assumptions.\n4. There are not very many motivations provided in terms of applications.\n5. There is a definition provided for certification in the regression case, but the theorems do not make use of this definition (even if they do imply that the models satisfy it).\n6. It is not clear to me that norm-based perturbation is the right model for practical adversarial robustness, although such results could end up being useful beyond the scope of the paper.\n\n## Summary \nI think the paper is of sufficient interest to the ML community to be accepted at TMLR after addressing the issues with the presentation of Theorems 2 and 3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an extension of the randomized smoothing (RS) approach to certifiable adversarial robustness to regression tasks. The authors propose a definition of probabilistic robustness in this new setting, show the type of guarantee a standard perturbation-based approach yields (Theorem 1), and then discusses a new approach in which an average over the outputs of randomly perturbed inputs is taken (Theorems 2 and 3, and Proposition 1). The paper concludes with experimental results demonstrating the certification on a synthetic regression task and on a computer vision task.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. A discussion of directions for extending the guarantees beyond l2 perturbations. Are there naive ways to do so? Is there a principled geometric approach?\n2. In the third bullet point of Section 1, it’s not very clear without reading the paper what “RS leads to [a] multivariate normal density” means, although point (ii) in the paragraph immediately below is clearer about this.\n3. In the last paragraph of Section 1, it is not clear that the asymptotic variable is the number of samples from a Gaussian used to perturb the input and then average the output.\n4. “recent study Hammoudeh” -> “recent study by Hammoudeh”\n5. Are there notable applications that you can cite for certified regression outside the case of object detection where the target is the Softmax?\n6. Fixing some uses of \\citet instead of \\citep (e.g. in the Randomized Smoothing part of Section 3).\n7. Definition 1: “If” -> “if”\n8. Theorem 1: “definition Eq. (2)” -> “Definition 1”\n9. The discussion around Theorem 1 is confusing to me: in the paragraph before it says it is about an arbitrary model that has _not_ undergone randomized smoothing, but in the theorem itself the input to the model is perturbed by a vector e in the exact same way that randomized smoothing operates. How is the theorem _not_ about a function that has undergone randomized smoothing?\n10. [critical] The meaning of Theorem 2 is very hard to parse. It seems to take n to infinity, but the primary equation (5) has n in it. Furthermore, in the remarks below it would be helpful to have an explainer for how to interpret the double summation in equation 5, and specifically the effect of the subsets in R_k.\n11. [critical] Similarly, Theorem 3 and Proposition 1 are challenging to parse and involve unclear asymptotic notation.", "broader_impact_concerns": "Broader impacts are sufficiently addressed in the submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The relevant details are discussed above.
Decision: Accept (poster)