SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We infer a regression function  $f$  from a GP prior as described above with the additional supervision of  $[\partial f(\mathbf{x}) / \partial x_2]|_{\mathbf{x}^{(i)}} = 0, \quad \forall i \in [1, N]$ . Then the function value deviations to perturbations on irrelevant feature are lower bounded by a value proportional to the perturbation strength  $\delta$  as shown below.
$$
f (\mathbf {x} + [ 0, \delta ] ^ {T}) - f (\mathbf {x}) \geq \frac {2 \delta \alpha}{\beta} \Theta (x _ {1} ^ {2} x _ {2} ^ {6} + \delta x _ {1} ^ {2} x _ {2} ^ {5})
$$
Proof outline. We first show that the posterior mean of the function estimates marginalised over hyperparameters with Gamma prior has the following closed form where  $d(x,y) = (x - y)^2 /2$  and  $\tilde{y}$  denotes original observations  $y$  augmented with observations on gradients, which is described in more detail further below.
$$
f (x) \triangleq \mathbb {E} _ {\theta} [ m _ {x} ] = \int \int m _ {x} \mathcal {G} (\theta_ {1} ^ {- 2}; \alpha , \beta) \mathcal {G} (\theta_ {2} ^ {- 2}; \alpha , \beta) d \theta_ {1} ^ {- 2} d \theta_ {2} ^ {- 2}
$$
$$
f (\mathbf {x}) = \sum_ {n = 1} ^ {N} \left(\frac {1}{1 + \frac {d (x _ {1} , x _ {1} ^ {(n)})}{\beta}}\right) ^ {\alpha} \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} \left[ \tilde {y} ^ {(n)} + \frac {\frac {\alpha}{\beta} (x _ {2} - x _ {2} ^ {(n)})}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}} \tilde {y} ^ {(n + N)} \right]
$$
We then derive the following lower bound on the function value deviations and finally use simple inequalities to arrive at the final result.
$$
\begin{array}{l} f (\mathbf {x} + [ 0, \delta ] ^ {T} ]) - f (\mathbf {x}) \geq \frac {2 \delta \alpha}{\beta} \sum_ {n} \left(\frac {1}{1 + \frac {d (x _ {1} , x _ {1} ^ {(n)})}{\beta}}\right) ^ {\alpha} \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha + 1} \\ \left[ (\alpha + 1) \tilde {y} _ {n + N} \left(\frac {2 (x _ {2} - x _ {2} ^ {(n)}) [ x _ {2} + \delta - x _ {2} ^ {(n)} ]}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} - 1\right) - \tilde {y} _ {n} \right] \\ \end{array}
$$
Proof. We first derive the augmented set of observations  $(\hat{y})$  and  $\hat{K}$  explained in the main section.
$$
\hat {y} = \left[ y _ {1}, y _ {2}, \dots , y _ {N}, \partial f (\mathbf {x} ^ {(1)}) / \partial x _ {2}, \partial f (\mathbf {x} ^ {(2)}) / \partial x _ {2}, \dots , \partial f (\mathbf {x} ^ {(N)}) / \partial x _ {2} \right] ^ {T}
$$
$$
k (x ^ {(i)}, x ^ {(j)}) = \left\{ \begin{array}{l l} \exp (- \frac {1}{2} \sum_ {k = 1} ^ {2} \frac {(x _ {k} ^ {(i)} - x _ {k} ^ {(j)}) ^ {2}}{\theta_ {k} ^ {2}}) & \mathrm {w h e n i , j \leq N} \\ \frac {(x _ {2} ^ {(i)} - x _ {2} ^ {(j)})}{\theta_ {2} ^ {2}} \exp (- \frac {1}{2} \sum_ {k = 1} ^ {2} \frac {(x _ {k} ^ {(i)} - x _ {k} ^ {(j)}) ^ {2}}{\theta_ {k} ^ {2}}) & \mathrm {w h e n i \leq N , j > N} \\ - \frac {(x _ {2} ^ {(i)} - x _ {2} ^ {(j)})}{\theta_ {2} ^ {2}} \exp (- \frac {1}{2} \sum_ {k = 1} ^ {2} \frac {(x _ {k} ^ {(i)} - x _ {k} ^ {(j)}) ^ {2}}{\theta_ {k} ^ {2}}) & \mathrm {w h o n j \leq N , i > N} \\ - 2 \frac {(x _ {2} ^ {(i)} - x _ {2} ^ {(j)}) ^ {2}}{\theta_ {2} ^ {4}} \exp (- \frac {1}{2} \sum_ {k = 1} ^ {2} \frac {(x _ {k} ^ {(i)} - x _ {k} ^ {(j)}) ^ {2}}{\theta_ {k} ^ {2}}) & \\ + \frac {1}{\theta_ {2} ^ {2}} \exp (- \frac {1}{2} \sum_ {k = 1} ^ {2} \frac {(x _ {k} ^ {(i)} - x _ {k} ^ {(j)}) ^ {2}}{\theta_ {k} ^ {2}}) & \mathrm {w h e n i , j > N} \end{array} \right.
$$
These results follow directly from the results on covariance between observations of  $f$  and its partial derivative below (Hennig et al., 2022).
$$
\operatorname {c o v} (f (x), \frac {\partial f (\tilde {x})}{\partial \tilde {x}}) = \frac {\partial k (x , \tilde {x})}{\partial \tilde {x}}
$$
$$
\operatorname {c o v} \left(\frac {\partial f (x)}{x}, \frac {\partial f (\tilde {x})}{\tilde {x}}\right) = \frac {\partial^ {2} k (x , \tilde {x})}{\partial x \partial \tilde {x}}
$$
The posterior value of the function at an arbitrary point  $\mathbf{x}$  would then be the form  $p(f(\mathbf{x})\mid \mathcal{D})\sim \mathcal{N}(f(\mathbf{x});m_x,k_x)$  where  $m_{x}$  and  $k_{x}$  are have the following closed form for Gaussian prior and
Gaussian likelihood in our case.
$$
m _ {x} = k (x, X) K _ {X X} ^ {- 1} \hat {y}
$$
$$
k _ {x} = k (x, x) - k (x, X) K _ {X X} ^ {- 1} k (X, x)
$$
Since  $m_x, k_x$  are functions of the parameters  $\theta_1, \theta_2$ , we obtain the closed form for posterior mean by imposing a Gamma prior over the two parameters. For brevity, we denote by  $d(x, \tilde{x}) = (x - \tilde{x})^2 / 2$  and  $\tilde{y}^{(i)}$  is the  $i^{(th)}$  component of  $\hat{K}_{XX}^{-1}\hat{y}$ .
$$
\begin{array}{l} f (x) \triangleq \mathbb {E} _ {\theta} [ m _ {x} ] = \iint m _ {x} \mathcal {G} \left(\theta_ {1} ^ {- 2}; \alpha , \beta\right) \mathcal {G} \left(\theta_ {2} ^ {- 2}; \alpha , \beta\right) d \theta_ {1} ^ {- 2} d \theta_ {2} ^ {- 2} \\ = \int \int \left[ \sum_ {n = 1} ^ {N} k (x, x ^ {(n)}) \tilde {y} _ {n} + \sum_ {n = 1} ^ {N} \frac {(x _ {2} - x _ {2} ^ {(n)})}{\theta_ {2} ^ {2}} k (x, x ^ {(n)}) \tilde {y} _ {n + N} \right] \mathcal {G} (\theta_ {1} ^ {- 2}; \alpha , \beta) \mathcal {G} (\theta_ {2} ^ {- 2}; \alpha , \beta) d \theta_ {1} ^ {- 2} d \theta_ {2} ^ {- 2} \\ \end{array}
$$
$$
\begin{array}{l} \int \int k (\mathbf {x}, \mathbf {x} ^ {(n)}) \tilde {y} _ {n} \mathcal {G} \left(\theta_ {1} ^ {- 2}; \alpha , \beta\right) \mathcal {G} \left(\theta_ {2} ^ {- 2}; \alpha , \beta\right) d \theta_ {1} ^ {- 2} d \theta_ {2} ^ {- 2} \\ = \int \int \exp \left(- \frac {\theta_ {1} ^ {- 2} (x _ {1} - x _ {1} ^ {(n)}) ^ {2}}{2} + \frac {\theta_ {2} ^ {- 2} (x _ {2} - x _ {2} ^ {(n)}) ^ {2}}{2}\right) \frac {\beta^ {\alpha}}{\Gamma (\alpha)} \theta_ {1} ^ {- 2 \alpha + 2} \exp (- \beta \theta_ {1} ^ {- 2}) \\ \frac {\beta^ {\alpha}}{\Gamma (\alpha)} \theta_ {2} ^ {- 2 \alpha + 2} \exp (- \beta \theta_ {2} ^ {- 2}) \tilde {y} _ {n} d \theta_ {1} ^ {- 2} d \theta_ {2} ^ {- 2} \\ = \left(\frac {\beta}{\beta + \frac {(x _ {1} - x _ {1} ^ {(n)}) ^ {2}}{2}}\right) ^ {\alpha} \left(\frac {\beta}{\beta + \frac {(x _ {2} - x _ {2} ^ {(n)}) ^ {2}}{2}}\right) ^ {\alpha} \tilde {y} _ {n} \\ \end{array}
$$
$$
\begin{array}{l} \int \int \frac {x _ {2} - x _ {2} ^ {(n)}}{\theta_ {2} ^ {2}} k (\mathbf {x}, \mathbf {x} ^ {(n)}) \tilde {y} _ {n + N} \mathcal {G} (\theta_ {1} ^ {- 2}; \alpha , \beta) \mathcal {G} (\theta_ {2} ^ {- 2}; \alpha , \beta) d \theta_ {1} ^ {- 2} d \theta_ {2} ^ {- 2} \\ = \left(x _ {2} - x _ {2} ^ {(n)}\right) \left(\frac {\beta}{\beta + \frac {\left(x _ {1} - x _ {1} ^ {(n)}\right) ^ {2}}{2}}\right) ^ {\alpha} \frac {\beta^ {\alpha} / \Gamma (\alpha)}{\left(\beta + \frac {\left(x _ {2} - x _ {2} ^ {(n)}\right) ^ {2}}{2}\right) ^ {\alpha + 1} / \Gamma (\alpha + 1)} \tilde {y} _ {n + N} \\ = \left(\frac {\beta}{\beta + \frac {(x _ {1} - x _ {1} ^ {(n)}) ^ {2}}{2}}\right) ^ {\alpha} \frac {\alpha (x _ {2} - x _ {2} ^ {(n)})}{\beta + \frac {(x _ {2} - x _ {2} ^ {(n)}) ^ {2}}{2}} \left(\frac {\beta}{\beta + \frac {(x _ {2} - x _ {2} ^ {(n)}) ^ {2}}{2}}\right) ^ {\alpha} \tilde {y} _ {n + N} \\ \end{array}
$$
Overall, we have the following result.
$$
f (x) = \sum_ {n = 1} ^ {N} \left(\frac {1}{1 + \frac {d (x _ {1} , x _ {1} ^ {(n)})}{\beta}}\right) ^ {\alpha} \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} \left[ \tilde {y} _ {n} + \frac {\frac {\alpha}{\beta} (x _ {2} - x _ {2} ^ {(n)})}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}} \tilde {y} _ {n + N} \right]
$$
We now derive the sensitivity to perturbations on the second dimension for  $\Delta \mathbf{x} = [0,\delta ]^T$
$$
\begin{array}{l} f (\mathbf {x} + \Delta \mathbf {x}) - f (\mathbf {x}) = \sum_ {n = 1} ^ {N} \left(\frac {1}{1 + \frac {d (x _ {1} , x _ {1} ^ {(n)})}{\beta}}\right) ^ {\alpha} \left\{\left[ \left(\frac {1}{1 + \frac {d (x _ {2} + \delta , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} - \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} \right] \tilde {y} _ {n} \right. \\ \left. \left[ \frac {\frac {\alpha}{\beta} \left(x _ {2} + \delta - x _ {2} ^ {(n)}\right)}{\left(1 + \frac {d \left(x _ {2} + \delta , x _ {2} ^ {(n)}\right)}{\beta}\right) ^ {\alpha + 1}} - \frac {\frac {\alpha}{\beta} \left(x _ {2} - x _ {2} ^ {(n)}\right)}{\left(1 + \frac {d \left(x _ {2} , x _ {2} ^ {(n)}\right)}{\beta}\right) ^ {\alpha + 1}} \right] \tilde {y} _ {n + N} \right\} \tag {7} \\ \end{array}
$$
Using Bernoulli inequality,  $(1 + x)^r \geq 1 + rx$  if  $r \leq 0$ , we derive the following inequalities.
$$
\begin{array}{l} \left(\frac {1}{1 + \frac {d (x _ {2} + \delta , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} - \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} \\ = \left(\frac {1}{1 + \frac {d \left(x _ {2} , x _ {2} ^ {(n)}\right)}{\beta}}\right) ^ {\alpha} \left[ \left(\frac {\beta + d \left(x _ {2} , x _ {2} ^ {(n)}\right)}{\beta + d \left(x _ {2} + \delta , x _ {2} ^ {(n)}\right)}\right) ^ {\alpha} - 1 \right] \\ \geq \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} - \alpha \left[ \frac {\beta + d (x _ {2} + \delta , x _ {2} ^ {(n)})}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} - 1 \right] \\ = \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} \alpha \left[ \frac {d (x _ {2} , x _ {2} ^ {(n)}) - d (x _ {2} + \delta , x _ {2} ^ {(n)})}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} \right] \\ \text {A s s u m i n g} \left| x _ {2} - x _ {2} ^ {(n)} \right| \gg \delta \quad \forall n \in [ N ] (8) \\ \approx \left(\frac {1}{1 + \frac {d \left(x _ {2} , x _ {2} ^ {(n)}\right)}{\beta}}\right) ^ {\alpha} \alpha \left[ \frac {- 2 \delta \left(x _ {2} - x _ {2} ^ {(n)}\right)}{\beta + d \left(x _ {2} , x _ {2} ^ {(n)}\right)} \right] (9) \\ \end{array}
$$
$$
\frac {\frac {\alpha}{\beta} (x _ {2} + \delta - x _ {2} ^ {(n)})}{(1 + \frac {d (x _ {2} + \delta , x _ {2} ^ {(n)})}{\beta}) ^ {\alpha + 1}} - \frac {\frac {\alpha}{\beta} (x _ {2} - x _ {2} ^ {(n)})}{(1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}) ^ {\alpha + 1}}
$$
$$
\begin{array}{l} \geq \frac {\alpha}{\beta} (x _ {2} - x _ {2} ^ {(n)}) \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha + 1} (\alpha + 1) \left[ \frac {- 2 \delta (x _ {2} - x _ {2} ^ {(n)})}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} \right] + \frac {\delta \frac {\alpha}{\beta}}{(1 + \frac {d (x _ {2} + \delta , x _ {2} ^ {(n)})}{\beta}) ^ {\alpha + 1}} \\ \geq \frac {\alpha}{\beta} (x _ {2} - x _ {2} ^ {(n)}) \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha + 1} (\alpha + 1) \left[ \frac {- 2 \delta (x _ {2} - x _ {2} ^ {(n)})}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} \right] \\ + \frac {\delta \frac {\alpha}{\beta}}{(1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}) ^ {\alpha + 1}} (\alpha + 1) \left[ \frac {- 2 \delta (x _ {2} - x _ {2} ^ {(n)})}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} + 1 \right] \\ = \frac {\alpha + 1}{(1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}) ^ {\alpha + 1}} \left[ \frac {- 2 \delta (x _ {2} - x _ {2} ^ {(n)}) ^ {2} \alpha / \beta - 2 \delta^ {2} \alpha / \beta (x _ {2} - x _ {2} ^ {(n)})}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} + \frac {\delta \alpha}{\beta} \right] \\ = \frac {- 2 \delta \alpha (\alpha + 1)}{\beta \left(1 + \frac {d \left(x _ {2} , x _ {2} ^ {(n)}\right)}{\beta}\right) ^ {\alpha + 1}} \left[ \frac {- 2 \left(x _ {2} - x _ {2} ^ {(n)}\right) \left[ x _ {2} + \delta - x _ {2} ^ {(n)} \right]}{\beta + d \left(x _ {2} , x _ {2} ^ {(n)}\right)} + 1 \right] \tag {10} \\ \end{array}
$$
Using inequalities 9, 10 in Equation 7, we have the following.
$$
\begin{array}{l} f (\mathbf {x} + \Delta \mathbf {x}) - f (\mathbf {x}) \geq \sum_ {n} \left(\frac {1}{1 + \frac {d (x _ {1} , x _ {1} ^ {(n)})}{\beta}}\right) ^ {\alpha} \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha} \\ \left[ \frac {- 2 \delta \alpha \tilde {y} _ {n}}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} + \frac {- 2 \delta \alpha (\alpha + 1) \tilde {y} _ {n + N}}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} \left(\frac {- 2 (x _ {2} - x _ {2} ^ {(n)}) [ x _ {2} + \delta - x _ {2} ^ {(n)} ]}{\beta + d (x _ {2} , x _ {2} ^ {(n)})} + 1\right) \right] \\ \end{array}
$$
$$
\begin{array}{l} f (\mathbf {x} + \Delta \mathbf {x}) - f (\mathbf {x}) \geq \frac {2 \delta \alpha}{\beta} \sum_ {n} \left(\frac {1}{1 + \frac {d (x _ {1} , x _ {1} ^ {(n)})}{\beta}}\right) ^ {\alpha} \left(\frac {1}{1 + \frac {d (x _ {2} , x _ {2} ^ {(n)})}{\beta}}\right) ^ {\alpha + 1} \\ \left[ (\alpha + 1) \tilde {y} _ {n + N} \left(\frac {2 \left(x _ {2} - x _ {2} ^ {(n)}\right) \left[ x _ {2} + \delta - x _ {2} ^ {(n)} \right]}{\beta + d \left(x _ {2} , x _ {2} ^ {(n)}\right)} - 1\right) - \tilde {y} _ {n} \right] \tag {11} \\ \end{array}
$$
Using the inequality  $(1 + x)^r\geq 1 + rx$  if  $r\leq 0$  , we have
$$
\begin{array}{l} f (\mathbf {x} + \Delta \mathbf {x}) - f (\mathbf {x}) \geq \frac {2 \delta \alpha}{\beta} \sum_ {n} \left\{\left(1 - \frac {\alpha}{\beta} d \left(x _ {1}, x _ {1} ^ {(n)}\right)\right) \left(1 - \frac {\alpha + 1}{\beta} d \left(x _ {2}, x _ {2} ^ {(n)}\right)\right) \right. \\ \left. \left[ \frac {\alpha + 1}{\beta} \tilde {y} _ {n + N} \left(2 (x _ {2} - x _ {2} ^ {(n)}) [ x _ {2} + \delta - x _ {2} ^ {(n)} ] (1 - d (x _ {2}, x _ {2} ^ {(n)})) - 1\right) - \tilde {y} _ {n} \right] \right\} \\ = \frac {2 \delta \alpha}{\beta} \Theta \left(x _ {1} ^ {2} x _ {2} ^ {6} + \delta x _ {1} ^ {2} x _ {2} ^ {5}\right) \\ \end{array}
$$
We restate the result of Theorem 2 for clarity.
When we use an adversarial robustness algorithm to regularize the network, the fitted function has the following property.
$$
\left| f \left(\mathbf {x} + [ 0, \delta ] ^ {T}\right) - f (\mathbf {x}) \right| \leq \frac {\alpha}{\beta} \delta_ {\max } f _ {\max } C
$$
$$
\text {w h e r e} C = \max  _ {\mathbf {x} \in \mathcal {X}} \min  _ {\hat {\mathbf {x}} \in \hat {\mathcal {X}}} | \mathbf {x} _ {2} - \hat {\mathbf {x}} _ {2} |
$$
$\delta_{max}$  and  $f_{max}$  are maximum value of  $\Delta x_2$  and  $f(\mathbf{x})$  in the input domain  $(\mathcal{X})$  respectively.  $\hat{\mathcal{X}}$  denotes the subset of inputs covered by the robustness method. C therefore captures the maximum gap in coverage of the robustness method.
Proof. We begin by estimating the Lipschitz constant of a GP with squared exponential kernel.
$$
\begin{array}{l} f (\mathbf {x}) = K _ {x X} K _ {X X} ^ {- 1} y \\ \frac {\partial f (x)}{\partial x _ {2}} = \frac {\partial K _ {x X} K _ {X X} ^ {- 1} y}{\partial x _ {2}} = \tilde {K} _ {x X} K _ {X X} ^ {- 1} y \\ \text {w h e r e} [ \tilde {K} _ {x X} ] _ {n} = \frac {\partial}{\partial x _ {2}} \exp \left(- \frac {\left(\left(x _ {1} - x _ {1} ^ {(n)}\right) ^ {2} + \left(x _ {2} - x _ {2} ^ {(n)}\right) ^ {2}\right)}{2 \theta^ {2}}\right) \\ = - \frac {\left(x _ {2} - x _ {2} ^ {(n)}\right)}{\theta^ {2}} \left[ K _ {x X} \right] _ {n} \\ \Longrightarrow \frac {\partial f (x)}{\partial x _ {2}} = - \left[ \sum_ {n = 1} ^ {N} \frac {\left(x _ {2} - x _ {2} ^ {(n)}\right)}{\theta^ {2}} \left[ K _ {x X} \right] _ {n} \right] K _ {X X} ^ {- 1} y \\ \end{array}
$$
We denote with  $\delta_{max}$  the maximum deviation of any input from the training points, i.e. we define  $\delta_{max}$  as  $\max_{\mathbf{x} \in \mathcal{X}} \min_{n \in [N]} |x_2 - x_2^{(n)}|$ . Also, we denote by  $f_{max}$  the maximum function value in the input domain, i.e.  $f_{max} \triangleq \max_{\mathbf{x} \in \mathcal{X}} f(\mathbf{x})$ . We can then bound the partial derivative wrt second dimension as follows.
$$
\frac {\partial f (\mathbf {x})}{\partial x _ {2}} \leq \frac {\delta_ {m a x} f (\mathbf {x})}{\theta^ {2}} \leq \frac {\delta_ {m a x} f _ {m a x}}{\theta^ {2}}
$$
For any arbitrary point  $\mathbf{x}$ , the maximum function deviation is upper bounded by the product of maximum slope and maximum distance from the closest point covered by the adversarial distance method.
$$
| f ([ x _ {1}, x _ {2} ] ^ {T}) - f ([ x _ {1}, \hat {x} _ {2} ] ^ {T}) | \leq \frac {\delta_ {m a x} f _ {m a x}}{\theta^ {2}} \max _ {\mathbf {x} \in \mathcal {X}} \min _ {\hat {\mathbf {x}} \in \hat {\mathcal {X}}} | x _ {2} - \hat {x} _ {2} | = \frac {\delta_ {m a x} f _ {m a x}}{\theta^ {2}} C
$$
$$
\left| f (\mathbf {x} + [ 0, \delta ] ^ {T}) - f (\mathbf {x}) \right| \leq 2 \frac {\delta_ {\max } f _ {\max }}{\theta^ {2}} C
$$
Marginalising  $\theta^{-2}$  with the Gamma prior leads to the final form below.
$$
\left| f (\mathbf {x} + \left[ 0, \delta \right] ^ {T}) - f (\mathbf {x}) \right| \leq 2 C \frac {\alpha}{\beta} \delta_ {\max } f _ {\max }
$$
We restate the result here for clarity.
Consider a regression task with  $D + 1$ -dimensional inputs  $\mathbf{x}$  where the first  $D$  dimensions are irrelevant, and assume they are  $x_{d} = y, d \in [1, D]$  while  $x_{D+1} \sim \mathcal{N}(y, 1 / K)$ . The MAP estimate of linear regression parameters  $f(\mathbf{x}) = \sum_{d=1}^{D+1} w_{d} x_{d}$  when fitted using Avg-Ex are as follows:  $w_{d} = 1 / (D + K), d \in [1, D]$  and  $w_{D+1} = K / (K + D)$ .
Proof. Without loss of generality, we assume  $\alpha, \sigma^2$  parameters of Avg-Ex are set to 1. In effect, our objective is to fit parameters that predict well for inputs sampled using standard normal perturbations, i.e.  $\mathbf{x}^{(n)} + \mathbf{m}\epsilon, \forall n \in [1, N]$ ,  $\epsilon \sim \mathcal{N}(0,1)$ ,  $\mathbf{m} = [1,1,\dots,1,0]^T \in \{0,1\}^{D+1}$ . The original problem therefore is equivalent to fitting on transformed input  $\hat{\mathbf{x}}$  such that  $\hat{\mathbf{x}}_i^{(n)} \sim \mathcal{N}(y, \sigma_i^2)$  where  $\sigma_i^2 = 1$  for all  $i \leq D$  and is  $1 / K$  when  $i = D + 1$ .
Likelihood of observations for the equivalent problem is obtained as follows.
$$
\begin{array}{l} P (y \mid \hat {x} _ {1}, \hat {x} 2, \dots , \hat {x} _ {D + 1}) = \prod_ {i = 1} ^ {D + 1} P (y \mid \hat {x} _ {i}) \propto \prod_ {i = 1} ^ {D + 1} P (\hat {x} _ {i} \mid y) P (y) \\ = \prod_ {i} \mathcal {N} (\hat {x} _ {i}; y, \sigma_ {i} ^ {2}) \propto \exp (- \sum_ {i} \frac {(y - \hat {x} _ {i}) ^ {2}}{2 \sigma_ {i} ^ {2}}) \\ = \exp \left\{- y ^ {2} \left(\sum_ {i} \frac {1}{2 \sigma_ {i} ^ {2}}\right) + y \left(\sum_ {i} \frac {\hat {x} _ {i}}{\sigma_ {i} ^ {2}}\right) + \sum_ {i} \frac {\hat {x} _ {i} ^ {2}}{2 \sigma_ {i} ^ {2}} \right\} \\ \propto \mathcal {N} (y; \sum_ {i} \frac {\hat {x} _ {i}}{\sigma_ {i} ^ {2}} P, P) \\ \end{array}
$$
$$
\text {w h e r e} P = \frac {1}{\sum_ {i} 1 / \sigma_ {i} ^ {2}}
$$
Substituting, the value of  $\sigma_{i}$  defined as above, we have  $\mathrm{P = D + K}$  and the MLE estimate for the linear regression parameters are as shown in the statement. The MAP estimate also remains the same since we do not impose any informative prior on the regression weights.
D Parametric Model Analysis
In this section we show that a similar result to what is shown for non-parametric models also holds for parametric models. We will analyse the results for a two-layer neural networks with ReLU activations. We consider a more general case of  $D$  dimensional input where the first  $d$  dimensions identify the spurious features. We wish to fit a function  $f:\mathbb{R}^{D}\to \mathbb{R}$  such that  $f(\mathbf{x})$  is robust to perturbations to the spurious features. We have the following bound when training a model using gradient regularization of Ross et al. (2017).
Proposition 2. We assume that the model is parameterised as a two-layer network with ReLU activations such that  $f(\mathbf{x}) = \sum_{j} \beta_{j} \phi \left( \sum_{i} w_{ji} x_{i} + b_{j} \right)$  where  $\vec{\beta} \in \mathbb{R}^{F}, \vec{w} \in \mathbb{R}^{F \times D}, \vec{b} \in \mathbb{R}^{F}$  are the parameters, and  $\phi(z) = \max(z, 0)$  is the ReLU activation. For any function such that gradients
wrt to the first  $d$  features is exactly zero, i.e.  $\left.\frac{\partial f}{\partial x_i}\right|_{\mathbf{x}_i^{(n)}} = 0 \forall i \in [1,d], n \in [1,N]$ , we have the following bound on the function value deviations for input perturbations from a training instance  $\mathbf{x}$ :  $\tilde{x} - x = \Delta \mathbf{x} = [\Delta \mathbf{x}_{1:d}^T, \mathbf{0}_{d+1:D}^T]^T$ .
$$
\left| f (\tilde {x}) - f (x) \right| = \Theta \left(\left(\| \vec {\beta} \| ^ {2} + \| \vec {w} \| _ {F} ^ {2}\right) \| \Delta \mathbf {x} \|\right) \tag {12}
$$
For a two-layer network trained to regularize gradients wrt first d dimensions on training data, the function value deviation from an arbitrary point  $\tilde{\mathbf{x}}$  from a training point  $\mathbf{x}$  such that  $\tilde{\mathbf{x}} -\mathbf{x} = \Delta \mathbf{x} = [\Delta \mathbf{x}_{1:d}^T,\mathbf{0}_{d + 1:D}^T ]^T$  is bounded as follows.
$$
\left| f (\tilde {x}) - f (x) \right| = \Theta \left(\left(\| \vec {\beta} \| ^ {2} + \| \vec {w} \| _ {F} ^ {2}\right) \| \Delta \mathbf {x} \|\right)
$$
Proof. Recall that the function is parameterised using parameters  $\vec{w},\vec{b},\vec{\beta}$  such that  $f(\mathbf{x}) = \sum_{j}\beta_{j}\phi (\sum_{i}w_{ji}x_{i} + b_{j})$  where  $\vec{\beta}\in \mathbb{R}^{F},\vec{w}\in \mathbb{R}^{F\times D},\vec{b}\in \mathbb{R}^{F}$  are the parameters, and  $\phi (z) = \max (z,0)$  is the ReLU activation.
Since we train such that  $\frac{\partial f(\mathbf{x})}{\partial x_i} = 0$ ,  $i \in [1, d]$ , we have that  $\frac{\partial f(\mathbf{x})}{x_i} = \sum_j \beta_j \hat{\phi} \left( \sum_i w_{ij} x_i + b_i \right) w_{ij}$  where  $\hat{\phi}(a) = \max \left( \frac{a}{|a|}, 0 \right)$ .
We now bound the variation in the function value for changes in the input when moving from  $\mathbf{x} \rightarrow \tilde{\mathbf{x}}$  where  $\mathbf{x}$  is an instance from the training data. We define four groups of neurons based on the sign of  $\sum_{i} w_{ji} x_i + b_j$  and  $\sum_{i} w_{ji} \tilde{x}_i + b_j$ .  $g_1$  is both positive,  $g_2$  is negative and positive,  $g_3$  is positive and negative,  $g_4$  is both negative. By defining groups, we can omit the ReLU activations as below.
$$
\begin{array}{l} f (\tilde {\mathbf {x}}) - f (\mathbf {x}) = \sum_ {j} \beta_ {j} \phi \left(\sum_ {i} w _ {j i} \tilde {x} _ {i} + b _ {j}\right) - \sum_ {j} \beta_ {j} \phi \left(\sum_ {i} w _ {j i} x _ {i} + b _ {j}\right) \\ = \sum_ {j \in g _ {1}} \beta_ {j} \sum_ {i} w _ {j i} (\tilde {x} _ {i} - x _ {i}) + \sum_ {j \in g _ {2}} \beta_ {j} \left(\sum_ {i} w _ {j i} \tilde {x} _ {i} + b _ {j}\right) - \sum_ {j \in g _ {3}} \beta_ {j} \left(\sum_ {i} w _ {j i} x _ {i} + b _ {j}\right) \\ = \sum_ {j \in g _ {1}} \beta_ {j} \sum_ {i = 1} ^ {d} w _ {j i} \left(\tilde {x} _ {i} - x _ {i}\right) + \sum_ {j \in g _ {2}} \beta_ {j} \left(\sum_ {i = 1} ^ {D} w _ {j i} \tilde {x} _ {i} + b _ {j}\right) - \sum_ {j \in g _ {3}} \beta_ {j} \left(\sum_ {i = 1} ^ {D} w _ {j i} x _ {i} + b _ {j}\right) \\ \end{array}
$$
Since we have that  $\sum_{j\in g_1\cup g_3}\beta_jw_{ij} = 0,\forall i\in [1,d]$  , we have
$$
\begin{array}{l} = \sum_ {j \in g _ {1}} \beta_ {j} \sum_ {i = 1} ^ {d} w _ {j i} \tilde {x} _ {i} + \sum_ {j \in g _ {2}} \beta_ {j} \left(\sum_ {i = 1} ^ {d} w _ {j i} \tilde {x} _ {i} + \sum_ {i = d + 1} ^ {D} w _ {j i} x _ {i} + b _ {j}\right) - \sum_ {j \in g _ {3}} \beta_ {j} \left(\sum_ {i = d + 1} ^ {D} w _ {j i} x _ {i} + b _ {j}\right) \\ - \underbrace {\sum_ {j \in g _ {1}} \beta_ {j} \sum_ {i = 1} ^ {d} w _ {j i} x _ {i} - \sum_ {j \in g _ {3}} \beta_ {j} \sum_ {i = 1} ^ {d} w _ {j i} x _ {i}} _ {= \sum_ {i = 1} ^ {d} x _ {i} \sum_ {j \in g _ {1} \cup g _ {3}} \beta_ {j} w _ {j i} = 0} \\ = \sum_ {j \in g _ {1} \cup g _ {2}} \beta_ {j} \sum_ {i = 1} ^ {d} w _ {j i} \tilde {x} _ {i} + \sum_ {j \in g _ {2}} \beta_ {j} \left(\sum_ {i = d + 1} ^ {D} w _ {j i} x _ {i} + b _ {j}\right) - \sum_ {j \in g _ {3}} \beta_ {j} \left(\sum_ {i = d + 1} ^ {D} w _ {j i} x _ {i} + b _ {j}\right) \\ \end{array}
$$
retaining only the terms that depend on  $\Delta x = \tilde{x} -x$ , the expression is further simplified as a term that grows with  $\Delta \mathbf{x}$  and a constant term that depends on the value of  $\mathbf{x}$