# Contribution to Error Analysis of Deep Neural Networks: Case of the Activation Functions

Maria L. Reyna Cruz, Kristalys Ruiz-Rohena, Yahriel I. Guel, Henry Salgado, Lisa Taldir, Elian Pena Ramos, Natalia Cervantes, Tzetzaith Rivero, Martine Ceberio, Christoph Lauter, et al.

► To cite this version:

Maria L. Reyna Cruz, Kristalys Ruiz-Rohena, Yahriel I. Guel, Henry Salgado, Lisa Taldir, et al. Contribution to Error Analysis of Deep Neural Networks: Case of the Activation Functions. 2025. [https://arxiv.org/abs/2505.05756](https://arxiv.org/abs/2505.05756)

HAL Id: hal-05367563

https://inria.hal.science/hal-05367563v1

Preprint submitted on 16 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Contribution to Error Analysis of Deep Neural Networks: Case of the Activation Functions

María L. Reyna Cruz*, Kristalys Ruiz-Rohena*, Yahriel I. Guel*, Henry Salgado*,
Lisa Taldir†, Elian Peña Ramos*, Natalia Cervantes*, Tzetzaith Rivero*,
Martine Ceberio*, Christoph Lauter*, Anastasia Volkova†

*University of Texas at El Paso, El Paso, TX, USA

†Université Perpignan Via Domitia, UPVD, LAMPS, France

†Inria, INSA Lyon, CITI, Villeurbanne, France

{mlreynacruz,kruizrohena,yiguel,hsalgado,epenaramos,ncervantes6,trivero2}@miners.utep.edu,

lisa.taldir@univ-perp.fr, mceberio@utep.edu, cqlauter@utep.edu, anastasia.volkova@inria.fr

**Abstract**—Deep neural networks are widely used in high-stakes applications such as medical imaging and autonomous perception, where numerical reliability is as critical as predictive accuracy. While prior work has focused on gradient stability, the propagation of small floating-point errors through nonlinear network components remains underexplored. This paper provides a quantitative analysis of error amplification in common activation functions, including ReLU, sigmoid, tanh, and softmax. For each function, we evaluate the perturbation response using two complementary approaches: absolute-error analysis, which captures the exact or bounded output change for a given additive input perturbation, and relative-error analysis, which measures the multiplicative amplification of errors relative to the nominal function value. By systematically deriving these bounds and expressions, we compare the numerical behavior across activations. These results enable a detailed description of the error propagation characteristics of each function, offering practical guidance for designing numerically stable and robust neural architectures in contexts where even minor computational errors can significantly affect downstream predictions.

## I. INTRODUCTION

Modern image analysis systems increasingly rely on deep neural networks for decision-making in high-stakes contexts such as medical imaging and autonomous perception. While progress in accuracy is increasing, the trustworthiness of these models depends not only on robustness to input perturbations but also on their numerical stability across computational steps.

In practice, each floating-point operation introduces a small numerical error due to finite precision. Although individually negligible, these local numerical deviations can accumulate through non-linear transformations, resulting in significant errors at the output [1]. Among the most critical nonlinear components are activation functions, which transform intermediate results and therefore modulate how numerical errors are propagated or amplified throughout the network.

This paper focuses on the quantitative analysis of error amplification caused by common activation functions such as hyperbolic tangent (tanh), sigmoid, exponential linear unit (ELU), rectified linear unit (ReLU) and softmax. Our motivation is to understand how theoretical error bounds at the activation level influence the trustworthiness of the overall model, particularly when applied to analysis pipelines where even minute errors may affect segmentation boundaries or classification logs.

By isolating and analyzing the delta error per single inference step, we aim to characterize how a small computational error

($\Delta x$) propagates through each activation function, producing a resulting error ($\Delta y$) that can accumulate over network layers.

This analysis provides the foundation for establishing activation specific stability metrics, guiding the selection of activation functions in trustworthy AI systems. While the linear components of neural computation can be analyzed separately, this work focuses on the non-linear amplification step, which often dominates the cumulative error.

## II. BACKGROUND

Existing research has primarily explored gradient stability but much less attention has been given to numerical propagation of forward inference errors. Prior studies in floating-point arithmetic, interval analysis, and robust training offer partial insights but do not provide analyses for common non-linear activations. This paper bridges this gap by formalizing how bounded numerical uncertainty propagates through specific non-linearities, with the aim of quantifying amplification factors.

Activation functions play a critical role in determining how information flows through a neural network. Among them, the Rectified Linear Unit (ReLU) became a popular default choice due to its computational efficiency and effectiveness in deep architectures. By setting negative inputs to zero, ReLU introduces sparsity and accelerates convergence during training. However, this hard zero cutoff leads to a gradient of zero for negative inputs, preventing weight updates and effectively “freezing” some neurons. This limitation motivates the development of smoother alternatives such as the Exponential Linear Unit (ELU), which retains the benefits of ReLU while addressing its inability to learn from negative activations. The Exponential Linear Unit (ELU) addresses the limitation of ReLU by allowing the negative values to push the mean activation of the unit closer to zero, allowing the gradient to approach the natural gradient of the unit as closely as possible. Therefore, the ELU has the purpose of maintaining information integrity by returning the same value if the given argument is positive, and returning a smooth close-to-zero negative value if the argument is less than zero. In other words, the ELU attains a noise robust state that reduces the variation in both the forward propagation and information. [2]–[4]

The Sigmoid function mathematical definition is the unique representation of an “s” shape for any mathematical function. This function is useful since we represent the exponential growth of the negative value of x, and its output range will always stay between
<table><caption>TABLE I: Absolute and relative error bounds for common activation functions.</caption><thead><tr><th>Activation</th><th>Absolute Error Bound</th><th>Relative Error Bound</th></tr></thead><tbody><tr><td>ReLU f(x) = max(0, x)</td><td>|ReLU(x + Δx) - ReLU(x)| ≤ |Δx|.<br>Exact piecewise characterization: if x and x + Δx are in the same region, Δy = Δx (if x > 0) or Δy = 0 (if x ≤ 0).</td><td>For any perturbation δ,<br>|ReLU(x + δ) - ReLU(x)| ≤ |δ|.<br>When the relative ratio is defined,<br>|Δy| / |ReLU(x)| ≤ |δ| / |x| (x > 0).<br>Undefined at x = 0.</td></tr><tr><td>Sigmoid σ(x)</td><td>|σ(x + Δx) - σ(x)| ≤ 1/4 |Δx|.<br>Sharper non-asymptotic estimate:<br>|σ(x + Δx) - σ(x)| ≤ σ(x)(1 - σ(x))(e^|Δx| - 1).<br>Second-order refinement:<br>σ(x + Δx) - σ(x) - σ'(x)Δx ≤ 1 / (12√3) |Δx|^2.</td><td>If Δx = xε and |ε| ≤ ε̄ ≤ 1/4, then<br>ε' = (1 - σ(x)) xε + O((xε)^2),<br>and<br>|ε'| ≤ e^|x|ε̄ - 1.</td></tr><tr><td>Tanh tanh(x)</td><td>|tanh(x + Δx) - tanh(x)| ≤ (1 - tanh^2 x)|Δx| + |tanh x|(1 - tanh^2 x)|Δx|^2 + 1/3(1 - tanh^2 x)|1 - 3tanh^2 x||Δx|^3 + 0.29535|Δx|^4.</td><td>If |ε| ≤ ε̄ ≤ 1/4, then there exists ε̄' ≤ 2.6262ε̄ such that<br>tanh(x(1 + ε̄)) = tanh(x)(1 + ε̄')<br>with |ε'| ≤ ε̄'.</td></tr><tr><td>Softmax s(x)<sub>i</sub></td><td>For δ ∈ ℝ<sup>+</sup>, δ ≤ 1/4 such that |δ<sub>i</sub>| ≤ δ for all i in a vector δ,<br>|s(x̄ + δ)<sub>i</sub> - s(x)<sub>i</sub>| ≤ 4.4445δ̄.</td><td></td></tr></tbody></table>

0 and 1. Taking us to explain why this backpropagation function is useful in minimizing the errors in neural networks outputs. The Sigmoid activation function is necessary in implementing it in neuro networks in order to get a more accurate result by minimizing the possibility of getting a wrong output between multilayered neuron weighted inputs. Since this function can be derived, we can re attack that output error by sending it back to its hidden layer (sigmoid function) until we get to a point where the error stays in a low and stable range of slopes. [5], [6]

The hyperbolic tangent function (tanh) is an activation function, often used in neural networks due to its properties like non-linearity and being zero centered with outputs in the range of [-1,1]. Non-linearity lets neural networks to learn more difficult, non-linear patterns without sacrificing gradient flow. Its nature around zero facilitates the convergence in optimization in contrast to the sigmoid function, and it is better than ReLU in extracting feature. Nonetheless, tanh, when applied to deep networks, presents the vanishing gradient problem. However, tanh is still valuable in cases where symmetric output and smoother activation dynamics are needed. [7], [8]

The softmax function is the extension of the sigmoid activation function, suitable for classification tasks involving more than two classes. It converts a vector of prediction score into probabilities, which indicate the likelihood of each class being the correct one. It is commonly used in Multilayer Perceptron (MLPs) and Convolutional Neural Network (CNNs).

### III. METHODOLOGY

For each activation function we evaluate the perturbation response in two complementary ways:

* **Absolute-error analysis:** We determine the exact or bounded change in the function output, Δy, resulting from a small additive perturbation Δx in the input. This characterization captures worst-case deviations regardless of the input magnitude.

* **Relative-error analysis:** For nonzero inputs, we measure the perturbation amplification in a multiplicative sense, computing the ratio between the output change and the nominal function value. This provides insight into how the function scales relative input noise, highlighting potential numerical instabilities or sensitivity to input magnitude.

By systematically computing these bounds and expressions, we can compare the error behavior across activation functions.
Table II summarizes the obtained error bounds.

### IV. TANH-RELATIVE ERROR

**Lemma 1.** Let $f(x) = \frac{x}{\sinh(x)}$, We have $f(0) = 1$, $0 < f(x) \le 1$ for all $x \in \mathbb{R}$ and $f$ is continuous everywhere.

*Proof.* Remark that $f(-x) = \frac{-x}{\sinh(-x)} = \frac{-x}{-\sinh(x)} = f(x)$. It hence suffices to consider values $x \le 0$. Remark that using L'Hopital's rule we obtain,

$$f(0) = \lim_{x \to 0} \frac{x}{\sinh x} = \lim_{x \to 0} \frac{1}{\cosh x} = 1$$
As sinh has only a zero at zero and as $x \to x$ and sinh are continuous, $f$ is hence continuous. As $x \to x$ has only a zero at zero, $f$ has no zero. As it is continuous and $f(0) = 1 > 0$ this means that $f(x) > 0$ everywhere.

Now consider $f'(x)$;

$$ 
\begin{aligned}
f'(x) &= \frac{\sinh(x) - x \cdot \cosh(x)}{\sinh^2(x)} \\
&= \frac{1}{\sinh(x)} - \frac{x}{\sinh(x)} \cdot \frac{1}{\tanh(x)} \\
&= \frac{x}{\sinh(x)} \cdot \left( \frac{1}{x} - \frac{1}{\tanh(x)} \right) \\
&= \frac{x}{\sinh(x)} \cdot \frac{\tanh(x) - x}{x \cdot \tanh(x)}
\end{aligned}
$$

Where,

$$ \frac{x}{\sinh(x)} > 0 $$

We have, again with L'Hopital's rule, at order 2 this time.

$$ 
\begin{aligned}
f'(0) &= \lim_{x \to 0} f'(x) = \lim_{x \to 0} \frac{\sinh(x) - x \cdot \cosh(x)}{\sinh^2(x)} \\
&= \lim_{x \to 0} \frac{-\sinh(x) - x \cdot \cosh(x)}{2 \cdot \sinh^2(x) + 2 \cdot \cosh^2(x)} = 0
\end{aligned}
$$

As $x \to \frac{x}{\sinh(x)}$ is continuous everywhere and $x \to \tanh(x) - x$ is continuous everywhere and $x \to x \cdot \tanh(x)$ has only a zero at zero as $x \to x$ and $x \to \tanh(x)$ have only a zero at zero. But $x \to x \cdot \tanh(x)$ is continuous everywhere and $f'$ is continuous at zero, $f'$ is continuous everywhere.

Now consider

$$ g(x) = \tanh(x) - x $$

We have

$$ g'(x) = -\tanh^2(x) \le 0 $$

$g$ is continuous everywhere and as $g'(x) \le 0$ for all $x \in \mathbb{R}$ it is decreasing everywhere. It has a zero at zero, as $g(0)=0$ but as it is continuous and decreasing it has no other zero. We have

$$ 
\begin{aligned}
f'(x) &= \frac{x}{\sinh(x)} \cdot \frac{g(x)}{x \cdot \tanh(x)} \\
\text{with } f'(0) &= 0 \text{ and } \frac{x}{\sinh(x)} > 0
\end{aligned}
$$

This means the only zero of $g$ at zero is the only zero of $f'$. This means $f$ has no local extremum other than at 0. At zero, $f(0) = 1$.

As $f$ is continuous and $f(x) > 0$ everywhere, this means $0 < f(x) \le 1$ for all $x$,

**Lemma 2.** Let $g(x) = \cosh(x) - \frac{x}{\sinh(x)}$. Then $g$ has only a zero at zero.

*Proof.* We know that $\cosh \ge 1$ for all $x \in \mathbb{R}$. With Lemma 1, we proved that $f(x) = \frac{x}{\sinh(x)} \le 1$ for all $x \in \mathbb{R}$.

Hence $g$ can only have zero when $\cosh(x^*) = 1$, which occurs for $x^* = 0$, where $f(x^*) = 1$. This proves that $g$ has only one zero at zero, as $x \to \cosh(x)$ and $f$ are continuous everywhere and bounded as shown.

**Lemma 3.** Let $h \cdot \mathbb{R} \times \mathbb{R} \to \mathbb{R}$ be defined by $h(x, \varepsilon) = \frac{\tanh(x \cdot \varepsilon)}{\varepsilon \cdot \tanh(x)}$. For all $(x, \varepsilon) \in \mathbb{R}^2$ where $h$ has a local extremum $x \cdot \varepsilon = 0$.

*Proof.* We have,

$$ 
\begin{aligned}
& \frac{\delta}{\delta \varepsilon} h(x, \varepsilon) \\
= & \frac{x \cdot \varepsilon \cdot (\tanh(x) - \tanh(x) \cdot \tanh^2(x \cdot \varepsilon)) - \tanh(x \cdot \varepsilon) \cdot \tanh(x)}{\varepsilon^2 \cdot \tanh^2(x)}
\end{aligned}
$$

Suppose that $x \ne 0$ and $\varepsilon \ne 0$. Therefore $x \cdot \varepsilon \ne 0$. If $h$ has a local extremum nevertheless (with $x \cdot \varepsilon \ne 0$), we have $\frac{\delta}{\delta \varepsilon} h(x, \varepsilon) = 0$.

Hence

$$ 
\begin{aligned}
& x^* \cdot \varepsilon^* \cdot (\tanh(x^*) - \tanh(x) \cdot \tanh^2(x^* \cdot \varepsilon^*)) - \tanh(x^* \cdot \varepsilon^*) \cdot \tanh(x^*) = 0
\end{aligned}
$$

Hence

$$ 
\begin{aligned}
x^* \cdot \varepsilon^* &= \frac{\tanh(x^* \cdot \varepsilon^*) \cdot \tanh(x^*)}{\tanh(x^*) - \tanh(x^* \cdot \varepsilon^*) \cdot \tanh(x^*)} \\
&= \frac{\tanh(x^* \cdot \varepsilon^*)}{1 - \tanh(x^* \cdot \varepsilon^*)}
\end{aligned}
$$

Let $a = x^* \cdot \varepsilon^*$, therefore

$$ a = \frac{\tanh(a)}{1 - \tanh^2(a)} $$

It is well-known that $\cosh^2(a) = \frac{1}{1-\tanh^2(a)}$. Hence,

$$ a = \frac{\sinh(a)}{\cosh(a)} \cdot \cosh^2(a) $$

which is equivalent to,

$$ \cosh(a) - \frac{a}{\sinh(a)} = 0 $$

However, Lemma 2 shows that $g$ defined by $g(a) = \cosh(a) - \frac{a}{\sinh(a)}$ has only one zero at zero, so for $x^* \cdot \varepsilon^* = a = 0$, which is a contraction. $\square$

**Lemma 4.** Let $h : \mathbb{R} \times \mathbb{R} \to \mathbb{R}$ still be defined by $h(x, \varepsilon) = \frac{\tanh(x \cdot \varepsilon)}{\varepsilon \cdot \tanh(x)}$. Let $x$ be bounded by $|x| \le 1$ and $\varepsilon$ be bounded by $|x| \le \frac{1}{4}$. Then $|h(x, \varepsilon)|$ is bounded by $|h(x, \varepsilon)| \le 1287$

*Proof.* As per Lemma 2, it suffice to consider the value of $h(x, \varepsilon)$ on the bounds $(-1, 1$ for $x, -\frac{1}{4}, \frac{1}{4}$ for $\varepsilon$) and to consider

$$ \lim_{(x, \varepsilon) \to (0,0)} h(x, \varepsilon) \text{ as } h \text{ has no other local extrema} $$

Let us first consider

$$ 
\begin{aligned}
\lim_{(x, \varepsilon) \to (0,0)} h(x, \varepsilon) &= \lim_{(x, \varepsilon) \to (0,0)} \frac{\tanh}{\varepsilon \cdot \tanh(x)} \\
&= \lim_{(x, \varepsilon) \to (0,0)} \frac{x \cdot \varepsilon - \frac{x^3 \cdot \varepsilon^3}{3} + \frac{2x^5 \cdot \varepsilon^5}{15} - \dots}{x \cdot \varepsilon - \frac{x^3 \cdot \varepsilon}{3} + \frac{2x^5 \cdot \varepsilon}{15} - \dots} \\
&= \lim_{(x, \varepsilon) \to (0,0)} \frac{1 - \frac{x^2 \cdot \varepsilon^2}{3} + \frac{2x^4 \cdot \varepsilon^4}{15} - \dots}{1 - \frac{x^2}{3} + \frac{2x^4}{15} - \dots}
\end{aligned}
$$

As $h(x, \varepsilon)$ is symmetrical around $(0,0)$ for both $x$ and $\varepsilon$, it suffices to consider an additional value.
$$h(1, \frac{1}{4}) = \frac{\tanh(1 \cdot \frac{1}{4})}{\frac{1}{4} \cdot \tanh(1)} = 1.28634732 \le 1.287$$

<figure><img src="image_2.png" /></figure>

**Lemma 5.** Let $x \in \mathbb{R}$ be bounded by $|x| \le 1$ and $\varepsilon \in \mathbb{R}$ be bounded $\varepsilon \le \bar{\varepsilon}$ with $\bar{\varepsilon} \in \mathbb{R}^+$, $\bar{\varepsilon} \le \frac{1}{4}$. Then there exist $\bar{\varepsilon}^1 \in \mathbb{R}^+$, $\bar{\varepsilon}^1 \le 1.898 \cdot \bar{\varepsilon}$ such that for all $x$, $\varepsilon$ there exist $\varepsilon'$ such that

$$\tanh(x \cdot (1 + \varepsilon)) = \tanh(x) \cdot (1 + \varepsilon')$$
with, $|\varepsilon'| \le \bar{\varepsilon}'$$

*Proof.* We can assume $x \ge 0$ as tanh is symmetrical and $|\varepsilon| < 1$. We have,

$$\begin{aligned} \tanh(x \cdot (1 + \varepsilon)) &= \frac{\tanh(x \cdot \varepsilon) \cdot \tanh(x)}{1 + \tanh(x) \cdot \tanh(x \cdot \varepsilon)} \\ &= \frac{\tanh(x)}{\tanh(x)} \cdot \frac{\tanh(x \cdot \varepsilon) \cdot \tanh(x)}{1 + \tanh(x) \cdot \tanh(x \cdot \varepsilon)} \\ &= \tanh(x) \cdot \left( 1 + \frac{\tanh(x \cdot \varepsilon) \cdot \tanh(x)}{\tanh(x) \cdot (1 + \tanh(x) \cdot \tanh(x \cdot \varepsilon))} - 1 \right) \\ &= \tanh \cdot (1 + \varepsilon') \end{aligned}$$

with,

$$\varepsilon' = \frac{\tanh(x \cdot \varepsilon) \cdot \tanh(x)}{\tanh(x) \cdot (1 + \tanh(x) \cdot \tanh(x \cdot \varepsilon))} - 1$$

Hence

$$\begin{aligned} \varepsilon' &= \frac{\tanh(x) + \tanh(x\varepsilon) - \tanh(x) \cdot (1 + \tanh(x) \cdot \tanh(x\varepsilon))}{\tanh(x) \cdot (1 + \tanh(x) \cdot \tanh(x\varepsilon))} \\ &= \frac{\tanh(x\varepsilon) - \tanh^2(x) \cdot (\tanh(x\varepsilon))}{\tanh(x) \cdot (1 + \tanh(x) \cdot \tanh(x\varepsilon))} \\ &= \varepsilon \cdot \frac{\tanh(x\varepsilon)}{\varepsilon \cdot \tanh(x)} \cdot \frac{1 - \tanh(x)}{1 + \tanh(x) \cdot \tanh(x\varepsilon)} \\ &= \varepsilon \cdot \frac{\tanh(x\varepsilon)}{\varepsilon \cdot \tanh(x)} \cdot \frac{1 - \tanh(x)}{1 + \tanh^2(x) \cdot \varepsilon \cdot \frac{\tanh(x\varepsilon)}{\varepsilon \cdot \tanh(x)}} \\ &= \varepsilon \cdot \alpha \end{aligned}$$

with

$$\alpha = \frac{\tanh(x\varepsilon)}{\varepsilon \cdot \tanh(x)} \cdot \frac{1 - \tanh(x)}{1 + \tanh^2(x) \cdot \varepsilon \cdot \frac{\tanh(x\varepsilon)}{\varepsilon \cdot \tanh(x)}}$$

where,

$\tanh(x) \ge 0$ for $x \ge 0$, 
Since is a subtraction then, $1 - \tanh(x) \le 1$
$\varepsilon \cdot \tanh(x) \le 1.287$ by Lemma 4

$$\begin{aligned} \tanh^2(x) &\le 1 \\ \varepsilon &\le \frac{1}{4} \end{aligned}$$

$$\text{Therefore, } 1 + \tanh^2(x) \cdot \varepsilon \cdot \frac{\tanh(x\varepsilon)}{\varepsilon \cdot \tanh(x)} \ge 1 - \frac{1}{4} \cdot 1.287$$

Then,

Therefore

$$|\alpha| \le 1.287 \cdot \frac{1}{1 - \frac{1}{4} \cdot 1.287} \le 1.898$$

<figure><img src="image_3.png" /></figure>

**Lemma 6.** Consider the family of functions

$$\begin{aligned} g_a(\mu) &= \mu \cdot a^\mu \cdot \ln(a) - (a^\mu - 1) \\ \text{for all } a \in [1, 1.563] \end{aligned}$$

Any of the $g_a$ has no other zero than at zero.

*Proof.*

$$\begin{aligned} g_a(\mu) &= \mu \cdot a^\mu \cdot \ln(a) - a^\mu + 1 \\ &= a^\mu \cdot (\mu \cdot \ln(a) - 1) + 1 \\ g_a(0) &= a^0 \cdot (0 \cdot \ln(a) - 1) + 1 \\ &= 1 \cdot (-1) + 1 \end{aligned}$$

This means $g_a$ has a zero at zero. Now consider.

$$\begin{aligned} g'_{a}(\mu) &= \mu \cdot a^{\mu} \cdot \ln^{2}(a) + a^{\mu} \cdot \ln(a) - a^{\mu} \cdot \ln(a) \\ &= \mu \cdot a^{\mu} \cdot \ln^{2}(a) \end{aligned}$$

$\mu^* = 0$ is the only zero of $g'_a$ as $\mu \cdot a^\mu \cdot \ln^2(a) = 0$ implies $\mu^* = 0$, since $a^\mu \cdot \ln^2(a) > 0$.

This means $g_a$ has its only local extremum at zero, where its value $g_a(0) = 0$. This means it cannot have any others zero as $g_a$ is continuous. This means $g_a$ has no other zero than at zero.

<figure><img src="image_4.png" /></figure>

**Lemma 7.** Consider the family of functions

$$f_a(\mu) = \frac{a^\mu - 1}{\mu}$$

For all $a \in [1, 1.563]$ and for $|\mu| \le 1$. Then $|f_a(\mu)|$ is bounded by $|f_a(\mu)| \le 0.563$

*Proof.* Consider

$$f'_{a}(\mu) = \frac{\mu \cdot a^{\mu} \cdot \ln(a) - (a^{\mu} - 1)}{\mu^{2}} = \frac{g_{a}(\mu)}{\mu^{2}}$$

with $g_a$ defined as in Lemma 6. We have,

$$\lim_{\mu \to 0} f'_{a}(\mu) = \lim_{\mu \to 0} \frac{\mu \cdot a^{\mu} \cdot \ln(a) - (a^{\mu} - 1)}{\mu^{2}}$$

By L'Hopital's Rule

$$= \lim_{\mu \to 0} \frac{\mu \cdot a^{\mu} \cdot \ln^2(a)}{2\mu}$$

By L'Hopital's Rule

$$\begin{aligned} &= \lim_{\mu \to 0} \frac{\ln^2(a) \cdot (\mu \cdot a^{\mu} \cdot \ln^2(a) + a^{\mu})}{2} \\ &= \lim_{\mu \to 0} \frac{\ln^2(a) \cdot (\mu \cdot \ln(a) + 1)}{2} \\ &= \frac{\ln^2(a)}{2} \ge 0 \end{aligned}$$

As per Lemma 6, $g_a$ has no other zero than at zero. This means $f'_a$ has no zero for $|\mu| \le 1$. This means $f_a$ is monotonous decreasing or monotonous increasing. It suffices to consider $\mu = -1$ and $\mu = 1$ to bound it. We have,
$$
\begin{align*}
f_a(1) &= \frac{a-1}{1} = a-1 \le 0.563 \text{ as } a \in [1, 1.563] \\
f_a(-1) &= \frac{a^{-1}-1}{-1} = \frac{1-\frac{1}{a}}{1} \\
&= 1 - \frac{1}{a} \le 1 - \frac{1}{1.563} \le 0.361
\end{align*}
$$

where

$$
\begin{align*}
& a \le 1.563 \\
& \frac{1}{a} \ge \frac{1}{1.563} \\
& 1 - \frac{1}{a} \le -\frac{1}{1.563}
\end{align*}
$$

This means that $|f_a(\mu)| \le 0.563$ for all $\mu$ s.t. $|\mu| \le 1$

<figure><img src="image_5.png" /></figure>

**Lemma 8.** Consider the family of functions

$$
\alpha_x(\varepsilon) = \frac{e^{2x\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}}
$$

for all $x \ge 1$ and $|\varepsilon| \le e^{-\frac{3}{2}x}$. Then $|\alpha_x(\varepsilon)| \le 0.563$ holds.

Proof. Consider the auxiliary function $t(x) = 2x \cdot e^{-\frac{3}{2}x} > 0$ as $x \ge 1 > 0$. We have $t'(x) = e^{-\frac{3}{2}x} \cdot (2-3x) < 0 \quad \forall x \ge 1$. This means $t$ is monotonously decreasing while $x$ increases. $t$ is therefore maximum for $x = 1$. This means $0 < t(x) \le t(1) = 2 \cdot 1 \cdot e^{-\frac{3}{2}} < 0.447$.

Now let $a = e^t$. This value $a$ is bounded by $a \in [1, e^{t(1)}]$ hence $a \in [1, 1.563]$.

Let $\mu = \varepsilon \cdot e^{\frac{3}{2}x}$. Therefore $\varepsilon = e^{-\frac{3}{2}x}\mu$. We know that $|\varepsilon| \le e^{-\frac{3}{2}x}$ implies $|\mu| \le 1$. We have,

$$
\alpha = \frac{e^{2x\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}}
$$

Which using the above, becomes.

$$
\alpha = \frac{e^{2x e^{-\frac{3}{2}x}} \mu - 1}{\mu} = \frac{e^{t\mu} - 1}{\mu} = \frac{(e^t)^\mu - 1}{\mu} = \frac{a^\mu - 1}{\mu}
$$

As per Lemma 7, it follows that, $|\alpha_x(\varepsilon)| \le 0.563$ holds for all $x \ge 1$ and $|\varepsilon| \le e^{-\frac{3}{2}x}$

<figure><img src="image_6.png" /></figure>

**Lemma 9.** Consider the family of functions

$$
\alpha_x(\varepsilon) = \frac{e^{2x\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}}
$$

for all $x \ge 1$ and $-\frac{1}{4} \le \varepsilon < -e^{-\frac{3}{2}x}$ Then. $|\alpha_x(\varepsilon)| \le 2$ holds.

Proof. We have,

$\varepsilon < -e^{\frac{3}{2}x}$

Therefore

$$
-e > e^{-\frac{3}{2}x} \quad \text{and} \quad -\frac{1}{\epsilon} < e^{\frac{3}{2}x}
$$

This means that

$$
\frac{1}{-\varepsilon} \cdot e^{-\frac{3}{2}x} < e^{\frac{3}{2}x} \cdot e^{-\frac{3}{2}x} = 1
$$

Similarly

$$
\begin{gather*}
\frac{1}{-\varepsilon} \cdot e^{\beta \cdot x} < 1 \\
\text{for all } \beta \le -\frac{3}{2}, \text{ as } x \ge 1 > 0.
\end{gather*}
$$

Therefore,

$$
\begin{align*}
|\alpha_x(\varepsilon)| &= \left| \frac{1 - e^{2x\varepsilon}}{-\varepsilon \cdot e^{\frac{3}{2}x}} \right| \\
&= \left| \frac{1}{-\varepsilon} \cdot e^{-\frac{3}{2}x} - \frac{1}{-\varepsilon} \cdot e^{2x\varepsilon - \frac{3}{2}x} \right| \\
&\le \left| \frac{1}{-\varepsilon} \cdot e^{-\frac{3}{2}x} \right| + \left| \frac{1}{-\varepsilon} \cdot e^{2x\varepsilon - \frac{3}{2}x} \right| \\
&= \frac{1}{-\varepsilon} \cdot e^{-\frac{3}{2}x} + \frac{1}{-\varepsilon} \cdot e^{2x\varepsilon} \le 1 + 1 = 2
\end{align*}
$$

<figure><img src="image_7.png" /></figure>

**Lemma 10.** Consider the family of functions

$$
\begin{align*}
\gamma_x(\varepsilon) &= e^{-\frac{1}{2}} \cdot \alpha_x(\varepsilon) \\
\alpha_x(\varepsilon) &= \frac{e^{2\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}}
\end{align*}
$$

for all $x \ge 1$ and $e^{-\frac{3}{2}x} \le \varepsilon < \frac{1}{4}$ Then. $|\gamma_x(\varepsilon)| \le 2$ holds.

Proof. We have $\varepsilon > e^{-\frac{3}{2}x}$ and therefore $\frac{1}{\varepsilon} < e^{\frac{3}{2}x}$. This means $\frac{1}{\varepsilon} \cdot e^{-\frac{3}{2}x} < 1$ and $e^{-\frac{1}{2}x} \cdot \frac{1}{\varepsilon} \cdot e^{-\frac{3}{2}x} < \frac{1}{1.65} \cdot 1 < 1$. Similarly $e^{-\frac{1}{2}x} \cdot \frac{1}{\varepsilon} \cdot e^{\beta x} < 1$ for all $\beta \le -1$.

Therefore

$$
\begin{align*}
|\gamma_x(\varepsilon)| &= \left| e^{\left|1\frac{1}{2}x\right|} \cdot \frac{e^{2x\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}} \right| \\
&= e^{-\frac{1}{2}x} \cdot \left| \frac{1}{\varepsilon} \cdot e^{2x\varepsilon - \frac{3}{2}x} - \frac{1}{\varepsilon} \cdot e^{-\frac{3}{2}x} \right| \\
&\le e^{-\frac{1}{2}x} \cdot \left( \frac{1}{\varepsilon} \cdot e^{2x\varepsilon - \frac{3}{2}x} + \frac{1}{\varepsilon} \cdot e^{-\frac{3}{2}x} \right) \\
&= e^{-\frac{1}{2}x} \cdot \frac{1}{\varepsilon} \cdot e^{x \cdot (2\varepsilon - \frac{3}{2})} + e^{-\frac{1}{2}x} \cdot \frac{1}{\varepsilon} \cdot e^{-\frac{3}{2}x} \le 2
\end{align*}
$$

Where,

$$
\begin{align*}
x \cdot (2\varepsilon - \frac{3}{2}) &= x \cdot \beta \\
e^{-\frac{1}{2}x} \cdot \frac{1}{\varepsilon} \cdot e^{x \cdot \beta} &\le 1 \\
e^{-\frac{1}{2}x} \cdot \frac{1}{\varepsilon} \cdot e^{2x\varepsilon - \frac{3}{2}x} &\le 1
\end{align*}
$$

As $\beta = 2\varepsilon - \frac{3}{2}$ is bounded by $-\frac{3}{2} \le \beta \le -1$ since $0 \le \varepsilon \le \frac{1}{4}$ and $0 \le 2 \cdot \varepsilon \le \frac{1}{2}$, then $-\frac{3}{2} \le 2 \cdot \varepsilon - \frac{3}{2} \le -1$

<figure><img src="image_8.png" /></figure>

**Lemma 11.** Let $x \in \mathbb{R}$ be such that $|x| > 1$ and $\varepsilon \in \mathbb{R}$ be bounded by $\bar{\varepsilon} \in \mathbb{R}^+$, $\bar{\varepsilon} \le \frac{1}{4}$. Then if $-\frac{1}{4} \le \varepsilon \le e^{-\frac{3}{2}|x|}$ holds, then there exists $\bar{\varepsilon}' \le 2.6262 \cdot \bar{\varepsilon}$ such that for all $x, \varepsilon$ as specified there exists $\varepsilon'$ such that

$$
\tanh(x \cdot (1 + \varepsilon)) = \tanh(x) \cdot (1 + \varepsilon')
$$
$$
\text{with } |\varepsilon'| \le \varepsilon'
$$
**Proof.** We can assume $x > 1$ as tanh is symmetrical and $|\varepsilon| < 1$. Therefore We have,

$$
\begin{align*}
\tanh(x \cdot (1 + \varepsilon)) &= 1 - \frac{2}{e^{2x \cdot (1+\varepsilon)} + 1} \\
&= \left(1 - \frac{2}{e^{2x} + 1}\right) \cdot \left(\frac{1 - \frac{2}{e^{2x \cdot (1+\varepsilon)} + 1}}{1 - \frac{2}{e^{2x} + 1}}\right) \\
&= \tanh(x) \cdot \left(1 + \frac{1 - \frac{2}{e^{2x \cdot (1+\varepsilon)} + 1}}{1 - \frac{2}{e^{2x} + 1}} - 1\right) \\
&= \tanh(x) \cdot (1 + \varepsilon')
\end{align*}
$$

with

$$
\begin{align*}
\varepsilon' &= \frac{1 - \frac{2}{e^{2x \cdot (1+\varepsilon)} + 1}}{1 - \frac{2}{e^{2x} + 1}} - 1 \\
&= \frac{1 - \frac{2}{e^{2x \cdot (1+\varepsilon)} + 1} - \left(1 - \frac{2}{e^{2x} + 1}\right)}{1 - \frac{2}{e^{2x} + 1}} \\
&= \frac{1 - \frac{2}{e^{2x \cdot (1+\varepsilon)} + 1} - 1 + \frac{2}{e^{2x} + 1}}{1 - \frac{2}{e^{2x} + 1}} \\
&= \frac{\frac{2e^{2x}(1+\varepsilon)}{(e^{2x}+1) \cdot (e^{2x \cdot (1+\varepsilon)}+1)} + 2 - 2e^{2x} - \frac{2}{(e^{2x}+1) \cdot (e^{2x \cdot (1+\varepsilon)}+1)}}{1 - \frac{2}{e^{2x} + 1}} \\
&= \varepsilon \cdot \frac{2 \cdot e^{2x} \cdot (e^{2x\varepsilon} - 1)}{\varepsilon \cdot \left(1 - \frac{2}{e^{2x} + 1}\right) e^{2x} \cdot \left(1 + \frac{1}{e^{2x}}\right) \cdot \left(e^{\frac{3}{2}x}\right) \cdot \left(e^{2x \cdot (1+\varepsilon-\frac{3}{2}x)} + \frac{1}{e^{\frac{3}{2}x}}\right)} \\
&= \varepsilon \cdot \frac{2}{\left(1 - \frac{2}{e^{2x} + 1}\right) \cdot \left(1 + \frac{1}{e^{2x}}\right)} \cdot \frac{1}{e^{2x \cdot (1+\varepsilon)-2x \cdot \frac{3}{4}} + \frac{1}{e^{\frac{3}{2}x}}} \cdot \frac{e^{2x\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}}
\end{align*}
\[
\begin{gathered}
\varepsilon' \le |\varepsilon| \cdot 1.3131 \cdot 1 \cdot 2 = |\varepsilon| \cdot 2.6262 \quad \Box \\
\text{Lemma 12. Let } x \in \mathbb{R} \text{ be such that } |x| > 1 \text{ and } \varepsilon \in \mathbb{R} \text{ be bounded by } |\varepsilon| \le \bar{\varepsilon} \text{ with } \bar{\varepsilon} \in \mathbb{R}^+, \bar{\varepsilon} \le \frac{1}{4}. \text{ Then if } e^{-\frac{3}{2}|x|} < \varepsilon \le \frac{1}{4} \text{ holds then there exist } \bar{\varepsilon}' \in \mathbb{R}^+, \varepsilon' \le 2.6262 \cdot \bar{\varepsilon} \text{ such that for all } x, \varepsilon \text{ as specified there exists } \varepsilon' \text{ such that } \tanh(x \cdot (1 + \varepsilon)) = \tanh(x) \cdot (1 + \varepsilon') \text{ with } |\varepsilon'| \le \varepsilon'
\end{gathered}
\]
$$

Now observe that, $2x \cdot (1 + \varepsilon) - 2x \cdot \frac{3}{4} = 2x \cdot (1 + \varepsilon - \frac{3}{4})$
where, $-\frac{1}{4} \le 1 + \varepsilon \le \frac{5}{4}$ so $0 \le 1 + \varepsilon - \frac{3}{4} \le \frac{1}{2}$ with $-\frac{1}{4} \le \varepsilon \le \frac{1}{4}$
is bounded by $0 \le 2x \cdot (1 + \varepsilon) - 2x \cdot \frac{3}{4} \le x$ which means that
$e^{2x \cdot (1+\varepsilon)-2x \cdot \frac{3}{4}} \ge 1$. As $\frac{1}{e^{\frac{3}{2}x}} > 0$ we have $e^{2x \cdot (1+\varepsilon)-2x \cdot \frac{3}{4}} + \frac{1}{e^{\frac{3}{2}x}} >$
$1$ and therefore $\frac{1}{e^{2x \cdot (1+\varepsilon)-2x \cdot \frac{3}{4}} + \frac{1}{e^{\frac{3}{2}x}}} < 1$

Therefore

$$
\left(1 - \frac{2}{e^{2x} + 1}\right) \cdot \left(1 + \frac{1}{e^{2x}}\right) > 1 - \frac{2}{e^2 + 1}
$$

Therefore

$$
\frac{2}{\left(1 - \frac{2}{e^{2x} + 1}\right) \cdot \left(1 + \frac{1}{e^{2x}}\right)} < \frac{1}{1 - \frac{2}{e^{2x} + 1}} < 1.3131
$$

As per Lemmas 8 and 9 as $\varepsilon$ satisfies $-\frac{1}{4} < \varepsilon < e^{-\frac{3}{2}x}$

$$
\left| \frac{e^{2x\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}} \right| \le 2
$$

Therefore

$$
|\varepsilon'| \le |\varepsilon| \cdot 1.3131 \cdot 1 \cdot 2 = |\varepsilon| \cdot 2.6262
$$

**Lemma 12.** Let $x \in \mathbb{R}$ be such that $|x| > 1$ and $\varepsilon \in \mathbb{R}$ be bounded by $|\varepsilon| \le \bar{\varepsilon}$ with $\bar{\varepsilon} \in \mathbb{R}^+$, $\bar{\varepsilon} \le \frac{1}{4}$. Then if $e^{-\frac{3}{2}|x|} < \varepsilon \le \frac{1}{4}$ holds then there exist $\bar{\varepsilon}' \in \mathbb{R}^+$, $\varepsilon' \le 2.6262 \cdot \bar{\varepsilon}$ such that for all $x, \varepsilon$ as specified there exists $\varepsilon'$ such that $\tanh(x \cdot (1 + \varepsilon)) = \tanh(x) \cdot (1 + \varepsilon')$ with $|\varepsilon'| \le \varepsilon'$

Proof. We can assume $x > 1$ as tanh is symmetrical and $|\varepsilon| < 1$.
We still have (as in the proof for Lemma 11)
$$ \frac{2 \cdot e^{2x} \cdot (e^{2x\varepsilon} - 1)}{\varepsilon \cdot \left(1 - \frac{2}{e^{2x} + 1}\right) e^{2x} \cdot \left(1 + \frac{1}{e^{2x}}\right) \cdot \left(e^{\frac{3}{2}x}\right) \cdot \left(e^{2x \cdot \left(1+\varepsilon-\frac{3}{2}x\right)} + \frac{1}{e^{\frac{3}{2}x}}\right)} $$

$$
\tanh(x \cdot (1 + \varepsilon)) = \tanh(x) \cdot (1 + \varepsilon')
$$

with

$$
\begin{align*}
\varepsilon' = \varepsilon \cdot \frac{2}{\left(1 - \frac{2}{e^{2x} + 1}\right) \cdot \left(1 + \frac{1}{e^{2x}}\right)} \cdot \frac{1}{e^{2x \cdot (1+\varepsilon)-2x \cdot \frac{3}{4}} + \frac{1}{e^{\frac{3}{2}x}}} \cdot \\
\qquad \cdot e^{\frac{1}{2}x} \cdot e^{-\frac{1}{2}x} \cdot \frac{e^{2x\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}}
\end{align*}
$$

We still have

$$
\frac{2}{\left(1 - \frac{2}{e^{2x} + 1}\right) \cdot \left(1 + \frac{1}{e^{2x}}\right)} < 1.3131 \text{ as in the proof for Lemma 11}
$$

We further have,

$$
\frac{1}{e^{2x \cdot (1+\varepsilon)-2x \cdot \frac{3}{4}} + \frac{1}{e^{\frac{3}{2}x}}} \cdot e^{\frac{1}{2}x} = \frac{1}{e^{2x \cdot (1+\varepsilon)-2x \cdot \frac{3}{4} - \frac{1}{2}x} + \frac{1}{e^{\frac{3}{2}x}}}
$$

Further

$x > 1$

Hence

$$
\begin{array}{l}
2x > 2 \\
e^{2x} > e^2 \\
e^{2x} + 1 > e^2 + 1
\end{array}
$$

Therefore

$$
\frac{2}{e^{2x} + 1} < \frac{2}{e^2 + 1}
$$

Thus

$$
-\frac{2}{e^{2x} + 1} > -\frac{2}{e^2 + 1}
$$

Hence

$$
\begin{align*}
1 - \frac{2}{e^{2x} + 1} &> 1 - \frac{2}{e^2 + 1} \\
\text{Clearly } 1 + \frac{1}{e^{2x}} &> 1 \text{ as } \frac{1}{e^{2x}} > 0
\end{align*}
$$

Now observe that

$$
\begin{align*}
& 2x \cdot (1 + \varepsilon) - 2x \cdot \frac{3}{4} - \frac{1}{2}x \\
&= 2x \cdot (1 + \varepsilon) - 2x \cdot \frac{3}{4} - 2x \cdot \frac{1}{4} \\
&= 2x \cdot \left( \left(1 + \varepsilon\right) - \frac{3}{4} - \frac{1}{4} \right) \\
&= 2x \cdot (1 + \varepsilon - 1) \\
&= 2x \cdot \varepsilon
\end{align*}
$$
As per this lemma's hypothesis, $0 < e^{-\frac{3}{2}|x|} < \varepsilon \le \frac{1}{4}$. We know with $x > 1 > 0$ that $2x \cdot (1 + \varepsilon) - 3x \cdot \frac{3}{4} - \frac{1}{2}x > 0$. This means that $e^{2x \cdot (1+\varepsilon) - 2x \cdot \frac{3}{4} - \frac{1}{2}x} > 1$. As $\frac{1}{e^{2\varepsilon}} > 0$, we have $e^{2x \cdot (1+\varepsilon) - 2x \cdot \frac{3}{4} - \frac{1}{2}x} + \frac{1}{e^{2x}} > 1$ and therefore $\frac{1}{e^{2x \cdot (1+\varepsilon) - 2x \cdot \frac{3}{4} - \frac{1}{2}x} + \frac{1}{e^{2x}} < 1$.

As per lemma 10 as $\varepsilon$ satisfies $e^{-\frac{3}{2}x} < \varepsilon \le \frac{1}{4}$, we have $\left|e^{-\frac{1}{2}x} \cdot \frac{e^{2x\varepsilon} - 1}{\varepsilon \cdot e^{\frac{3}{2}x}}\right| \le 2$. Therefore $|\varepsilon'| \le |\varepsilon| \cdot 1.3131 \cdot 1.2 = |\varepsilon| \cdot 2.6262$ □

**Theorem 1.** Let $x \in \mathbb{R}$ be real numbers and $\varepsilon \in \mathbb{R}$ be bounded by $|\varepsilon| \le \bar{\varepsilon}$ with $\bar{\varepsilon} \in \mathbb{R}^+$, $\bar{\varepsilon} \le \frac{1}{4}$. Then there exist $\varepsilon' \in \mathbb{R}^+$, $\varepsilon' \le 2.6262\bar{\varepsilon}$ such that for all $x, \varepsilon$ there exist $\varepsilon'$ such that $\tanh(x \cdot (1+\varepsilon)) = \tanh(x) \cdot (1+\varepsilon')$ with $|\varepsilon'| \le \bar{\varepsilon}'$

*Proof.* Trivial by application of Lemma 5, 11, and 12

<figure><img src="image_9.png" /></figure>

## V. TANH-ABSOLUTE ERROR

**Lemma 1 (Convert relative perturbation to additive perturbation).**
For any $x \in \mathbb{R}$ and any $\varepsilon \in \mathbb{R}$,

$$x(1 + \varepsilon) = x + \Delta x, \quad \text{with } \Delta x := x\varepsilon.$$

Hence,

$$\tanh(x(1 + \varepsilon)) = \tanh(x + \Delta x).$$

*Proof.* Direct algebra shows:

$$x(1 + \varepsilon) = x + x\varepsilon.$$

Define

$$\Delta x := x\varepsilon.$$

Then by substitution,

$$x(1 + \varepsilon) = x + \Delta x.$$

Consequently, for the hyperbolic tangent function,

$$\tanh(x(1 + \varepsilon)) = \tanh(x + \Delta x).$$

This proves that any relative perturbation $x\varepsilon$ can be converted into an additive perturbation $\Delta x$ for use in the Taylor expansion. □

**Lemma 2 (Taylor expansion of $\tanh(x + \Delta x)$).** Let

$$\Delta y := \tanh(x + \Delta x) - \tanh(x).$$

Then, expanding around $x$ up to third order with remainder gives:

$$\Delta y = f'(x)\Delta x + \frac{f''(x)}{2}\Delta x^2 + \frac{f^{(3)}(x)}{6}\Delta x^3 + R_3(x, \Delta x),$$

where $R_3(x, \Delta x)$ is the Lagrange remainder.

*Proof.* By Taylor's theorem with Lagrange remainder:

$$f(x+\Delta x) = f(x)+f'(x)\Delta x+\frac{f''(x)}{2}\Delta x^2+\frac{f^{(3)}(x)}{6}\Delta x^3+R_3(x, \Delta x),$$

where there exists $\xi$ between $x$ and $x + \Delta x$ such that

$$R_3(x, \Delta x) = \frac{f^{(4)}(\xi)}{24} \Delta x^4.$$

Subtracting $f(x)$ gives

$$\Delta y = f'(x)\Delta x + \frac{f''(x)}{2}\Delta x^2 + \frac{f^{(3)}(x)}{6}\Delta x^3 + R_3(x, \Delta x),$$

with

$$|R_3(x, \Delta x)| \le \frac{\sup_{t \in [x, x+\Delta x]} |f^{(4)}(t)|}{24} |\Delta x|^4.$$

This provides a polynomial approximation plus remainder which bounds the error. □

<figure><img src="image_10.png" /></figure>

**Lemma 3 (Explicit derivatives for tanh).** *The derivatives of $f(x) = \tanh(x)$ are:*

$$\begin{aligned} f'(x) &= 1 - \tanh^2(x), \\ f''(x) &= -2\tanh(x)(1 - \tanh^2(x)), \\ f^{(3)}(x) &= 2(1 - \tanh^2(x))(1 - 3\tanh^2(x)), \\ f^{(4)}(x) &= 8\tanh(x)(1 - \tanh^2(x))(3\tanh^2(x) - 2). \end{aligned}$$

*Proof.* Step by step differentiation:

1) $f'(x) = \frac{d}{dx}\tanh(x) = 1 - \tanh^2(x)$.

2) $f''(x) = \frac{d}{dx}(1 - \tanh^2(x)) = -2\tanh(x)(1 - \tanh^2(x))$.

3) $f^{(3)}(x) = \frac{d}{dx}[-2\tanh(x)(1 - \tanh^2(x))] = 2(1 - \tanh^2(x))(1 - 3\tanh^2(x))$.

4) $f^{(4)}(x) = \frac{d}{dx}[2(1 - \tanh^2(x))(1 - 3\tanh^2(x))] = 8\tanh(x)(1 - \tanh^2(x))(3\tanh^2(x) - 2)$.

Hence all derivatives are expressed in terms of $\tanh(x)$, which will be used for the remainder bound. □

**Lemma 4 (Bound the 4th derivative).** Let $f^{(4)}(x) = 8\tanh(x)(1 - \tanh^2(x))(3\tanh^2(x) - 2)$. Then

$$\sup_{x \in \mathbb{R}} |f^{(4)}(x)| \approx 7.0882.$$

*Proof.* Step 1. Change of variable: $u = \tanh(x) \in (-1, 1)$. Then

$$f^{(4)}(x) = g(u) = 8u(1 - u^2)(3u^2 - 2), \quad u \in (-1, 1).$$

Step 2. Expand:

$$g(u) = 8u(1-u^2)(3u^2-2) = 8u(-3u^4+5u^2-2) = -24u^5+40u^3-16u.$$

Step 3. Find critical points of $|g(u)|$:

$$g'(u) = -120u^4 + 120u^2 - 16 = -8(15u^4 - 15u^2 + 2) = 0.$$

Solve $15u^4 - 15u^2 + 2 = 0$ with $v = u^2$:

$$15v^2 - 15v + 2 = 0 \implies v = \frac{15 \pm \sqrt{105}}{30} \approx 0.84157, 0.15843.$$

Thus candidate points: $u = \pm 0.9174, \pm 0.3980$.

Step 4. Evaluate $|g(u)|$ at critical points:

$$|g(\pm 0.9174)| \approx 7.088, \quad |g(\pm 0.3980)| \approx 1.984.$$

Endpoints $u = \pm 1$ give $g(\pm 1) = 0$.

Step 5. Maximum:

$$\sup_{x \in \mathbb{R}} |f^{(4)}(x)| = \max |g(u)| \approx 7.0882.$$

<figure><img src="image_11.png" /></figure>

**Theorem 1 (Absolute-error bound for tanh).** For $x \in \mathbb{R}$ and perturbation $\Delta x$:

$$\begin{aligned} & |\tanh(x + \Delta x) - \tanh(x)| \le (1 - \tanh^2(x)) |\Delta x| \\ & \qquad + |\tanh(x)|(1 - \tanh^2(x)) |\Delta x|^2 \\ & \qquad + \frac{1}{3}(1 - \tanh^2(x))|1 - 3\tanh^2(x)| |\Delta x|^3 \\ & \qquad + 0.29535 |\Delta x|^4. \end{aligned}$$
Proof. From Lemma 2:

$$ \Delta y = f'(x)\Delta x + \frac{f''(x)}{2}\Delta x^2 + \frac{f'''(x)}{6}\Delta x^3 + R_3(x, \Delta x), $$

and Lemma 3 gives derivatives. Lemma 4 bounds the remainder:

$$ |R_3(x, \Delta x)| \le \frac{\sup |f^{(4)}(x)|}{24} |\Delta x|^4 \approx 0.29535 |\Delta x|^4. $$

Substituting explicit derivatives into $\Delta y$ and taking absolute values term by term gives the final inequality. □

## VI. RELU- RELATIVE ERROR

Let $x \in \mathbb{R}$ and consider a small perturbation $\Delta x \in \mathbb{R}$. The perturbed input is defined as

$$ \tilde{x} = x + \delta. $$

The ReLU function is given by

$$ \mathrm{ReLU}(x) = \max(0, x). $$

### Perturbation

The relative perturbation (for $x \neq 0$) is

$$ \frac{|\delta x|}{|x|} $$

The corresponding change in the output is

$$ \Delta y = \mathrm{ReLU}(\tilde{x}) - \mathrm{ReLU}(x). $$

If $\mathrm{ReLU}(x) > 0$, the relative perturbation in the output is

$$ \frac{|\Delta y|}{|\mathrm{ReLU}(x)|} $$

**Theorem 1.** Lipschitz Property of ReLU The Rectified Linear Unit (ReLU) function is Lipschitz continuous with Lipschitz constant $L = 1$. That is, for all $u, v \in \mathbb{R}$,

$$ |\mathrm{ReLU}(u) - \mathrm{ReLU}(v)| \le |u - v|. $$

Proof. We can prove this by considering four cases:

1) Case 1: $x \le 0$ and $x + \delta \le 0$

Then both values are zero:

$$ \mathrm{ReLU}(x) = 0, \quad \mathrm{ReLU}(x + \delta) = 0. $$

Hence,

$$ |\mathrm{ReLU}(x + \delta) - \mathrm{ReLU}(x)| = 0 \le |\delta|. $$

2) Case 2: $x \ge 0$ and $x + \delta \ge 0$

Here:

$$ \mathrm{ReLU}(x + \delta) - \mathrm{ReLU}(x) = (x + \delta) - x = \delta, $$

so

$$ |\mathrm{ReLU}(x + \delta) - \mathrm{ReLU}(x)| = |\delta|. $$

3) Case 3: $x \le 0 < x + \delta$

Here:

$$ \mathrm{ReLU}(x) = 0, \quad \mathrm{ReLU}(x + \delta) = x + \delta > 0. $$

The change in output is $x + \delta$. Since $x \le 0$, we have $\delta \ge -x$, implying

$$ 0 < x + \delta \le |\delta|. $$

4) Case 4: $x \ge 0 \ge x + \delta$

Here:

$$ \mathrm{ReLU}(x) = x > 0, \quad \mathrm{ReLU}(x + \delta) = 0. $$

The change in output is $|x|$. Hence,

$$ |\mathrm{ReLU}(x + \delta) - \mathrm{ReLU}(x)| = x \le |\delta|. $$

In all cases, the inequality

$$ |\mathrm{ReLU}(x + \delta) - \mathrm{ReLU}(x)| \le |\delta| $$

holds. Therefore, the ReLU function is Lipschitz continuous. The relative perturbation in the output (when defined) never exceeds that of the input:

$$ |\Delta y| \le |\delta|. $$

□

## VII. RELU - ABSOLUTE ERROR

**Lemma 1** (Piecewise exact expression for the output change).

Let $\Delta x \in \mathbb{R}$ be an additive perturbation, and set

$$ \Delta y := \mathrm{ReLU}(x + \Delta x) - \mathrm{ReLU}(x). $$

Then exactly:

$$ \Delta y = \begin{cases} \Delta x, & x > 0 \text{ and } x + \Delta x > 0, \\ 0, & x \le 0 \text{ and } x + \Delta x \le 0, \\ -(x), & x > 0 \text{ and } x + \Delta x \le 0, \\ x + \Delta x, & x \le 0 \text{ and } x + \Delta x > 0. \end{cases} $$

Proof. Direct substitution into the piecewise definition of ReLU:

1) If both $x > 0$ and $x + \Delta x > 0$, then ReLU is identity: $\Delta y = (x + \Delta x) - x = \Delta x$.

2) If both $x \le 0$ and $x + \Delta x \le 0$, outputs are zero: $\Delta y = 0 - 0 = 0$.

3) If $x > 0$ and $x + \Delta x \le 0$, outputs cross zero: $\Delta y = 0 - x = -x$.

4) If $x \le 0$ and $x + \Delta x > 0$, outputs cross zero: $\Delta y = (x + \Delta x) - 0 = x + \Delta x$.

□

**Lemma 2** (1-Lipschitz bound — simple absolute-error bound).

For any $x, \Delta x \in \mathbb{R}$,

$$ |\mathrm{ReLU}(x + \Delta x) - \mathrm{ReLU}(x)| \le |\Delta x|. $$

Proof. For all real $u, v$, it holds that

$$ |\max(0, u) - \max(0, v)| \le |u - v|, $$

which follows by considering $u \ge v$ or $u < v$ and using monotonicity of $\max(0, \cdot)$. Apply with $u = x + \Delta x$ and $v = x$. □

**Lemma 3** (Worst-case crossing bound). If the perturbation causes the input to cross the nondifferentiable point at 0 (i.e., the sign of $x$ and $x + \Delta x$ differ), then

$$ |\Delta y| \le \min\{|x|, |x + \Delta x|\} \le |\Delta x|. $$

Proof. From Lemma 1 in crossing cases:

• If $x > 0$ and $x + \Delta x \le 0$, then $|\Delta y| = |-x| = |x| \le |\Delta x|$ since $\Delta x \le -x$.

• If $x \le 0$ and $x + \Delta x > 0$, then $|\Delta y| = |x + \Delta x| \le |\Delta x|$ since $\Delta x \ge -x$.

□
**Theorem 1** (Absolute-error characterization and bound for ReLU). For all $x \in \mathbb{R}$ and additive perturbation $\Delta x \in \mathbb{R}$, the exact $\Delta y$ is given piecewise as in Lemma 1. Consequently, the uniform bound

$$|\mathrm{ReLU}(x + \Delta x) - \mathrm{ReLU}(x)| \leq |\Delta x|$$

holds for all $x, \Delta x$. Moreover, if $x$ and $x + \Delta x$ lie in the same linear region (both $> 0$ or both $\le 0$), then $\Delta y$ is exact and linear: either $\Delta y = \Delta x$ or $\Delta y = 0$.

*Proof.* Combine Lemmas 1–3. □

VIII. SIGMOID-RELATIVE ERROR

**Lemma 1** (Convert relative perturbation to additive perturbation). For any $x \in \mathbb{R}$ and any $\varepsilon \in \mathbb{R}$,

$$x(1 + \varepsilon) = x + \Delta x, \quad \text{with } \Delta x := x\varepsilon.$$

Hence,

$$\sigma(x(1 + \varepsilon)) = \sigma(x + \Delta x).$$

*Proof.* Direct algebra shows:

$$x(1 + \varepsilon) = x + x\varepsilon.$$

Define

$$\Delta x := x\varepsilon.$$

Then by substitution,

$$x(1 + \varepsilon) = x + \Delta x.$$

Consequently, for the sigmoid function,

$$\sigma(x(1 + \varepsilon)) = \sigma(x + \Delta x).$$

This proves that any relative perturbation $x\varepsilon$ can be converted into an additive perturbation $\Delta x$. □

**Lemma 2** (Exact relative error formula). Let $s := \sigma(x) \in (0, 1)$ and $\Delta x = x\varepsilon$. Then

$$\varepsilon' := \frac{\sigma(x + \Delta x) - \sigma(x)}{\sigma(x)} = \frac{(1 - s)(e^{\Delta x} - 1)}{1 - s + s e^{\Delta x}}.$$

*Proof.* From the identity

$$\sigma(x + \Delta x) - \sigma(x) = \frac{s(1 - s)(e^{\Delta x} - 1)}{1 - s + s e^{\Delta x}},$$

divide both sides by $s$ to obtain the relative error. □

**Lemma 3** (Uniform exponential bound). For all $x \in \mathbb{R}$ and $\varepsilon \in \mathbb{R}$,

$$|\varepsilon'| \leq e^{|x\varepsilon|} - 1.$$

*Proof.* From the previous lemma, the denominator satisfies

$$1 - s + s e^{\Delta x} \geq 1 - s > 0.$$

Hence,

$$|\varepsilon'| \leq \frac{(1 - s)|e^{\Delta x} - 1|}{1 - s} = |e^{\Delta x} - 1| \leq e^{|\Delta x|} - 1.$$

**Lemma 4** (First-order relative sensitivity). As $\varepsilon \to 0$, i.e., $\Delta x = x\varepsilon \to 0$,

$$\varepsilon' = (1 - \sigma(x)) x \varepsilon + \mathcal{O}((x\varepsilon)^2).$$

*Proof.* Expand $e^{\Delta x} = 1 + \Delta x + \mathcal{O}(\Delta x^2)$ in the relative error formula:

$$\begin{aligned} \varepsilon' &= \frac{(1 - s)(\Delta x + \mathcal{O}(\Delta x^2))}{1 - s + s(1 + \Delta x + \mathcal{O}(\Delta x^2))} = \frac{(1 - s)\Delta x}{1 + s\Delta x} + \mathcal{O}(\Delta x^2) \\ &= (1 - s)\Delta x + \mathcal{O}(\Delta x^2). \end{aligned}$$

□

**Theorem 1** (Relative-error bound for $\sigma$). Let $|\varepsilon| \le \bar{\varepsilon} \le \frac{1}{4}$. Then for all $x \in \mathbb{R}$,

$$|\varepsilon'| \leq e^{|x|\bar{\varepsilon}} - 1, \quad \varepsilon' = (1 - \sigma(x)) x \varepsilon + \mathcal{O}((x\varepsilon)^2).$$

*Proof.* Combine Lemma 3 and Lemma 4. The exponential bound gives a global worst-case estimate, while the linearization gives a local approximation for small $\varepsilon$. □

IX. SIGMOID-ABSOLUTE ERROR

**Lemma 1** (Convert relative to additive). For any $x \in \mathbb{R}$ and $\varepsilon \in \mathbb{R}$, set $\Delta x := x\varepsilon$. Then

$$\sigma(x(1 + \varepsilon)) = \sigma(x + \Delta x).$$

*Proof.* Immediate from $x(1 + \varepsilon) = x + \Delta x$. □

We will bound $\Delta y := \sigma(x + \Delta x) - \sigma(x)$.

**Lemma 2** (Global Lipschitz via MVT). For all $x, \Delta x \in \mathbb{R}$,

$$|\sigma(x + \Delta x) - \sigma(x)| \leq \frac{1}{4} |\Delta x|.$$

*Proof.* Mean Value Theorem with $\sigma'(z) = \sigma(z)(1 - \sigma(z)) \le \frac{1}{4}$. □

**Lemma 3** (Derivatives and curvature bound).

$$\sigma'(x) = \sigma(x)(1 - \sigma(x)), \quad \sigma''(x) = \sigma(x)(1 - \sigma(x))(1 - 2\sigma(x)).$$

Moreover,

$$\sup_{x \in \mathbb{R}} |\sigma''(x)| = \frac{1}{6\sqrt{3}} \approx 0.0962.$$

*Proof.* Differentiate and maximize $|\sigma''|$ by setting $t = \sigma(x) \in (0, 1)$ and optimizing $t(1 - t)(1 - 2t)$. □

**Lemma 4** (First-order Taylor with uniform remainder). For some $\xi$ between $x$ and $x + \Delta x$,

$$\sigma(x + \Delta x) = \sigma(x) + \sigma'(x) \Delta x + \frac{1}{2} \sigma''(\xi) \Delta x^2,$$

hence

$$|\sigma(x + \Delta x) - \sigma(x) - \sigma'(x)\Delta x| \leq \frac{1}{12\sqrt{3}} |\Delta x|^2.$$

*Proof.* Taylor's theorem and Lemma 3. □

**Lemma 5** (Exact difference). For any $x, \Delta x$,

$$\begin{aligned} \sigma(x + \Delta x) - \sigma(x) &= \frac{e^x (e^{\Delta x} - 1)}{(1 + e^x)(1 + e^{x+\Delta x})} \\ &= \frac{\sigma(x)(1 - \sigma(x))(e^{\Delta x} - 1)}{1 - \sigma(x) + \sigma(x) e^{\Delta x}}. \end{aligned}$$

*Proof.* Algebra from the logistic form. □

**Theorem 1** (Absolute-error bounds for $\sigma$). For $\Delta x = x\varepsilon$,

$$\begin{aligned} |\sigma(x + \Delta x) - \sigma(x)| &\leq \frac{1}{4} |\Delta x| \\ |\sigma(x + \Delta x) - \sigma(x) - \sigma'(x)\Delta x| &\leq \frac{1}{12\sqrt{3}} |\Delta x|^2. \end{aligned}$$
Moreover, the exact difference of Lemma 5 provides the sharper non-asymptotic estimate

$$|\sigma(x + \Delta x) - \sigma(x)| \leq \sigma(x)(1 - \sigma(x))(e^{|\Delta x|} - 1).$$

## X. SOFTMAX- RELATIVE ERROR

Let $\bar{\epsilon} \in \mathbb{R}^+$, $\bar{\epsilon} \le \frac{1}{4}$. Let $\underline{x} \in \mathbb{R}^N$. Let $\underline{\epsilon} \in \mathbb{R}^N$ such that $\forall \ i = 0, ..., N-1, |\underline{\epsilon}_i| \le \bar{\epsilon}$. Let $\tilde{\underline{x}} \in \mathbb{R}^N$ given by

$$\tilde{x}_i = x_i(1 + \epsilon_i), \quad \forall i = 0, ..., N-1$$

Let $\underline{y} = s(\underline{x})$ and $\tilde{\underline{y}} = s(\tilde{\underline{x}})$ for

$$s(\underline{x})_i = \frac{e^{x_i}}{\sum_{j=0}^{N-1} e^{x_j}}.$$

Let $\eta_i = \frac{\tilde{y}_i}{y_i} - 1, \quad \forall i = 0, ..., N-1$. We want to bound the $\eta_i$ for all $i$ in terms of $\bar{\epsilon}$. We have

$$\begin{aligned} \eta_i &= \frac{\frac{e^{x_i(1+\epsilon_i)}}{\sum_{j=0}^{N-1} e^{x_j(1+\epsilon_j)}} - 1}{\frac{e^{x_i}}{\sum_{j=0}^{N-1} e^{x_j}}} \\ &= e^{x_i \epsilon_i} \cdot \frac{\sum_{j=0}^{N-1} e^{x_j}}{\sum_{j=0}^{N-1} e^{x_j} e^{x_j \epsilon_j}} - 1 \\ &= \frac{\sum_{j=0}^{N-1} e^{x_j}}{\sum_{j=0}^{N-1} e^{x_j} e^{x_j \epsilon_j - x_i \epsilon_i}} - 1 \end{aligned}$$

Let $k = \text{argmax}_{i=0,...,N-1} x_i$. Clearly $x_i - x_k \le 0 \quad \forall i = 0, ..., N-1$. We have

$$\begin{aligned} \eta_i &= \frac{e^{x_k} \sum_{j=0}^{N-1} e^{x_j - x_k}}{e^{x_k} \sum_{j=0}^{N-1} e^{x_j - x_k} e^{x_j \epsilon_j - x_i \epsilon_i}} - 1 \\ &= \frac{\sum_{j=0}^{N-1} e^{x_j - x_k}}{\sum_{j=0}^{N-1} e^{x_j - x_k} (1 + (e^{x_j \epsilon_j - x_i \epsilon_i} - 1))} - 1 \end{aligned}$$

Hence

$$\begin{aligned} \eta_i &= \frac{\sum_{j=0}^{N-1} e^{x_j - x_k}}{\sum_{j=0}^{N-1} e^{x_j - x_k} + \sum_{j=0}^{N-1} e^{x_j - x_k} (e^{x_j \epsilon_j - x_i \epsilon_i} - 1)} - 1 \\ &= \frac{1}{1 + \frac{\sum_{j=0}^{N-1} e^{x_j - x_k} (e^{x_j \epsilon_j - x_i \epsilon_i} - 1)}{\sum_{j=0}^{N-1} e^{x_j - x_k}}} - 1 \\ &= \frac{1}{1 + \eta'_i} - 1 \end{aligned}$$

with $\eta'_i = \frac{\sum_{j=0}^{N-1} e^{x_j - x_k} (e^{x_j \epsilon_j - x_i \epsilon_i} - 1)}{\sum_{j=0}^{N-1} e^{x_j - x_k}}$.

We have

$$\begin{aligned} \eta'_i &= \sum_{j=0}^{N-1} \frac{e^{x_j - x_k}}{\sum_{l=0}^{N-1} e^{x_l - x_k}} \cdot (e^{x_j \epsilon_j - x_i \epsilon_i} - 1) \\ &= \sum_{j=0}^{N-1} \alpha_j \cdot (e^{x_j \epsilon_j - x_i \epsilon_i} - 1) \end{aligned}$$

with $\alpha_j = \frac{e^{x_j - x_k}}{\sum_{l=0}^{N-1} e^{x_l - x_k}}$. By applying the lemma on the image of softmax (see Lemma 1 of XI.),

$$\begin{aligned} |\eta'_i| &\leq |\sum_{j=0}^{N-1} \alpha_j \cdot (e^{x_j \epsilon_j - x_i \epsilon_i} - 1)| \\ &\leq \sum_{j=0}^{N-1} |\alpha_j| \cdot |e^{x_j \epsilon_j - x_i \epsilon_i} - 1| \\ &\leq \sum_{j=0}^{N-1} |e^{x_j \epsilon_j - x_i \epsilon_i} - 1| \end{aligned}$$

## XI. SOFTMAX- ABSOLUTE ERROR

**Lemma 1.** Let

$$s(\underline{x})_i = \frac{e^{x_i}}{\sum_{j=0}^{N-1} e^{x_j}}.$$

For any $\underline{x} \in \mathbb{R}^N$ and $i = 0, ..., N-1$,

$$0 \le s(\underline{x})_i \le 1.$$

*Proof.* For any $i = 0, ..., N-1$,

$$\begin{aligned} s(\underline{x})_i &= \frac{e^{x_i}}{\sum_{j=0}^{N-1} e^{x_j}} \\ &= \frac{e^{x_i}}{e^{x_i} + \sum_{j=0, j \neq i}^{N-1} e^{x_j}}. \end{aligned}$$

As $e^x \ge 0$ for all $x \in \mathbb{R}$, we have

$$\sum_{j=0, j \neq i}^{N-1} e^{x_j} \geq 0.$$

It follows that, for any $i = 0, ..., N-1$,

$$0 \le e^{x_i} \le e^{x_i} + \sum_{j=0}^{N-1} e^{x_j}.$$

Hence, for any $\underline{x} \in \mathbb{R}^N$ and $i = 0, ..., N-1$,

$$0 \le s(\underline{x})_i \le 1.$$

□

**Lemma 2.** Let $\bar{\delta} \le \frac{1}{4}$. For any $|\delta_i| \le \bar{\delta}$, $|e^{\delta_i} - 1| \le \frac{4}{3}\bar{\delta}$.

*Proof.* Suppose $\bar{\delta} \le \frac{1}{4}$. We have

$$|e^{\delta_i} - 1| = \left| \sum_{n=1}^{\infty} \frac{\delta_i^n}{n!} \right|$$

Thus,

$$|e^{\delta_i} - 1| \le \sum_{n=1}^{\infty} \frac{|\delta_i|^n}{n!} \le \sum_{n=1}^{\infty} \frac{\bar{\delta}^n}{n!}$$

By changing index, we have

$$|e^{\delta_i} - 1| \le \bar{\delta} \sum_{n=0}^{\infty} \frac{\bar{\delta}^n}{(n+1)!}$$

Thus,

$$|e^{\delta_i} - 1| \le \bar{\delta} \sum_{n=0}^{\infty} \frac{\bar{\delta}^n}{1 - \bar{\delta}} = \frac{\bar{\delta}}{1 - \frac{1}{4}}$$

Hence, as $\bar{\delta} \le \frac{1}{4}$,

$$|e^{\delta_i} - 1| \le \frac{\bar{\delta}}{1 - \frac{1}{4}} = \frac{4}{3}\bar{\delta}$$
□ We know that $|\beta| \le \Delta$. Hence $|\alpha - 1| \le |\beta|$. We have

$$ 
\begin{aligned}
|y_i| & \le |(e^{\delta_i} - 1) \cdot (\alpha - 1) + (e^{\delta_i} - 1) + (\alpha - 1)| \\
& \le |e^{\delta_i} - 1| \cdot |\alpha - 1| + |e^{\delta_i} - 1| + |\alpha - 1| \\
& \le \frac{4}{3}\bar{\delta} \cdot \frac{4}{3}\bar{\delta} + \frac{4}{3}\bar{\delta} + \frac{4}{3}\bar{\delta} \\
& = \frac{40}{9}\bar{\delta} \\
& \le 4.4445\bar{\delta}
\end{aligned}
$$

<figure><img src="image_12.png" /></figure>

**Theorem 1 (Absolute-error bound for softmax).** Let

$$s(\underline{x})_i = \frac{e^{x_i}}{\sum_{j=0}^{N-1} e^{x_j}}.$$

For $\bar{\delta} \in \mathbb{R}^+$, $\bar{\delta} \le \frac{1}{4}$ such that $|\delta_i| \le \bar{\delta}$ for all $i$ in a vector $\underline{\delta}$,

$$|s(\underline{x} + \underline{\delta})_i - s(\underline{x})_i| \le 4.4445\bar{\delta}.$$

*Proof.* Suppose we have $\bar{\delta} \le \frac{1}{4}$ such that $|\delta_i| \le \bar{\delta}$ for all $i$ in a vector $\underline{\delta}$. Let

$$\underline{y} = s(\underline{x} + \underline{\delta}) - s(\underline{x}).$$

Hence,

$$y_i = \frac{e^{x_i + \delta_i}}{\sum_{j=0}^{N-1} e^{x_j + \delta_j}} - \frac{e^{x_i}}{\sum_{j=0}^{N-1} e^{x_j}}.$$

Then,

$$ 
\begin{aligned}
\frac{e^{x_i + \delta_i}}{\sum_{j=0}^{N-1} e^{x_j + \delta_j}} &= e^{\delta_i} \cdot \frac{e^{x_i}}{\sum_{j=0}^{N-1} e^{x_j + \delta_j}} \\
&= e^{\delta_i} \cdot \frac{e^{x_i}}{\sum_{j=0}^{N-1} e^{x_j}} \cdot \frac{\sum_{j=0}^{N-1} e^{x_j}}{\sum_{j=0}^{N-1} e^{x_j + \delta_j}} \\
&= e^{\delta_i} \cdot \alpha \cdot s(\underline{x})_i, \quad \text{with } \alpha = \frac{\sum_{j=0}^{N-1} e^{x_j}}{\sum_{j=0}^{N-1} e^{x_j + \delta_j}}.
\end{aligned}
$$

Thus,

$$y_i = (e^{\delta_i} \cdot \alpha - 1)s(\underline{x})_i$$

We have

$$\alpha = \frac{1}{1 + \beta}, \quad \text{with } \beta = \frac{\sum_{j=0}^{N-1} e^{x_j + \delta_j}}{\sum_{j=0}^{N-1} e^{x_j}} - 1.$$

We can develop $\beta$,

$$ 
\begin{aligned}
\beta &= \frac{\sum_{j=0}^{N-1} e^{x_j + \delta_j} - \sum_{j=0}^{N-1} e^{x_j}}{\sum_{j=0}^{N-1} e^{x_j}} \\
&= \frac{\sum_{j=0}^{N-1} e^{x_j}(e^{\delta_j} - 1)}{\sum_{j=0}^{N-1} e^{x_j}}.
\end{aligned}
$$

Thus,

$$|\beta| \le \frac{\sum_{j=0}^{N-1} e^{x_j} |e^{\delta_j} - 1|}{\sum_{j=0}^{N-1} e^{x_j}} \le \frac{\sum_{j=0}^{N-1} e^{x_j} \cdot \Delta}{\sum_{j=0}^{N-1} e^{x_j}} = \Delta$$

with $\delta \in \mathbb{R}^+$ such that for all $j$, $|e^{\delta_j} - 1| \le \Delta$. By applying Lemma 1, we have

$$ 
\begin{aligned}
|y_i| & \le |e^{\delta_i} \cdot \alpha - 1| \cdot 1 \\
& = |((e^{\delta_i} - 1) + 1) \cdot ((\alpha - 1) + 1) - 1|
\end{aligned}
$$

Assume that $\bar{\delta} \le \frac{1}{4}$. We have

$$|\alpha - 1| = |\frac{1}{1 + \beta} - 1| = |\sum_{n=1}^{\infty} (-1)^n \beta^n|$$

The alternating series estimation theorem then gives

$$|\alpha - 1| \le |\beta|$$

Thus, by applying Lemma 2, we have for all $j$

$$|e^{\delta_j} - 1| \le \frac{4}{3}\bar{\delta} := \Delta$$

## XII. DISCUSSION

The error characterizations for each activation functions behave very differently under perturbations. ReLU demonstrates the simplest and most predictable behavior: its absolute error is always bounded by the magnitude of the input perturbation, and the bound is tight. Because ReLU is piecewise linear, its perturbation behavior is determined by whether the input crosses zero. When it does not, the mapping is exactly linear; when it does, the maximal deviation cannot exceed the original distance to the boundary. In contrast, smooth activations such as sigmoid and tanh require higher-order derivative information, making their perturbation responses more dependent on the local curvature of the function. ELU on the other hand shares similar characteristic to ReLU except for smoothing in the negative part, which in a way is similar to other smooth functions. While its error within linear and exponential bounds are lower, across regions it produces more complex errors due to the difference in scaling.

The expressions for sigmoid and tanh incorporate not only first-order derivatives but also second- and third-order terms, plus a fourth-order bounded remainder term. This is significant because it demonstrates that for smooth activations the perturbation response is not merely a scaled version of the input error; instead, it depends on the function's nonlinear growth around the point of evaluation. For sigmoid, the worst-case relative error grows exponentially with $x$, indicating sensitivity far from the origin. Tanh, while also nonlinear, benefits from bounded derivatives, leading to a well-controlled remainder term. These mathematical differences provide insight into why neural networks with smooth activations can exhibit more complex and sometimes less stable numerical behavior compared to ReLU.

## XIII. CONCLUSION

The absolute-error bounds show that ReLU is inherently stable with respect to perturbations: the output error never exceeds the input error, making it a robust choice for architectures sensitive to noise or quantization effects. Tanh and sigmoid also provide well-defined absolute bounds, but their higher-order remainder terms reveal additional curvature-induced error contributions. This implies that models relying on tanh or sigmoid activations may require more careful numerical handling when dealing with large inputs or when perturbations propagate through many layers. Nonetheless, the tanh bound shows that the growth is still controlled and globally limited.

The relative-error formulas highlight practical distinctions. Tanh has a multiplicative perturbation structure with bounded amplification, making it relatively stable in multiplicative-noise scenarios. Sigmoid, however, can experience exponential growth for large $x$, which is consistent with the empirical difficulty of
<table><caption>TABLE II: Absolute and relative error bounds for common activation functions.</caption><thead><tr><th>Activation</th><th>Absolute Error Bound</th><th>Relative Error Bound</th></tr></thead><tbody><tr><td>ReLU f(x) = max(0, x)</td><td>|ReLU(x + Δx) - ReLU(x)| ≤ |Δx|.<br>Exact piecewise characterization: if x and x + Δx are in the same region, Δy = Δx (if x > 0) or Δy = 0 (if x ≤ 0).</td><td>For any perturbation δ,<br>|ReLU(x + δ) - ReLU(x)| ≤ |δ|.<br>When the relative ratio is defined,<br>|Δy| / |ReLU(x)| ≤ |δ| / |x| (x > 0).<br>Undefined at x = 0.</td></tr><tr><td>Sigmoid σ(x)</td><td>Absolute bound:<br>|σ(x + Δx) - σ(x)| ≤ 1/4 |Δx|.<br>Sharper non-asymptotic estimate:<br>|σ(x + Δx) - σ(x)| ≤ σ(x)(1 - σ(x))(e^|Δx| - 1).<br>Second-order refinement:<br>σ(x + Δx) - σ(x) - σ'(x)Δx ≤ 1 / (12√3) |Δx|^2.</td><td>If Δx = xε and |ε| ≤ ε̄ ≤ 1/4, then<br>ε' = (1 - σ(x)) xε + O((xε)^2),<br>and<br>|ε'| ≤ e^|x|ε̄ - 1.</td></tr><tr><td>Tanh tanh(x)</td><td>|tanh(x + Δx) - tanh(x)| ≤ (1 - tanh^2 x)|Δx| + |tanh x|(1 - tanh^2 x)|Δx|^2 + 1/3(1 - tanh^2 x)|1 - 3tanh^2 x||Δx|^3 + 0.29535|Δx|^4.</td><td>If |ε| ≤ ε̄ ≤ 1/4, then there exists ε̄' ≤ 2.6262 ε̄ such that<br>tanh(x(1 + ε)) = tanh(x)(1 + ε')<br>with |ε'| ≤ ε'.</td></tr><tr><td>Softmax s(x)<sub>i</sub></td><td>For δ ∈ ℝ<sup>+</sup>, δ ≤ 1/4 such that |δ<sub>i</sub>| ≤ δ for all i in a vector δ,<br>|s(x̄ + δ)<sub>i</sub> - s(x)<sub>i</sub>| ≤ 4.4445δ̄.</td><td></td></tr></tbody></table>

training deep sigmoidal networks. ReLU's relative error is only meaningful when the input remains in the same linear region. Otherwise, the concept breaks down because the denominator vanishes. Taken together, these characteristics explain the use for ReLU in deep networks, while also showing that smooth activations remain useful when their numerical behavior is well understood and matched to the application context.

## XIV. FUTURE WORK

While the present work focuses on the error amplification introduced by activation functions during the forward pass, a natural next step is to extend the same absolute- and relative-error analysis to their derivatives, which are fundamental to gradient-based training. During backpropagation, learning dynamics depend on $f'(x)$, making the numerical stability of the derivative equally important. Evaluating how perturbations propagate through $f'(x + Δx)$ will allow us to quantify the magnitude of gradient distortion introduced by floating-point arithmetic at each training step.

Such an analysis is needed because the derivatives of commonly used activation functions exhibit widely different behaviors. Linear activations yield constant derivatives, while the derivatives of sigmoid and tanh rapidly approach zero, potentially magnifying the relative component of numerical noise. ReLU introduces discontinuities where the derivative is undefined, creating challenges for both absolute-error bounds and backward-pass precision.

By characterizing the worst-case and nominal error propagation through $f'(x)$, future work can yield a principled understanding

of gradient-level numerical stability, complementing the forward-pass analysis presented here.

## REFERENCES

[1] D. K. Zhang and A. Aiken, "Automatic verification of floating-point accumulation networks," in *Computer Aided Verification*, R. Piskac and Z. Rakamarić, Eds. Cham: Springer Nature Switzerland, 2025, pp. 215–237.

[2] T. Devi and N. Deepa, "A novel intervention method for aspect-based emotion using exponential linear unit (elu) activation function in a deep neural network," in *2021 5th international conference on intelligent computing and control systems (ICICCS)*. IEEE, 2021, pp. 1671–1675.

[3] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, "Fast and accurate deep network learning by exponential linear units (elus)." arxiv 2015, arXiv preprint arXiv:1511.07289, vol. 10, 2020.

[4] P. Niu, "Convolutional neural network for gesture recognition human-computer interaction system design," *PloS one*, vol. 20, no. 2, p. e0311941, 2025.

[5] R. R. Asaad and R. I. Ali, "Back propagation neural network (bpnn) and sigmoid activation function in multi-layer networks," *Academic Journal of Nawroz University*, vol. 8, no. 4, 2019.

[6] B. Warner and M. Misra, "Understanding neural networks as statistical tools," *The american statistician*, vol. 50, no. 4, pp. 284–293, 1996.

[7] B. S. Anami and C. V. Sagarnal, "Influence of different activation functions on deep learning models in indoor scene images classification," *Pattern Recognition and Image Analysis*, vol. 32, no. 1, pp. 78–88, 2022.

[8] K. A. Sayin, N. K. Gürsoy, T. Yolcu, and A. Gürsoy, "On the synergy of optimizers and activation functions: A cnn benchmarking study," *Mathematics*, vol. 13, no. 13, p. 2088, 2025.