# Exploring bona-fide optimal noise for Bayesian
parameter estimation

Jia Liu, Fabing Duan, François Chapeau-Blondeau, Derek Abbott

<figure><img src="image_1.png" /></figure>

► To cite this version:

Jia Liu, Fabing Duan, François Chapeau-Blondeau, Derek Abbott. Exploring bona-fide optimal noise for Bayesian parameter estimation. IEEE Access, 2020, 8, pp.18822-18831. 10.1109/ACCESS.2020.2968420. [hal-02462493](https://HAL Id: hal-02462493)

https://univ-angers.hal.science/hal-02462493

Submitted on 15 Feb 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
Received December 16, 2019, accepted January 15, 2020, date of publication January 21, 2020, date of current version January 30, 2020.

Digital Object Identifier 10.1109/ACCESS.2020.2968420

# Exploring Bona Fide Optimal Noise for Bayesian Parameter Estimation

JIA LIU<fnref n="1" />, FABING DUAN<fnref n="1" />, FRANÇOIS CHAPEAU-BLONDEAU<fnref n="2" />, AND DEREK ABBOTT<fnref n="3" />, (Fellow, IEEE)

Corresponding author: Fabing Duan (fabing.duan@hotmail.com)

This work was supported by the National Natural Science Foundation of China under Grant 61573202.

**ABSTRACT** In this paper, we investigate the benefit of intentionally added noise to observed data in various scenarios of Bayesian parameter estimation. For optimal estimators, we theoretically demonstrate that the Bayesian Cramér-Rao bound for the case with added noise is never smaller than for the original data, and the updated minimum mean-square error (MSE) estimator performs no better. This motivates us to explore the feasibility of noise benefit in some useful suboptimal estimators. Several Bayesian estimators established from one-bit-quantizer sensors are considered, and for different types of pre-existing background noise, optimal distributions are determined for the added noise in order to improve the performance in estimation. With a single sensor, it is shown that the optimal added noise for reducing the MSE is actually a constant bias. However, with parallel arrays of such sensors, bona fide optimal added noise, no longer a constant bias, is shown to reduce the MSE. Moreover, it is found that the designed Bayesian estimators can benefit from the optimal added noise to effectively approach the performance of the minimum MSE estimator, even when the assembled sensors possess different quantization thresholds.

**INDEX TERMS** Bayesian estimator, Bayesian information, bona fide optimal noise, noise benefit, parameter estimation.

## I. INTRODUCTION

Adding some random noise to a signal before quantization has been shown to be beneficial for analog to digital converters resulting in smaller signal distortion and wide system dynamic range [1]–[3]. The technique of adding noise or *dithering* was perhaps the first that recognized a beneficial role for noise in a signal processing context [1]–[5]. Then the term *stochastic resonance* was initially coined to describe the possible mechanism for maximizing the response of a bistable system to a small periodic force by optimizing the noise intensity to a non-zero level [6]. Stochastic resonance attracted much attention in physics and biology [7]–[17] soon afterwards. Gammaitoni [1] first pointed out that stochastic resonance, far from being limited to a resonant phenomenon, can also be interpreted as a special case of dithering and is related to the notion of noise-induced threshold crossings. Similarly, Collins *et al.* [18], [19] coined the

The associate editor coordinating the review of this manuscript and approving it for publication was Cesar Vargas-Rosales<fnref n="1" />.

term of *aperiodic stochastic resonance* for characterizing the noise-induced behavior in excitable systems with aperiodic inputs, and Stocks [20] defined the *suprathreshold stochastic resonance* using Shannon's average mutual information measure between the input and the output of a summing network of threshold devices. These widened concepts of stochastic resonance that are closely relate to the field of statistical signal processing, are now widely referred to as noise enhancement or noise benefit [21]–[50].

There are two main situations whereby the noise benefit has been exploited in signal estimation: one is implementing suboptimal estimators in practical estimation problems to avoid too complex or intractable optimal estimators in general [34], [51]. The other is estimating a signal from observed data of a number of low-cost sensors (e.g. quantizers) in a fusion center. These sensors with a few bits are often deployed over a sensing field to compose wireless sensor networks in distributed estimation problems [38], [52], [53]. For the first situation, the performances of some easily implemented suboptimal estimators were shown to be substantially

<footnote n="1">¹Institute of Complexity Science, Qingdao University, Qingdao 266071, China</footnote>

<footnote n="2">²Laboratoire Angevin de Recherche en Ingénierie des Systèmes (LARIS), University of Angers, 49000 Angers, France</footnote>

<footnote n="3">³Centre for Biomedical Engineering (CBME), School of Electrical and Electronic Engineering, The University of Adelaide, Adelaide, SA 5005, Australia</footnote>
improved by exploiting the benefits of added noise [22], [32], [34]–[38], [42], [46]–[49]. In the second situation, rich results from utilizing various kinds of noise have been reported for quantized observations [1], [2], [12]–[14], [20], [21], [23]–[25], [27], [30], [31], [33], [36]–[43], [45]–[49]. For instance, Papadopoulos et al. developed a methodology of additive control input before signal quantization at the sensor to achieve the maximum possible performance for quantizer-based networks [23]. They also noticed the option of using feedback from past observations for efficient estimators in terms of mean-square error (MSE) [23]. Modeling the suprathreshold stochastic resonance as stochastic quantization, McDonnell et al. systematically studied the optimal linear and nonlinear decoding schemes associated with the information bound on the MSE [12]. The optimal Bayesian estimators constructed by the quantizer outputs were also explicitly derived, and a basic mechanism was provided for the performance improvement of optimal Bayesian estimator by increasing the noise level [24], [25], [30], [31].

Since the addition of noise can be artificially designed, then finding the optimal probability density function (PDF) of added noise becomes an interesting question [26], [28], [29], [32], [33], [36]–[43], [45]–[49]. Especially, Chen et al. considered all possible PDFs of added noise to optimize an arbitrary fixed or variable estimator and proved that the optimal noise, if it exists, is just a finite number of (no more than two) constant vectors by using the properties of convex hull and Caratheodory theorem [28], [29], [32], [38], [39]. Then, this kind of optimal noise PDFs inspired a series of theoretical improvability of estimation under various estimation criteria [26], [33], [37], [40]–[43], [45]–[49]. An interesting question is whether optimal bona fide noise, rather than a constant bias, exists for enhancing the estimator performance or not. Interestingly, Uhlich [42] proposed a new estimator constructed by bagged estimators that are modified by mutually independent noise samples, and derived the necessary and sufficient conditions for the existence of the optimal noise. Uhlich [42] also found that the optimal noise PDF, not limited to the noise type revealed in [28], [29], [32], [38], [39], has non-trivial complicated shapes. For minimizing the MSE of a combiner of identical estimators, we also found that solving the optimal noise PDF is a constrained nonlinear functional optimization problem, and approximate optimal PDFs of the optimal noise are also found to be complicated [46], [48].

Although many important results for noise benefits in estimators have been obtained, there are still some unsolved questions. For instance, it is known that, for random parameter estimation, a lower bound on the MSE of any estimator is called the Bayesian Cramér-Rao bound (BCRB) that is directly calculated from the primary observations [54]. Then, two interesting questions need to be addressed: after artificially injecting noise into the primary observed data, resulting updated data—so, does the corresponding new BCRB calculated on the updated data increase or decrease? Can the minimum MSE (MMSE) estimator deduced from the updated

data achieve a lower MSE than that of the original MMSE estimator based on the primary observations?

In this paper, we will theoretically provide the solutions to aforementioned crucial questions, and elucidate the possibility of exploiting the noise benefits in some easily implemented suboptimal estimators. We argue that the noise-enhanced Bayesian estimators proposed by [22]–[43], [45]–[49] in recent years can be mainly classified into four categories: (i) the noise-modified estimator established on a single sensor [32], (ii) a linear minimum MSE (LMMSE) estimator based on a single sensor, (iii) the noise-enhanced Bayesian estimator as the average of outputs of an ensemble of identical sensors [42] and (iv) the linear combination estimator executing the LMMSE transform on an array of identical or nonidentical sensors [48].

For a noise-modified estimator, it was proved that the optimal added noise is just a constant bias [32]. However, the optimized MSE achieved by the noise-modified estimator has a long way to catch up the MSE of the original MMSE estimator. In order to provide useful parameter estimation, it is useful to incorporate both the statistical properties of the original background noise as well as the prior knowledge of the random parameter. This design principle leads to the LMMSE estimator with adaptively adjustable weights that depend only on the first two moments of the joint PDF. We demonstrate that, based on a single sensor, the LMMSE estimator can obtain a lower MSE than the noise-modified estimator does, but the minimum MSE achieved by the LMMSE estimator is still larger than that of the MMSE estimator. Moreover, the optimal added noise is still a constant bias for minimizing the MSE of the LMMSE estimator, and not bona fide random noise.

Furthermore, based on a sufficiently large number of identical sensors, it is shown that the noise-enhanced estimator can efficiently approach the MMSE estimator by the bona fide optimal noise that is not restricted to a constant bias. However, the noise-enhanced estimator is inapplicable to an ensemble of nonidentical sensors. For the general case of nonidentical sensors, we theoretically demonstrate that the linear combination estimator always outperforms the noise-enhanced estimator, and is able to perform as efficiently as the MMSE estimator. From observations of one-bit-quantizer sensors, we illustratively confirm the aforementioned conclusions of the performance comparison between four considered estimators. The complicated PDFs of bona fide optimal noise for the linear combination estimator are also presented. These interesting results of mutually independent added noise components in sensors manifest their potential benefits to the parameter estimation problems.

## II. PARAMETER ESTIMATION MODEL AND PROBLEM FORMULATION

Consider a parameter estimation scenario with the scalar observation

$$x_n = s(\theta) + \xi_n, \quad (1)$$
where $s(\theta)$ is a function of an unknown random parameter $\theta$ with a prior PDF $f_\theta$, and the mutually independent samples $\xi_n$ of background noise, uncorrelated with $\theta$, are with zero mean and common PDF $f_\xi$ for $n = 1, 2, \dots, N$. Letting $\mathbf{x} = [x_1, x_2, \dots, x_N]^\top$, the statistical characteristic of observation data can be described by the joint PDF $f_{\mathbf{x},\theta}(\mathbf{x}, \theta) = f_{\mathbf{x}|\theta}(\mathbf{x}|\theta)f_\theta(\theta) = f_\xi(\mathbf{x}-s(\theta))f_\theta(\theta)$. Here, $f_{\mathbf{x}|\theta}$ is the conditional PDF. It is well known [54] that the MSE $\mathcal{R}$ of any estimator $\hat{\theta}(\mathbf{x})$ satisfies the inequality

$$ \mathcal{R} = E_{\mathbf{x},\theta}[(\hat{\theta}(\mathbf{x}) - \theta)^2] = E_{\mathbf{x},\theta}[\varepsilon^2] \geq J_B^{-1} = \text{BCRB}, \quad (2) $$

where the error of estimator is $\varepsilon = \hat{\theta}(\mathbf{x}) - \theta$, and the Bayesian information $J_B$ is defined as

$$ J_B = E_{\mathbf{x},\theta} \left[ \left( \frac{\partial \ln f_{\mathbf{x},\theta}(\mathbf{x}, \theta)}{\partial \theta} \right)^2 \right] = E_{\theta}[J_F(\theta)] + J_{\theta} \quad (3) $$

with the prior Fisher information $J_\theta = E_\theta[(\partial \ln f_\theta(\theta)/\partial \theta)^2]$ of the prior PDF $f_\theta$ and the Fisher information $J_F(\theta) = E_{\mathbf{x}|\theta}[(\partial \ln f_{\mathbf{x}|\theta}(\mathbf{x}|\theta)/\partial \theta)^2]$ of observation data $\mathbf{x}$ with respect to the parameter $\theta$ [54]. This lower bound of $J_B^{-1}$ in (2) on the MSE $\mathcal{R}$ of any estimator is also called BCRB [54]. Here, $E_{\mathbf{x},\theta}(\cdot)$, $E_{\mathbf{x}|\theta}(\cdot)$ and $E_\theta(\cdot)$ denote expectations with respect to the joint PDF $f_{\mathbf{x},\theta}$, the conditional PDF $f_{\mathbf{x}|\theta}$ and the prior PDF $f_\theta$, respectively.

**Theorem 1:** After the injection of added noise $\eta_n$ into $x_n$, the updated observation data $\tilde{x}_n = x_n + \eta_n = s(\theta) + \xi_n + \eta_n = s(\theta) + z_n$, and the updated BCRB is not less than the original one.

*Proof:* Letting $j_F$ be the Fisher information of one sample $x_n$, we have

$$ \begin{aligned} j_F(\theta) &= E_{\mathbf{x}|\theta} \left[ \left( \frac{\partial \ln f_{\mathbf{x}|\theta}(x|\theta)}{\partial \theta} \right)^2 \right] \\ &= E_{\mathbf{x}|\theta} \left[ \left( \frac{\partial \ln f_{\xi}(x-s(\theta))}{\partial \theta} \right)^2 \right] \\ &= E_{\xi} \left[ \left( \frac{\partial \ln f_{\xi}(x)}{\partial x} \right)^2 \right] \left( \frac{\partial s}{\partial \theta} \right)^2 = j_{\xi} \left( \frac{\partial s}{\partial \theta} \right)^2 \end{aligned} \quad (4) $$

with the Fisher information $j_\xi = E_\xi[(\partial \ln f_\xi(x)/\partial x)^2]$ of the PDF $f_\xi$. Then, for the independent identically distributed (i.i.d.) noise samples $\xi_n$, the Fisher information of the observation vector $\mathbf{x}$ is $J_F(\theta) = Nj_F(\theta)$. Similarly, the Fisher information of the updated data $\tilde{x}_n$ can be expressed as $\tilde{j}_F(\theta) = j_z(\partial s/\partial \theta)^2$ with the Fisher information $j_z = E_z[(\partial \ln f_z(z)/\partial z)^2]$ of the PDF $f_z(z) = \int f_\xi(z - \eta)f_\eta(\eta)d\eta$ of the composite noise $z_n$. Since the Fisher information quantities $j_z, j_\xi, j_\eta > 0$ satisfy the convolution inequality $j_z^{-1} \ge j_\xi^{-1} + j_\eta^{-1}$ [55], resulting in

$$ j_z \le j_\xi \left( \frac{j_\eta}{j_\xi + j_\eta} \right) \le j_\xi. \quad (5) $$

Thus, for i.i.d. noise samples $z_n$, the Fisher information of the updated data $\tilde{\mathbf{x}} = [\tilde{x}_1, \tilde{x}_2, \dots, \tilde{x}_N]^\top$ becomes $\tilde{J}_F(\theta) = N\tilde{j}_F(\theta) = Nj_z(\partial s/\partial \theta)^2$ and the updated Bayesian information satisfies

$$ \tilde{J}_B = E_\theta[\tilde{J}_F(\theta)] + J_\theta \leq J_B = E_\theta[J_F(\theta)] + J_\theta. \quad (6) $$

Substituting (6) into (2) proves Theorem 1.

**Theorem 1** only tells us the increase of the updated BCRB of the updated data vector $\tilde{\mathbf{x}}$. However, based on the MSE criterion and among all estimators, the minimum MSE $\mathcal{R}_{\text{ms}}$ is achieved by the MMSE estimator $\hat{\theta}_{\text{ms}}(\mathbf{x}) = E_{\theta|\mathbf{x}}(\theta|\mathbf{x}) = \int \theta f_{\mathbf{x}|\theta}(\mathbf{x}|\theta)f_\theta(\theta)d\theta / \int f_{\mathbf{x}|\theta}(\mathbf{x}|\theta)f_\theta(\theta)d\theta$ [51], [54]. Therefore, an interesting question is whether the updated MMSE estimator $\hat{\theta}_{\text{ms}}(\tilde{\mathbf{x}}) = E_{\theta|\tilde{\mathbf{x}}}(\theta|\tilde{\mathbf{x}})$ can achieve a lower MSE $\tilde{\mathcal{R}}_{\text{ms}}$ than that of the original MMSE estimator $\hat{\theta}_{\text{ms}}$ or not. The answer is given in Theorem 2.

**Theorem 2:** It is impossible to design an updated MMSE estimator $\hat{\theta}_{\text{ms}}(\tilde{\mathbf{x}})$ to achieve a lower MSE $\tilde{\mathcal{R}}_{\text{ms}}$ than the original MMSE estimator $\hat{\theta}_{\text{ms}}(\mathbf{x})$ does.

*Proof of Theorem 2* is presented in Appendix A. Although this theorem leads to a negative aspect of the added noise to the optimal MMSE estimator $\hat{\theta}_{\text{ms}}(\mathbf{x})$, it also indicates the possibility of noise benefits in some suboptimal estimators beyond the restricted conditions of [12], [20], [22]–[25], [27], [30], [31], [33], [36]–[43], [45]–[49]. In practice, the MMSE estimator $\hat{\theta}_{\text{ms}}(\mathbf{x})$ is usually too computationally intensive to implement [51], [54], thus we will exploit the optimal added noise in some easily implementable suboptimal estimators as follows.

<figure><img src="image_3.png" /><figcaption>**FIGURE 1.** Block diagram representations of (a) the noise-modified estimator $\hat{\theta}_{\text{NM}}$ in (7) and (b) the LMMSE estimator $\hat{\theta}_{\text{L}}$ in (13). The optimal noise $\eta$ is intentionally injected into a given sensor $g$ for the improvement of the MSE of the designed estimator.</figcaption></figure>

### III. NOISE BENEFITS IN SUBOPTIMAL ESTIMATORS

#### A. NOISE BENEFITS IN A NOISE-MODIFIED ESTIMATOR

Consider the scalar-parameter observation model $x = \theta + \xi$, and the observation $x$ plus the added noise $\eta$ is applied to a fixed sensor $g$, as shown in Fig. 1 (a). Then, the noise-modified estimator

$$ \hat{\theta}_{\text{NM}} = g(x + \eta) \quad (7) $$

is established on the updated sensor output $g(x + \eta)$. Then, the artificially added noise $\eta$ is optimized to minimize the MSE $\mathcal{R}_{\text{NM}} = E_{x,\eta}[(\hat{\theta}_{\text{NM}} - \theta)^2] = E_\theta(\theta^2) - E_\eta(E_x[2\theta g(x + \eta) - g^2(x + \eta)])$. Since the two-moment $E_\theta(\theta^2)$ is given, then Chen et al. [32] proved

$$ \begin{aligned} \min_{f_\eta} \mathcal{R}_{\text{NM}} &= E_\theta(\theta^2) - \max_{f_\eta} E_\eta\{E_x[2\theta g(x + \eta) - g^2(x + \eta)]\} \\ &\geq E_\theta(\theta^2) - \max_\eta E_x[2\theta g(x + \eta) - g^2(x + \eta)], \end{aligned} \quad (8) $$
<table><caption>TABLE 1. MSEs of various estimators with optimal added noise.</caption><thead><tr><th rowspan="2">Estimator</th><th colspan="2">Gaussian</th><th colspan="2">Rayleigh</th><th colspan="2">Laplace</th></tr><tr><th>MSE</th><th>f<sub>η</sub><sup>o</sup></th><th>MSE</th><th>f<sub>η</sub><sup>o</sup></th><th>MSE</th><th>f<sub>η</sub><sup>o</sup></th></tr></thead><tbody><tr><td>θ̂<sub>NM</sub>(x)</td><td>0.2571</td><td>/</td><td>0.3333</td><td>/</td><td>0.2643</td><td>/</td></tr><tr><td>θ̂<sub>NM</sub>(x̄)</td><td>0.1643</td><td>δ(η + 0.5)</td><td>0.1242</td><td>δ(η + 0.9)</td><td>0.1771</td><td>δ(η + 0.5)</td></tr><tr><td>θ̂<sub>L</sub>(x)</td><td>0.0701</td><td>/</td><td>0.0833</td><td>/</td><td>0.0741</td><td>/</td></tr><tr><td>θ̂<sub>L</sub>(x̄)</td><td>0.0547</td><td>δ(η + 0.5)</td><td>0.0396</td><td>δ(η + 0.9)</td><td>0.0589</td><td>δ(η + 0.5)</td></tr><tr><td>θ̂<sub>NE</sub>(x̄)</td><td>0.0448</td><td>Fig. 3(a)</td><td>0.0267</td><td>Fig. 3(b)</td><td>0.0536</td><td>Fig. 3(c)</td></tr><tr><td>θ̂<sub>LC</sub>(x̄)</td><td>0.0447</td><td>Fig. 3(d)</td><td>0.0266</td><td>Fig. 3(e)</td><td>0.0533</td><td>Fig. 3(f)</td></tr><tr><td>θ̂<sub>ms</sub>(x)</td><td>0.0446</td><td>/</td><td>0.0256</td><td>/</td><td>0.0533</td><td>/</td></tr></tbody></table>

and the optimal added noise accords with the PDF $f_{\eta}^{o}(\eta) = \delta(\eta - \eta^{*})$ and the constant [32]

$$ \eta^* = \arg\max_{\eta} E_x[2\theta g(x + \eta) - g^2(x + \eta)]. \quad (9) $$

Thus, with this optimal bias $\eta^*$, the MSE $\mathcal{R}_{NM}$ of the noise-modified estimator $\hat{\theta}_{NM}$ has a minimum

$$ \mathcal{R}_{NM}^{\min} = E_{\theta}(\theta^2) - E_x[2\theta g(x + \eta^*) - g^2(x + \eta^*)]. \quad (10) $$

Based on Theorem 2 and compared with the minimum MSEs achieved by the MMSE estimators $\hat{\theta}_{ms}(\bar{x})$ and $\hat{\theta}_{ms}(x)$, $\mathcal{R}_{NM}^{\min}$ in (10) satisfies

$$ \mathcal{R}_{NM}^{\min} \geq \bar{\mathcal{R}}_{ms} \geq \mathcal{R}_{ms}. \quad (11) $$

**Example 1:** Consider an uniformly distributed parameter $\theta$ with its PDF $f_{\theta}(\theta) = 1/a (a > 0)$ over the interval $(0, a)$ and zero otherwise. A quantizer sensor is given by

$$ g(u) = \begin{cases} 1, & u > \gamma, \\ 0, & u \le \gamma, \end{cases} \quad (12) $$

where $\gamma$ is the threshold value of the quantizer. In Table 1, consider three background noise types with Gaussian PDF $f_{\xi}(u) = \exp(-u^2/2\sigma_{\xi}^2)/\sqrt{2\pi\sigma_{\xi}^2}$, Rayleigh PDF $f_{\xi}(u) = (4 - \pi)u \exp[-(4-\pi)u^2/4\sigma_{\xi}^2]/2\sigma_{\xi}^2$ ($u \ge 0$), Laplace PDF $f_{\xi}(u) = \sqrt{2}\exp(-\sqrt{2}|u|/\sigma_{\xi})/2\sigma_{\xi}$ and the standard deviation $\sigma_{\xi} > 0$. Here, the interval bound $a = 1$, the quantizer threshold $\gamma = 0$ and the standard deviations $\sigma_{\xi}$ take $\sqrt{0.1}$, $\sqrt{(4-\pi)/20}$ and $1/\sqrt{5}$ for three types of considered background noise, respectively. It is seen in Table 1 that MSEs of $\hat{\theta}_{NM}(x)$ without the added noise are 0.2571, 0.3333 and 0.2643, respectively. With the optimal bias $\eta = \eta^*$ given in Table 1, the noise-modified quantizer $\hat{\theta}_{NM}(x + \eta^*)$ has the optimized MSEs of 0.1643, 0.1242 and 0.1771. However, compared with the MSEs of 0.0446, 0.0256 and 0.0533 achieved by the corresponding MMSE estimators $\hat{\theta}_{ms}(x)$, the improvement on the MSE of $\hat{\theta}_{NM}$ by the optimal added noise $\eta = \eta^*$ is limited.

## B. NOISE BENEFITS IN A LMMSE ESTIMATOR

In order to further reduce the MSE of the noise-modified estimator, we perform the LMMSE transform on the sensor

output $g(x + \eta)$ and establish a LMMSE estimator

$$ \hat{\theta}_{L} = wg(x + \eta) + w_0 \quad (13) $$

with adjustable weights $w$ and $w_0$, as shown in Fig.1 (b).

**Theorem 3:** For the LMMSE estimator $\hat{\theta}_{L}$ of (13), the optimal noise $\eta$ has the PDF $f_{\eta}^{o}(\eta) = \delta(\eta - \eta^{\dagger})$ with the constant

$$ \eta^{\dagger} = \arg\max_{\eta} \frac{\{E_x[\theta g(x + \eta)] - E_{\theta}(\theta)E_x[g(x + \eta)]\}^2}{E_x[g^2(x + \eta)] - E_x^2[g(x + \eta)]} \quad (14) $$

The MSE $\mathcal{R}_L$ of $\hat{\theta}_L$ has the minimum

$$ \begin{aligned} \mathcal{R}_{L}^{\min} &= \mathrm{var}(\theta) - \frac{\{E_x[\theta g(x + \eta^{\dagger})] - E_{\theta}(\theta)E_x[g(x + \eta^{\dagger})]\}^2}{E_x[g^2(x + \eta^{\dagger})] - E_x^2[g(x + \eta^{\dagger})]} \\ &\le \mathcal{R}_{NM}^{\min}, \end{aligned} \quad (15) $$

where $\mathrm{var}(\theta) = E_{\theta}(\theta^2) - E_{\theta}^2(\theta)$ is the variance of $\theta$.

Proof of Theorem 3 is given in Appendix B. It is shown in Table 1 that, without the added noise, the MSEs of the LMMSE estimator $\hat{\theta}_L(x)$ are 0.0701, 0.0833 and 0.0741 for three background noise types, respectively, which already are lower than that of the noise-modified estimator $\hat{\theta}_{NM}(x + \eta^*)$ with its optimal added noise $\eta^*$ in (9). Utilizing the optimal added noise $\eta^\dagger$ in (14), the MSEs of $\hat{\theta}_L(x + \eta^\dagger)$ can be further reduced to 0.0547, 0.0396 and 0.0589. However, the improved MSEs of $\hat{\theta}_L(x + \eta^\dagger)$ still cannot approach the MSE achieved by the MMSE estimator $\hat{\theta}_{ms}(x)$. Moreover, the optimal “noise” $\eta = \eta^\dagger$ is still a constant bias, rather than a bona fide random noise.

<figure><img src="image_4.png" /><figcaption>FIGURE 2. Block diagram representation of the noise-enhanced estimator $\hat{\theta}_{NE}$ in (16). Here, $M$ mutually i.i.d. noise components $\eta_m$ in sensors are optimized to minimize the MSE of $\hat{\theta}_{NE}$.</figcaption></figure>

## C. NOISE BENEFITS IN IDENTICAL SENSORS

The configurations of both noise-modified estimator $\hat{\theta}_{NM}$ and the LMMSE estimator $\hat{\theta}_L$ are only operated on one sensor. Next, consider an ensemble of identical sensors that receive the same input data $x$, and the mutually i.i.d. noise components $\eta_m$ are also fed into each sensor, as shown in Fig. 2. Here, $\eta_m$ are with the common PDF $f_{\eta}$ and satisfy $E_{\eta}(\eta_m\eta_k) = 0$ for $m \neq k$ ($m, k = 1, 2, \dots, M, M \ge 2$). Then, the average value of all outputs of sensors forms the noise-enhanced estimator

$$ \hat{\theta}_{NE} = \frac{1}{M} \sum_{m=1}^{M} g(x + \eta_m). \quad (16) $$
The MSE of $\hat{\theta}_{NE}$ can be computed as

$$ \begin{aligned} \mathcal{R}_{NE} &= E_{x,\eta}[(\theta - \hat{\theta}_{NE})^2] \\ &= \frac{E_x\{E_\eta[g^2(x+\eta)]\} + (M-1)E_x\{E_\eta^2[g(x+\eta)]\}}{M} \\ &\quad - 2E_x\{\theta E_\eta[g(x+\eta)]\} + E_\theta(\theta^2), \end{aligned} \quad (17) $$

where the correlations $E_{x,\eta}[\theta g(x+\eta_m)] = E_x\{\theta E_\eta[g(x+\eta)]\}$ ($\forall m$) and $E_x\{E_\eta[g(x+\eta_m)]E_\eta[\theta g(x+\eta_k)]\} = E_x\{E_\eta^2[g(x+\eta)]\}$ for $m \neq k$ ($m, k = 1, 2, \dots, M$).

**Theorem 4:** For minimizing the MSE $\mathcal{R}_{NE}$ of the noise-enhanced estimator $\hat{\theta}_{NE}$ by mutually i.i.d. noise components $\eta_m$, the optimal noise is not with the PDF $f_{\eta}^o(\eta) = \delta(\eta - \eta^\dagger)$ for a constant bias $\eta = \eta^\dagger$. For the given background noise $\xi$ and added noise components $\eta_m$, the MSE $\mathcal{R}_{NE}$ is a monotonically decreasing function of the sensor number $M$. Moreover, for a sufficiently large sensor number $M \to \infty$, the MSE satisfies

$$ \lim_{M \to \infty} \min_{f_{\eta}} \mathcal{R}_{NE} \le \mathcal{R}_{NM}^{\min}. \quad (18) $$

Proof of Theorem 4 is given in Appendix C. Interestingly, as the sensor number $M \to \infty$, (16) can be asymptotically represented as

$$ \hat{\theta}_{NE} = \lim_{M \to \infty} \frac{1}{M} \sum_{m=1}^{M} g(x + \eta_m) = E_{\eta}[g(x + \eta)], \quad (19) $$

which is just the noise-enhanced estimator proposed by Uhlich [42]. Theorem 4 only tells us that the optimal noise is not a constant bias, but which type of noise is optimal to minimizing the MSE of the noise-enhanced estimator $\hat{\theta}_{NE}$? This non-convex problem is in general intractable, because the term $E_x\{E_\eta^2[g(x+\eta)]\}$ in (17) is a nonlinear functional of the PDF $f_\eta$. Therefore, the minimization problem of the MSE $\min_{f_\eta} \mathcal{R}_{NE}$ usually employs the PDF approximation method [26], [42], [47], [48], [56] to obtain an approximate optimal solution form as

$$ \tilde{f}_{\eta}^{o}(\eta) = \sum_{k=1}^{K} \lambda_{k} \phi \left( \frac{\eta - \mu_{k}}{\sigma_{k}} \right) \quad (20) $$

with the normalization coefficients $\lambda_k \ge 0$ satisfying the constraint $\sum_{k=1}^K \lambda_k = 1$, and the Gaussian window function $\phi(u) = \exp(-u^2/2)/\sqrt{2\pi}$, means $\mu_k$, standard deviations $\sigma_k \ge 0$. The approximate PDF $\tilde{f}_{\eta}^o$ will asymptotically converge to the existing optimal PDF $f_{\eta}^o$ as the number $K$ of the window function increases [26], [42], [47], [48], [56].

**Example 2:** For instance, consider $M = 1000$ identical quantizers of (12) and other parameters are the same as in Example 1. The sequential quadratic programming [56] is used to numerically solve the approximate PDF $\tilde{f}_{\eta}^o$ of (20). In Figs. 3 (a), (b) and (c), the approximate PDFs $\tilde{f}_{\eta}^o$ are plotted for estimating the uniform distributed parameter $\theta$ buried in three background noise types, respectively. It is seen in Fig. 3 that the approximate optimal added noise PDF $\tilde{f}_{\eta}^o$ exhibits non-trivial complicated shapes and varies with the

<figure><img src="image_5.png" /><figcaption>**FIGURE 3.** Approximate PDFs $\tilde{f}_{\eta}^o(\eta)$ for the noise enhanced estimator $\hat{\theta}_{NE}$ with background noise types of (a) Gaussian, (b) Rayleigh and (c) Laplace distributions. For the linear combination estimator $\hat{\theta}_{LC}$ in (21), approximate PDFs $\tilde{f}_{\eta}^o(\eta)$ are also presented for (d) Gaussian, (e) Rayleigh and (f) Laplace background noise types. Here, the windows number $K = 10$ in (20).</figcaption></figure>

background noise types. These approximate PDFs $\tilde{f}_{\eta}^o$ implies a bona fide random noise, rather than a constant bias. Moreover, substituting the obtained approximate optimal noise PDFs $\tilde{f}_{\eta}^o$ into (17), the corresponding MSE values of $\mathcal{R}_{NE}(\bar{x})$ are reduced to 0.0448, 0.0267 and 0.0536, as listed in Table 1, which are almost equal to the corresponding MSEs achieved by the MMSE estimator $\hat{\theta}_{MS}(x)$.

## D. NOISE BENEFITS IN NONIDENTICAL SENSORS

From (16), the noise-enhanced estimator $\hat{\theta}_{NE}$ uniformly sets the same weight $1/M$ to the identical sensors $g$, which is inappropriate for an ensemble of nonidentical sensors $g_m$ shown in Fig. 4. Carrying out a LMMSE transform on the sensor outputs $g_m(x + \eta_m)$, a linear combination estimator is established as

$$ \hat{\theta}_{LC} = w_0 + \mathbf{w}^{\top}\mathbf{g}, \quad (21) $$

where the sensor output vector $\mathbf{g} = [g_1(x + \eta_1), g_2(x + \eta_2), \dots, g_M(x + \eta_M)]^\top$, the weight vector $\mathbf{w} = [w_1, w_2, \dots, w_M]^\top$ and $w_0$ is the bias weight [48]. Then, the MSE of $\hat{\theta}_{LC}$ can be expressed as

$$ \mathcal{R}_{LC} = E_{x,\eta}[(\theta - \hat{\theta}_{LC})^2] = E_{x,\eta}[(\theta - w_0 - \mathbf{w}^{\top}\mathbf{g})^2]. \quad (22) $$

An interesting fact of $\mathcal{R}_{LC}$ in (22) is that the minimization of $\mathcal{R}_{LC}$ with respect to weights, uncoupled with the minimization of $\mathcal{R}_{LC}$ with respect to the added noise, can be first theoretically solved. Setting the derivative $\partial \mathcal{R}_{LC}/\partial w_0 = 0$ and the gradient $\partial \mathcal{R}_{LC}/\partial \mathbf{w} = \mathbf{0}$ produce $w_0 = E_\theta(\theta) - \mathbf{w}^\top E_{x,\eta}(\mathbf{g})$ and $\mathbf{w} = \mathbf{C}^{-1}\mathbf{p}$, where $\mathbf{p} = E_{x,\eta}\{[\theta - E_\theta(\theta)][\mathbf{g} - E_{x,\eta}(\mathbf{g})]\}$ is the cross-correlation vector of the parameter $\theta$ and the sensor
<figure><img src="image_6.png" /><figcaption>FIGURE 4. Block diagram representation of the linear combination estimator $\hat{\theta}_{LC}$ in (21). Besides the injection of mutually independent added noise components $\eta_m$, each sensor is also endowed with an adjustable weight $w_m$.</figcaption></figure>

vector $\mathbf{g}$ and $\mathbf{C} = E_{x,\eta}([\mathbf{g} - E_{x,\eta}(\mathbf{g})][\mathbf{g} - E_{x,\eta}(\mathbf{g})]^{\top}]$ is the covariance matrix of $\mathbf{g}$ [48], [57]. Then, using the optimal weights $w_0$ and $\mathbf{w}$, the linear combination estimator $\hat{\theta}_{LC}$ in the LMMSE sense can be rewritten as

$$ \hat{\theta}_{LC} = E_{\theta}(\theta) + \mathbf{p}^{\top}\mathbf{C}^{-1}[\mathbf{g} - E_{x,\eta}(\mathbf{g})] \quad (23) $$

and the minimized MSE $\mathcal{R}_{LC}$ with respect to weights is given by

$$ \mathcal{R}_{LC} = \text{var}(\theta) - \mathbf{p}^{\top}\mathbf{C}^{-1}\mathbf{p}. \quad (24) $$

**Theorem 5:** The linear combination estimator $\hat{\theta}_{LC}$ is never worse than the noise-enhanced estimator $\hat{\theta}_{NE}$ in (16) in the same environment, and the MSEs satisfy $\mathcal{R}_{LC} \le \mathcal{R}_{NE}$.

Theorem 5 is proved in Appendix D, and is illustrated in the following examples.

**Example 3:** Consider again Example 1 and minimize the MSE $\mathcal{R}_{LC}$ in (24) by optimizing the added noise. For $M = 1000$ identical quantizers $g_m = g$ of (12) with the same threshold $\gamma = 0$, the approximate noise PDFs $\tilde{f}_{\eta}^o$ of the optimization problem of $\min_{f_{\eta}} \mathcal{R}_{LC}$ are also numerically solved for three background noise types, as shown in Fig. 3 (d), (e) and (f), respectively. The corresponding MSE values of $\mathcal{R}_{LC}$ are 0.0447, 0.0266 and 0.0533, as listed in Table 1, which also approach to the MSE achieved by the MMSE estimator $\hat{\theta}_{ms}$. However, compared to the noise-enhanced estimator $\hat{\theta}_{NE}$, $\hat{\theta}_{LC}$ improves the MSE slightly in estimating parameters from the observations of a large number of identical sensors.

**Example 4:** Consider $M$ (even number) quantizers $g_m$ with two groups: one group of $M/2$ quantizers has the same threshold $\gamma_1 = 0$, and the other group of $M/2$ quantizers is with the same threshold $\gamma_2 = 1$. The background noise $\xi$ is selected as Gaussian distributed with the zero-mean and standard deviation $\sigma_{\xi} = \sqrt{0.1}$, and other conditions are the same as in Example 1. The MSEs of estimators $\hat{\theta}_{NE}$ and $\hat{\theta}_{LC}$ are plotted as a function of the sensor number $M$ in Fig. 5, wherein the MSE is minimized with respect to the added noise by the approximate PDF in (20). It is seen in Fig. 5 that, upon increasing the sensor number $M$ and dividing these

<figure><img src="image_7.png" /><figcaption>FIGURE 5. MSEs of the linear combination estimator $\hat{\theta}_{LC}$ in (21) and the noise-enhanced estimator $\hat{\theta}_{NE}$ in (16) versus the sensor number $M$.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>FIGURE 6. Approximate PDF $\tilde{f}_{\eta}^o(\eta)$ for the linear combination estimator $\hat{\theta}_{LC}$ in (21) with two groups of nonidentical sensors $g_m$. Here, the number of sensors is $M = 200$, one group of $M/2$ quantizers has the threshold $\gamma_1 = 0$, and the other group of $M/2$ quantizers is with the threshold $\gamma_2 = 1$. The background Gaussian noise $\xi$ is with the zero-mean and standard deviation $\sigma_{\xi} = \sqrt{0.1}$. The windows number $K = 10$ in (20).</figcaption></figure>

sensors into two groups, the MSE of the linear combination estimator $\hat{\theta}_{LC}$ (□) still approaches 0.0446 achieved by the MMSE estimator $\hat{\theta}_{ms}$ (dashed line) asymptotically. While, under the same condition, the MSE of the noise enhanced estimator $\hat{\theta}_{NE}$ (○) approaches 0.0715 asymptotically at large $M$, rather than 0.0446. The reason is that the linear combination estimator $\hat{\theta}_{LC}$ allocates different weights to sensors $g_m$ with different thresholds. For instance, for $M = 200$, the weights $w_m = 3.0976 \times 10^{-3}$ ($m = 1, 2, \dots, 100$) for the group of quantizers with threshold $\gamma_1$, and the weights $w_k = 5.1738 \times 10^{-3}$ ($k = 1, 2, \dots, 100$) for the group of quantizers with threshold $\gamma_2$. But the estimator $\hat{\theta}_{NE}$ in (16) always endows all sensors with a fixed weight $1/M$, regardless of the distinction of thresholds for two groups of sensors. For comparison, the MSEs of both $\hat{\theta}_{LC}$ (▷) and $\hat{\theta}_{NE}$ (*) are also plotted in Fig. 5 for $M$ identical sensors $g$ with threshold $\gamma_1 = 0$, which also all approach the MMSE value of 0.0446 as the number $M$ increases. For a moderate sensor number (e.g. $2 \le M \le 10^2$), the MSE $\mathcal{R}_{LC}$ (▷) of $\hat{\theta}_{LC}$ also clearly outperforms the MSE $\mathcal{R}_{NE}$ (*) of $\hat{\theta}_{NE}$. These points indicate
the superiority of the linear combination estimator $\hat{\theta}_{LC}$ over the noise enhanced estimator $\hat{\theta}_{NE}$ for Bayesian parameter estimation. In addition, for the total number $M = 200$ of sensors with two groups, the approximate noise PDFs $\hat{f}_{\eta}^0$ that minimizes the MSE $\mathcal{R}_{LC}$ to 0.0451 is also numerically solved and shown in Fig. 6, which also has quite a complicated structure. For other threshold settings of $\gamma$ (not shown here), the superiority of the linear combination estimator is also confirmed.

## IV. CONCLUSION

In this paper, added noise is intentionally injected into the observed data, we first theoretically address a crucial question of the increase of the BCRB of the updated observations, and then prove that the updated MMSE estimator yet cannot provide a lower MSE than that of the original MMSE estimator. We mainly investigate the noise benefits in certain types of suboptimal Bayesian estimators that are widely employed due to their ease of implementation and low cost. For the noise-modified estimator in (7) or the LMMSE estimator in (13) based on a single sensor, the optimal added noise that minimizes the estimator MSE is just a constant bias, but not bona fide random noise. However, for the noise-enhanced estimator in (16) and the linear combination estimator in (21) established on an ensemble of sensors, it is observed that the optimal noise that improves the estimator and makes it as efficient as the original MMSE estimator is a bona fide random signal, rather than a constant bias. Especially, for an ensemble of two groups sensors with different settings, the linear combination estimator in (21), benefiting from the optimal added noise, can still approach the MSE of the original MMSE estimator when the sensor number is sufficiently large.

Some open questions remain. For instance, it is seen in (23) that the construction of the linear combination estimator $\hat{\theta}_{LC}$ requires the theoretical two-moments of the cross-correlation between the parameter and the sensor outputs and the covariance of the sensor outputs. To find these statistical quantities we need the joint PDF of the observation data and the added noise, of which, most of time, we have no knowledge of the probabilistic structure. Thus, under this circumstance, how do we establish a practical estimator? If the observation data is stationary and ergodic, can we approximately compute these desired statistical characteristics from one sampling realization of data? Or on the minimization of the least squares error criterion, how do we establish an easily implementable least-square estimator and which kind of added noise is optimal for improving the MSE of the least-square estimator? In addition, in many signal estimation problems, the observations are obtained in a sequential order as time processes. So, can we present a sequential linear combination estimator in the LMMSE sense that continuously update weights and the added noise according to the new incoming data? These interesting questions deserve the further study.

# APPENDIXES

## APPENDIX A

### PROOF OF THEOREM 2

Consider the case of $s(\theta) = \theta$ without loss of generality. For the updated observation data $\bar{x}_n = \theta + \xi_n + \eta_n = \theta + z_n$, the joint PDF of the random parameter $\theta$ and the data vector $\bar{x}$ is described as $f_{\bar{x},\theta}(\bar{x}, \theta) = f_{\bar{x}|\theta}(\bar{x}|\theta)f_{\theta}(\theta)$, where the conditional PDF can be expressed as $f_{\bar{x}|\theta}(\bar{x}|\theta) = \prod_{n=1}^N f_z(\bar{x}_n - \theta) = \prod_{n=1}^N \int f_{\xi}(\bar{x}_n - \theta - \eta_n)f_{\eta}(\eta_n)d\eta_n = \int f_{\xi}(\bar{x} - \theta - \eta)f_{\eta}(\eta)d\eta$. Then, the updated MMSE estimator is given by $\hat{\vartheta}_{ms}(\bar{x}) = E_{\bar{x},\theta}[\theta|\bar{x}] = \int \theta f_{\theta|\bar{x}}(\theta|\bar{x})d\theta$ with the conditional posterior PDF $f_{\theta|\bar{x}}(\theta|\bar{x}) = f_{\bar{x},\theta}(\bar{x}, \theta)/f_{\bar{x}}(\bar{x})$ and the PDF $f_{\bar{x}}(\bar{x}) = \int f_{\bar{x},\theta}(\bar{x}, \theta)d\theta$. The MSE of $\hat{\vartheta}_{ms}(\bar{x})$ is given by $\bar{\mathcal{R}}_{ms} = E_{\bar{x},\theta}[(\theta - \hat{\vartheta}_{ms}(\bar{x}))^2] = E_{\theta}[\theta^2] - E_{\bar{x}}[\hat{\vartheta}_{ms}(\bar{x})^2]$. Since the two-moment $E_{\theta}[\theta^2]$ is given, then we now find the optimal added noise vector $\eta$ to maximize the term $E_{\bar{x}}[\hat{\vartheta}_{ms}(\bar{x})^2]$.

Consider a real-value vector function $f(Z) = Z_1^2/Z_2$ with $Z = [Z_1, Z_2]^T$ and $Z_2 > 0$. Since its Hessian matrix $\nabla^2 f(Z) = 2[Z_2, -Z_1]^T[Z_2, -Z_1]/Z_2^3$ is positive semidefinite, then $f(Z)$ is convex [33], [46], [58]. Thus, Jensen's inequality $E_{\eta}^2[Z_1]/E_{\eta}[Z_2] \le E_{\eta}[Z_1^2/Z_2]$ indicates that $E_{\bar{x}}[\hat{\vartheta}_{ms}(\bar{x})^2]$ satisfies

$$ 
\begin{aligned}
E_{\bar{x}}[\hat{\vartheta}_{ms}(\bar{x})^2] &= \int \frac{[\int \theta f_{\theta|\bar{x}}(\theta|\bar{x})f_{\theta}(\theta)d\theta]^2}{f_{\bar{x}}(\bar{x})} d\bar{x} \\
&= \int \frac{E_{\eta}^2\{E_{\theta}[\theta f_{\xi}(\bar{x} - \theta - \eta)]\}}{E_{\eta}\{E_{\theta}[f_{\xi}(\bar{x} - \theta - \eta)]\}} d\bar{x} \\
&\le \int E_{\eta} \left\{ \frac{E_{\theta}^2[\theta f_{\xi}(\bar{x} - \theta - \eta)]}{E_{\theta}[f_{\xi}(\bar{x} - \theta - \eta)]} \right\} d\bar{x} \quad (25) \\
&= E_{\eta} \left[ \int \frac{E_{\theta}^2[\theta f_{\xi}(\bar{x} - \theta - \eta)]}{E_{\theta}[f_{\xi}(\bar{x} - \theta - \eta)]} d\bar{x} \right] \\
&\le \max_{\eta} \int \frac{E_{\theta}^2[\theta f_{\xi}(\bar{x} - \theta - \eta)]}{E_{\theta}[f_{\xi}(\bar{x} - \theta - \eta)]} d\bar{x} \\
&= \max_{\eta} p(\eta), \quad (26)
\end{aligned}
$$

where the inequality (25) accords to the Jensen's inequality [33], [46], [58] and (26) holds for $f_{\eta}^0(\eta) = \delta(\eta - \eta^*)$ with the constant vector $\eta^* = \arg\max_{\eta} p(\eta)$. Furthermore, from (26) and noting $d\bar{x} = d(\bar{x} - \eta^*)$, we find

$$ 
\begin{aligned}
E_{\bar{x}}[\hat{\vartheta}_{ms}(\bar{x})^2] &\le \int \frac{E_{\theta}^2[\theta f_{\xi}(\bar{x} - \theta - \eta^*)]}{E_{\theta}[f_{\xi}(\bar{x} - \theta - \eta^*)]} d\bar{x} \\
&= \int \frac{E_{\theta}^2[\theta f_{\xi}(\bar{x} - \theta)]}{E_{\theta}[f_{\xi}(\bar{x} - \theta)]} d\bar{x} \\
&= \int \frac{E_{\theta}^2[\theta f_{\xi}(\bar{x} - \theta)]}{f_{\bar{x}}^2(\bar{x})} f_{\bar{x}}(\bar{x}) d\bar{x} \\
&= \int \left( \int \theta f_{\bar{x}|\theta}(\bar{x}|\theta) \right)^2 f_{\bar{x}}(\bar{x}) d\bar{x} \\
&= E_{\bar{x}}[\hat{\vartheta}_{ms}(\bar{x})^2]. \quad (27)
\end{aligned}
$$

Thus, we find $\bar{\mathcal{R}}_{ms} = E_{\theta}[\theta^2] - E_{\bar{x}}[\hat{\vartheta}_{ms}(\bar{x})^2] \ge E_{\theta}[\theta^2] - E_{\bar{x}}[\hat{\vartheta}_{ms}(\bar{x})^2] = \mathcal{R}_{ms}$. Theorem 2 holds.
## APPENDIX B
### PROOF OF THEOREM 3

The MSE of $\hat{\theta}_L(x + \eta)$ in (13) can be written as $\mathcal{R}_L = E_{x,\eta}[(\theta - wg(x + \eta) - w_0)^2]$. Setting the derivative $\partial R_L/\partial w_0 = 0$, the optimum biasing weight $w_0$ is solved as $w_0 = E_\theta(\theta) - wE_{x,\eta}[g(x + \eta)]$, which happens to make $\hat{\theta}_L$ unbiased as $E_{x,\eta}(\hat{\theta}_L) = E_\theta(\theta)$. Substituting $w_0$ into $\mathcal{R}_L$ and setting $\partial R_L/\partial w = 0$, we have

$$w = \frac{E_{x,\eta}[\tilde{\theta}\tilde{g}(x + \eta)]}{E_{x,\eta}[\tilde{g}^2(x + \eta)]} \quad (28)$$

with $\tilde{g}(x+\eta) = g(x+\eta) - E_x\{E_\eta[g(x+\eta)]\}$ and $\tilde{\theta} = \theta - E_\theta(\theta)$. Subsequently, with the solved weight $w$ in (28), the MSE of $\hat{\theta}_L$ can be rewritten as

$$\mathcal{R}_L = \text{var}(\theta) - \frac{E_{x,\eta}^2[\tilde{\theta}\tilde{g}(x + \eta)]}{E_{x,\eta}[\tilde{g}^2(x + \eta)]} \quad (29)$$

Since the variance $\text{var}(\theta)$ of $\theta$ is given, then we maximize the second term of (29) as

$$\begin{aligned}
\frac{E_{x,\eta}^2[\tilde{\theta}\tilde{g}(x + \eta)]}{E_{x,\eta}[\tilde{g}^2(x + \eta)]} &= \frac{E_{x,\eta}^2[\tilde{\theta}\tilde{g}(x + \eta)]}{E_{x,\eta}[g^2(x + \eta)] - E_{x,\eta}^2[g(x + \eta)]} \\
&\le \frac{E_\eta^2\{E_x[\tilde{\theta}\tilde{g}(x + \eta)]\}}{E_\eta\{E_x[g^2(x + \eta)]\} - E_\eta\{E_x^2[g(x + \eta)]\}} \quad (30) \\
&= \frac{E_\eta^2\{E_x[\tilde{\theta}\tilde{g}(x + \eta)]\}}{E_\eta\{E_x[\tilde{g}^2(x + \eta)]\}} \\
&\le E_\eta\left\{\frac{E_x^2[\tilde{\theta}\tilde{g}(x + \eta)]}{E_x[\tilde{g}^2(x + \eta)]}\right\} \quad (31) \\
&\le \max_{\eta} \frac{E_\eta^2[\tilde{\theta}\tilde{g}(x + \eta)]}{E_x[\tilde{g}^2(x + \eta)]}.
\end{aligned}$$

Using Jensen's inequality [33], [46], [55], [58], the inequality of (30) holds due to $E_{x,\eta}^2[g(x + \eta)] = E_\eta^2\{E_x[g(x + \eta)]\} \le E_\eta\{E_x^2[g(x + \eta)]\}$ based on the convex function $x^2$, and the inequality of (31) is valid for the convex $f(\mathbf{Z}) = Z_1^2/Z_2$ given in Appendix A. Then, we find the optimal noise PDF $f_\eta^0(\eta) = \delta(\eta - \eta^\dagger)$ in (14) and the minimum MSE in (15). Furthermore, the minimum MSE $\mathcal{R}_L^{\min}$ of the estimator $\hat{\theta}_L$ also satisfies

$$\begin{aligned}
\mathcal{R}_L^{\min} &= \min_{w,f_\eta} E_{\tilde{x},\theta}[(\theta - E_\theta(\theta) - wg(x + \eta) + wE_{x,\eta}[g(x + \eta)])^2] \\
&\le \min_{f_\eta} E_{\tilde{x},\theta}[(\theta - E_\theta(\theta) - g(x + \eta) + E_{x,\eta}[g(x + \eta)])^2] \\
&\le E_{x,\theta}[(\theta - E_\theta(\theta) - g(x + \eta^*) + E_x[g(x + \eta^*)])^2] \\
&= \mathcal{R}_{\text{NM}}^{\min} - (E_\theta(\theta) - E_x[g(x + \eta^*)])^2 \\
&\le \mathcal{R}_{\text{NM}}^{\min}.
\end{aligned} \quad (32)$$

where the constant $\eta^*$ is given in (9). Then, Theorem 3 holds.

## APPENDIX C
### PROOF OF THEOREM 4

If the optimal noise $\eta$ has the PDF $f_\eta^0(\eta) = \delta(\eta - \eta^\dagger)$, then $M$ equivalent constants $\eta_m = \eta^\dagger$ are added to $M$ identical sensors $g$ with the same outputs $g(x + \eta^\dagger)$. Thus, the noise enhanced estimator $\hat{\theta}_{\text{NE}} = \sum_{m=1}^M g(x + \eta^\dagger)/M = g(x + \eta^\dagger)$ reduces to the output of a single sensor. Moreover, $E_\eta(\eta_m\eta_k) = (\eta^\dagger)^2 \neq 0$ does not satisfy the mutually independent assumption of $\eta_m$. Therefore, the optimal noise must not be a constant bias. Using the Jensen inequality and $E_\eta[g(x + \eta)] \neq g(x + \eta)$ for the given observation data $x$, we obtain the inequality $E_\eta[g^2(x + \eta)] > E_\eta^2[g(x + \eta)]$. Then, we have

$$E_x\{E_\eta[g^2(w + \eta)]\} > E_x\{E_\eta^2[g(w + \eta)]\}. \quad (33)$$

Immediately, we find

$$\frac{E_x[E_\eta(g^2(x+\eta))]+(M-1)E_x[E_\eta^2(g(x+\eta))]}{E_x[E_\eta(g^2(x+\eta))]+ME_xE_\eta^2(g(x+\eta))}>\frac{M}{M+1}. \quad (34)$$

From (34), we deduce that $\mathcal{R}_{\text{NE}}$ in (17) is a monotonically decreasing function of the sensor number $M$, when the observation data $x$ and the added noise are given. Furthermore, for a sufficiently large number $M \to \infty$, the MSE $\mathcal{R}_{\text{NE}}$ in (17) can be simplified as

$$\begin{aligned}
\lim_{M \to \infty} \min_{f_\eta} \mathcal{R}_{\text{NE}} &= \min_{f_\eta} \left(E_x[E_\eta^2(g(x + \eta))\right. \\
&\qquad \left. - 2E_x[\theta E_\eta[g(x + \eta)]] + E_\theta(\theta^2)\right) \\
&= \min_{f_\eta} E_x\{[\theta - E_\eta(g(x + \eta))]^2\} \\
&\le E_x\{[\theta - g(x + \eta^*)]^2\} = \mathcal{R}_{\text{NM}}^{\min}, \quad (35)
\end{aligned}$$

where the constant $\eta^*$ is given in (9). Then, Theorem 4 is proved.

## APPENDIX D
### PROOF OF THEOREM 5

For an ensemble of sensors $g_m$, the noise-enhanced estimator can be rewritten as

$$\hat{\theta}_{\text{NE}} = \frac{1}{M} \sum_{m=1}^{M} g_m(x + \eta_m) \quad (36)$$

and its minimum MSE can be expressed as

$$\begin{aligned}
\mathcal{R}_{\text{NE}}^{\min} &= \min_{f_\eta} E_{x,\eta} \left[ \left( \theta - \hat{\theta}_{\text{NE}} \right)^2 \right] \\
&= E_{x,\eta} \left\{ \left[ \theta - \frac{1}{M} \sum_{m=1}^{M} g_m(x + \eta_m) \right]^2 \right\} \Big|_{f_\eta = f_\eta^{\circ *}} \quad (37)
\end{aligned}$$

with the optimal added noise PDF $f_\eta^{\circ*}$. However, for any PDF $f_\eta$ of the added noise and the $M \times 1$ dimensional vector $\mathbf{1}$ of all ones, the MSE of the designed linear combination estimator in (23) can be expressed as

$$\begin{aligned}
\mathcal{R}_{\text{LC}} &= \min_{w_0, \mathbf{w}} E_{x,\eta}[(\theta - w_0 - \mathbf{w}^\top \mathbf{g})^2] \\
&\le E_{x,\eta}\left\{\left[\theta - \frac{1}{M}\sum_{m=1}^{M} g_m(x + \eta_m)\right]^2\right\}\Big|_{w_0=0, \mathbf{w}=\mathbf{1}/M} \\
&= \mathcal{R}_{\text{NE}}.
\end{aligned} \quad (38)$$
Of course, even when the added noise PDF $f_{\eta}$ receives the expression $f_{\eta}^{*}$ that is optimal for the noise-enhanced estimator $\hat{\theta}_{NE}$, the inequality (38) also holds, resulting in $R_{LC}|_{f_{\eta}=f_{\eta}^{*}} \le R_{NE}^{\min}$. Thus, Theorem 5 holds.

## REFERENCES

[1] L. Gammaitoni, “Stochastic resonance and the dithering effect in threshold physical systems,” Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top., vol. 52, no. 5, pp. 4691–4698, Jul. 2002.

[2] B. Ando and S. Graziani, “Adding noise to improve measurement,” IEEE Instrum. Meas. Mag., vol. 4, no. 1, pp. 24–31, Mar. 2001.

[3] S. Engelberg, “An introduction to dither,” IEEE Instrum. Meas. Mag., vol. 15, no. 6, pp. 50–54, Dec. 2012.

[4] L. Roberts, “Picture coding using pseudo-random noise,” IRE Trans. Inf. Theory, vol. 8, no. 2, pp. 145–154, Feb. 1962.

[5] L. Schuchman, “Dither signals and their effect on quantization noise,” IEEE Trans. Commun., vol. 12, no. 4, pp. 162–165, Dec. 1964.

[6] R. Benzi, A. Sutera, and A. Vulpiani, “The mechanism of stochastic resonance,” J. Phys. A, Math. Gen., vol. 14, no. 11, pp. L453–L457, 1981.

[7] L. Gammaitoni, P. Hänggi, P. Jung, and F. Marchesoni, “Stochastic resonance,” Rev. Mod. Phys., vol. 70, no. 1, pp. 233–287, 1998.

[8] V. S. Anishchenko, A. B. Neiman, F. Moss, and L. Shimansky-Geier, “Stochastic resonance: Noise-enhanced order,” Uspekhi Fizicheskikh Nauk, vol. 42, no. 1, pp. 7–36, 1999.

[9] G. Harmer, B. Davis, and D. Abbott, “A review of stochastic resonance: Circuits and measurement,” IEEE Trans. Instrum. Meas., vol. 51, no. 2, pp. 299–309, Apr. 2002.

[10] F. Moss, “Stochastic resonance and sensory information processing: A tutorial and review of application,” Clin. Neurophysiol., vol. 115, no. 2, pp. 267–281, Feb. 2004.

[11] L. M. Ward, “Physics of neural synchronisation mediated by stochastic resonance,” Contemp. Phys., vol. 50, no. 5, pp. 563–574, Sep. 2009.

[12] M. D. McDonnell, N. G. Stocks, C. E. M. Pearce, and D. Abbott, Stochastic Resonance: From Suprathreshold Stochastic Resonance To Stochastic Signal Quantization. New York, NY, USA: Cambridge Univ. Press, 2008.

[13] M. D. McDonnell and D. Abbott, “What is stochastic resonance? Definitions, misconceptions, debates, and its relevance to biology,” PLoS Comput. Biol., vol. 5, no. 5, May 2009, Art. no. e1000348.

[14] M. D. McDonnell, “Is electrical noise useful?” Proc. IEEE, vol. 99, no. 2, pp. 242–246, Feb. 2011.

[15] Z. Qiao, Y. Lei, and N. Li, “Applications of stochastic resonance to machinery fault detection: A review and tutorial,” Mech. Syst. Signal Process., vol. 122, pp. 502–536, May 2019.

[16] S. Lu, Q. He, and J. Wang, “A review of stochastic resonance in rotating machine fault detection,” Mech. Syst. Signal Process., vol. 116, pp. 230–260, Feb. 2019.

[17] O. White, J. Babić, C. Trenado, L. Johannsen, and N. Goswami, “The promise of stochastic resonance in falls prevention,” Frontiers Physiol., vol. 9, p. 1865, Jan. 2019.

[18] J. J. Collins, C. C. Chow, and T. T. Imhoff, “Stochastic resonance without tuning,” Nature, vol. 376, no. 6537, pp. 236–238, Jul. 1995.

[19] J. J. Collins, C. C. Chow, and T. T. Imhoff, “Aperiodic stochastic resonance in excitable systems,” Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top., vol. 52, no. 4, pp. R3321–R3324, Jul. 2002.

[20] N. G. Stocks, “Suprathreshold stochastic resonance in multilevel threshold systems,” Phys. Rev. Lett., vol. 84, no. 11, pp. 2310–2314, Jul. 2002.

[21] F. Chapeau-Blondeau and X. Godivier, “Theory of stochastic resonance in signal transmission by static nonlinear systems,” Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top., vol. 55, no. 2, pp. 1478–1495, Jul. 2002.

[22] A. B. Gershman and J. F. Böhme, “A pseudo-noise approach to direction finding,” Signal Process., vol. 71, no. 1, pp. 1–13, Nov. 1998.

[23] H. Papadopoulos, G. Wornell, and A. Oppenheim, “Sequential signal encoding from noisy measurements using quantizers with dynamic bias control,” IEEE Trans. Inf. Theory, vol. 47, no. 3, pp. 978–1002, Mar. 2001.

[24] F. Chapeau-Blondeau, “Noise-aided nonlinear Bayesian estimation,” Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top., vol. 66, no. 3, 2002, Art. no. 032101.

[25] F. Chapeau-Blondeau and D. Rousseau, “Noise-enhanced performance for an optimal Bayesian estimator,” IEEE Trans. Signal Process., vol. 52, no. 5, pp. 1327–1334, May 2004.

[26] S. Bayram, S. Gezici, and H. V. Poor, “Noise enhanced hypothesis-testing in the restricted Bayesian framework,” IEEE Trans. Signal Process., vol. 58, no. 8, pp. 3972–3989, Aug. 2010.

[27] S. Zozor and P.-O. Amblard, “Noise-aided processing: Revisting dithering in a $\Sigma\Delta$ quantizer,” IEEE Trans. Signal Process., vol. 53, no. 8, pp. 3202–3210, Aug. 2005.

[28] H. Chen, P. K. Varshney, S. M. Kay, and J. H. Michels, “Theory of the stochastic resonance effect in signal detection: Part I—fixed detectors,” IEEE Trans. Signal Process., vol. 55, no. 7, pp. 3172–3184, Jul. 2007.

[29] H. Chen and P. Varshney, “Theory of the stochastic resonance effect in signal detection—Part II: Variable detectors,” IEEE Trans. Signal Process., vol. 56, no. 10, pp. 5031–5041, Oct. 2008.

[30] D. Rousseau and F. Chapeau-Blondeau, “Noise-improved Bayesian estimation with arrays of one-bit quantizers,” IEEE Trans. Instrum. Meas., vol. 56, no. 6, pp. 2658–2662, Dec. 2007.

[31] F. Chapeau-Blondeau and D. Rousseau, “Raising the noise to improve performance in optimal processing,” J. Stat. Mech. Theory Exp., vol. 53, no. 1, pp. 215–238, 2009.

[32] H. Chen, P. Varshney, and J. Michels, “Noise enhanced parameter estimation,” IEEE Trans. Signal Process., vol. 56, no. 10, pp. 5074–5081, Oct. 2008.

[33] G. O. Balkan and S. Gezici, “CRLB based optimal noise enhanced parameter estimation using quantized observations,” IEEE Signal Process. Lett., vol. 17, no. 5, pp. 477–480, May 2010.

[34] S. Kay, “Can detectability be improved by adding noise?” IEEE Signal Process. Lett., vol. 7, no. 1, pp. 8–10, Jan. 2000.

[35] S. Zozor and P. Amblard, “Stochastic resonance in locally optimal detectors,” IEEE Trans. Signal Process., vol. 51, no. 12, pp. 3177–3181, Dec. 2003.

[36] A. Patel and B. Kosko, “Optimal noise benefits in Neyman-Pearson and inequality-constrained statistical signal detection,” IEEE Trans. Signal Process., vol. 57, no. 5, pp. 1655–1669, May 2009.

[37] A. Patel and B. Kosko, “Optimal mean-square noise benefits in quantizer-array linear estimation,” IEEE Signal Process. Lett., vol. 17, no. 12, pp. 1005–1009, Dec. 2010.

[38] H. Chen and P. Varshney, “Performance limit for distributed estimation systems with identical one-bit quantizers,” IEEE Trans. Signal Process., vol. 58, no. 1, pp. 466–471, Jan. 2010.

[39] H. Chen, L. R. Varshney, and P. K. Varshney, “Noise-enhanced information systems,” Proc. IEEE, vol. 102, no. 10, pp. 1607–1621, Oct. 2014.

[40] J. Zhu, X. Lin, R. S. Blum, and Y. Gu, “Parameter estimation from quantized observations in multiplicative noise environments,” IEEE Trans. Signal Process., vol. 63, no. 15, pp. 4037–4050, Aug. 2015.

[41] H. Soganci, S. Gezici, and O. Arikan, “Optimal signal design for multi-parameter estimation problems,” IEEE Trans. Signal Process., vol. 63, no. 22, pp. 6074–6085, Nov. 2015.

[42] S. Uhlich, “Bayes risk reduction of estimators using artificial observation noise,” IEEE Trans. Signal Process., vol. 63, no. 20, pp. 5535–5545, Oct. 2015.

[43] A. B. Akbay and S. Gezici, “Noise benefits in joint detection and estimation problems,” Signal Process., vol. 118, pp. 235–247, Jan. 2016.

[44] H. Dong, H. Wang, X. Shen, and Z. Jiang, “Effects of second-order matched stochastic resonance for weak signal detection,” IEEE Access, vol. 6, pp. 46505–46515, 2018.

[45] Q. Zhai and Y. Wang, “Noise effect on signal quantization in an array of binary quantizers,” Signal Process., vol. 152, pp. 265–272, Nov. 2018.

[46] Y. Pan, F. Duan, F. Chapeau-Blondeau, and D. Abbott, “Noise enhancement in robust estimation of location,” IEEE Trans. Signal Process., vol. 66, no. 8, pp. 1953–1966, Apr. 2018.

[47] T. Yang, S. Liu, H. Liu, M. Tang, X. Tan, and X. Zhou, “Noise benefits parameter estimation in LMMSE sense,” Digit. Signal Process., vol. 73, pp. 153–163, Feb. 2018.

[48] F. Duan, Y. Pan, F. Chapeau-Blondeau, and D. Abbott, “Noise benefits in combined nonlinear Bayesian estimators,” IEEE Trans. Signal Process., vol. 67, no. 17, pp. 4611–4623, Sep. 2019.

[49] D. Simon, J. Sulam, Y. Romano, Y. M. Lu, and M. Elad, “MMSE approximation for sparse coding algorithms using stochastic resonance,” IEEE Trans. Signal Process., vol. 67, no. 17, pp. 4597–4610, Sep. 2019.

[50] X. Liu, H. Wang, Y. Meng, and M. Fu, “Classification of hyperspectral image by CNN based on shadow area enhancement through dynamic stochastic resonance,” IEEE Access, vol. 7, pp. 134862–134870, 2019.

[51] S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory, vol. 1. Upper Saddle River, NJ, USA: Prentice-Hall, 1998.
[52] J.-J. Xiao, A. Ribeiro, Z.-Q. Luo, and G. B. Giannakis, "Distributed compression-estimation using wireless sensor networks," *IEEE Signal Process. Mag.*, vol. 23, no. 4, pp. 27–41, Jul. 2006.

[53] M. Shirazi and A. Vosoughi, "On Bayesian fisher information maximization for distributed vector estimation," *IEEE Trans. Signal Inf. Process. Netw.*, vol. 5, no. 4, pp. 628–645, Dec. 2019.

[54] H. L. von Trees, K. L. Bell, and Z. Tan, *Detection Estimation and Modulation Theory, Part I: Detection, Estimation, and Filtering Theory*. Hoboken, NJ, USA: Wiley, 2013.

[55] T. M. Cover and J. A. Thomas, *Elements of Information Theory*. New York, NY, USA: Wiley, 1991.

[56] J. Nocedal and S. J. Wright, *Numerical Optimization*. New York, NY, USA: Springer-Verlag, 2006.

[57] B. Widrow and S. D. Stearns, *Adaptive Signal Processing*. Upper Saddle River, NJ, USA: Prentice-Hall, 1985.

[58] S. Boyd and L. Vandenberghe, *Convex Optimization*. Cambridge, U.K.: Cambridge Univ. Press, 2004.

<figure><img src="image_9.png" /></figure>

**JIA LIU** was born in China, in 1995. She received the B.Eng. degree in communication engineering from Qingdao University, in 2018, where she is currently pursuing the master's degree in system science. Her research interests include signal estimation and machine learning.

<figure><img src="image_10.png" /></figure>

**FABING DUAN** was born in China, in 1974. He received the master's degree in engineering mechanics from the China University of Mining and Technology, Beijing, in 1999, and the Ph.D. degree in solid mechanics from Zhejiang University, China, in 2002. From 2002 to 2003, he was a Postdoctoral Fellow of the University of Angers, France. Since 2004, he has been with Qingdao University, China, where he is currently a Professor of system science. His research interests include nonlinear systems and signal processing.

<figure><img src="image_11.png" /></figure>

**FRANÇOIS CHAPEAU-BLONDEAU** was born in France, in 1959. He received the Diploma degree in engineering from ESEO, Angers, France, in 1982, the Ph.D. degree in electrical engineering from University Pierre et Marie Curie, Paris 6, France, in 1987, and the Habilitation degree from the University of Angers, Angers, in 1994. In 1988, he was a Research Associate with the Department of Biophysics, Mayo Clinic, Rochester, MN, USA, working on biomedical ultrasonics. Since 1990, he has been with the University of Angers, where he is currently a Professor of electrical and electronic engineering. His research interests include information theory, signal processing and imaging, and interactions between physics and information sciences.

<figure><img src="image_12.png" /></figure>

**DEREK ABBOTT** (Fellow, IEEE) was born in London, U.K., in 1960. He received the B.Sc. (Hons.) degree in physics from Loughborough University, Leicestershire, U.K., in 1982, and the Ph.D. degree in electrical and electronic engineering from The University of Adelaide, Adelaide, SA, Australia, in 1997, under the supervision of K. Eshraghian and B. R. Davis.

From 1978 to 1986, he was a Research Engineer with the GEC Hirst Research Centre, London. From 1986 to 1987, he was a VLSI Design Engineer with Austek Microsystems, Australia. Since 1987, he has been with The University of Adelaide, where he is currently a Full Professor with the School of Electrical and Electronic Engineering. He has coauthored *Stochastic Resonance* (Cambridge University Press, 2008), and *Terahertz Imaging for Biomedical Applications* (Springer-Verlag, 2012). His research interests include multidisciplinary physics and electronic engineering applied to complex systems, networks, game theory, energy policy, stochastics, and biophotonics.

Prof. Abbott is a Fellow of the Institute of Physics (IoP), U.K., and an Honorary Fellow of Engineers Australia. He currently serves on the IEEE Publication Services and Products Board (PSPB). He has received a number of awards, including the Tall Poppy Award for Science in 2004, the Australian Research Council Future Fellowship, in 2012, the David Dewhurst Medal, in 2015, the Barry Inglis Medal, in 2018, and the M. A. Sargent Medal for eminence in engineering, in 2019. He has been an Editor and/or Guest Editor of a number of journals, including the IEEE JOURNAL OF SOLID-STATE CIRCUITS, Journal of Optics B, Microelectronics Journal, PLOS ONE, PROCEEDINGS OF THE IEEE, and the IEEE PHOTONICS JOURNAL. He has served on the Editorial Board of the PROCEEDINGS OF THE IEEE, from 2009 to 2014. He has been on the Editorial Board of IEEE ACCESS, since 2015. He currently serves on the Editorial Board of IEEE ACCESS, Scientific Reports (Nature), Royal Society Open Science, and the Frontiers of Physics. He is the Co-Editor of Quantum Aspects of Life (Imperial College Press, 2008).

...