# Whitening Effects for ML-DoA Estimation using a Sparse Representation of Array Covariance

Thomas Aussaguès, Anne Ferréol, Alice Delmer, Pascal Larzabal

► To cite this version:

Thomas Aussaguès, Anne Ferréol, Alice Delmer, Pascal Larzabal. Whitening Effects for ML-DoA Estimation using a Sparse Representation of Array Covariance. IEEE 50th International Conference on Acoustics, Speech and Signal Processing (ICASSP 2025), Apr 2025, Hyderabad, India. hal-04863271v2

HAL Id: hal-04863271

https://hal.science/hal-04863271v2

Submitted on 13 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Whitening Effects for ML-DoA Estimation using a Sparse Representation of Array Covariance

Thomas Aussaguès *\*†*, Anne Ferréol*†*, Alice Delmer* and Pascal Larzabal †

*Thales, 4 avenue des Louvresses, 92230 Gennevilliers, France

†SATIE, Université Paris-Saclay, UMR CNRS 8029, 4 avenue des Sciences, 91190 Gif-sur-Yvette, France

**Abstract**—Maximum Likelihood (ML) Direction-of-Arrival (DoA) estimation on the Vectorized Covariance Matrix Model (VCMM) exhibits improved performance in severe conditions compared to standard methods. Indeed, it benefits from the VCMM capacities summarized through the Virtual Array (VA) concept. Due to finite number of samples, the VCMM is corrupted by a coloured Gaussian noise. As a remedy, we previously introduced a pre-whitening noise transform converting the coloured noise into white Gaussian noise. Using the whitened model, we recently shown equivalence between sparse DoA estimators and the ML thereby enabling efficient implementation of ML DoA estimation under white Gaussian noise.

In this work, the noise pre-whitening transform is shown to significantly improve the sparse problem conditioning by spatially decorrelating the dictionary vectors associated to sources directions thus simplifying the implementation of ML DoA estimation with a sparse representation. To this end, the expression of the spatial correlation coefficient after whitening is derived.

Numerical simulations confirm the performance improvement of sparse DoA estimators after whitening for closely separated sources.

**Index Terms**—regularization parameter, optimization, resolution, orthogonal dictionary

## I. INTRODUCTION

DoA estimation is a fundamental signal processing problem arising in various fields such as telecommunications or radar. Over the past few decades, a wide range of estimators have been proposed [1] with some of the most well-known being Capon's beamformer [2] as well as High Resolution (HR) techniques such as MUSIC [3], [4], ESPRIT [5] and the ML estimator [6].

Although aforementioned HR methods outperform classical DoA estimation techniques, they all rely on an array of $N$ antennas which restricts the number of identifiable sources to $N-1$ sources. Chevalier et al. described in [7] the VA concept, with space diversity only, relying on at most $N^2-N$ non-redundant antennas hence increasing the maximal number of sources to be processed to $N^2-N-1$ and simplifying the resolution of poorly angularly separated sources. Extensions of the MUSIC algorithm to higher order moments such as 4-MUSIC [8], [9] have been proposed to exploit the VA. Alternatively, the VCMM can be employed to leverage the VA [10].

To estimate DoAs from the VCMM, the signal processing community investigated sparse estimators through last decades [11], [12]. Although to our knowledge there exists no theoretical performance for these estimators, several authors

illustrate by simulation performance improvements in difficult scenarios (low Signal-to-Noise Ratio (SNR), few available array snapshots) compared to traditional DoA estimators [13]–[16]. Nonetheless, the separation of closely spaced sources remains challenging as the corresponding dictionary vectors are highly correlated.

As outlined in [16], the VCMM is corrupted by a coloured noise vector due to the finite number of samples. The presence of coloured noise can significantly deteriorate performance [17], [18]. To address this, several authors have proposed modified estimators [19]–[21]. When sufficient data is available, a pre-whitening noise transform is applied transforming the initially coloured noise into white noise [6]. Such procedure is performed on the VCMM in [16] leading to a whitened model corrupted by white noise.

Sparse DoA methods [13]–[16] rely on the minimization of a $l_0$-regularized objective function parametrized by $\lambda$ an hyperparameter referred as the regularization parameter. Using the whitened model, a novel regularization parameter choice is derived ensuring the equivalence between sparse and ML DoA estimators under white Gaussian noise [16]. This equivalence enables efficient implementation of the deterministic ML through sparse optimization.

However, the noise pre-whitening transform has additional consequences on the VCMM for which no study has been provided.

This paper aims to fill this gap by providing a detailed theoretical analysis of the pre-whitening noise transform effects on the VCMM. In addition to ensuring equivalence with the ML, the whitening has notable consequences as it brings the dictionary matrix closer to an orthogonal matrix by decorrelating the dictionary vectors corresponding to the sources directions. Consequently, the problem conditioning is significantly improved thereby increasing the resolution power.

## II. THE VECTORIZED COVARIANCE MATRIX MODEL

Assuming a noisy mixture of $M$ independent narrowband plane waves of directions $\theta = \{\theta_1, \dots, \theta_M\}$ impinging on an array of $N$ antennas, the received signal is:

$$x(t) = \sum_{m=1}^{M} a(\theta_m)s_m(t) + n(t) = A(\theta)s_{\theta}(t) + n(t) \quad (1)$$

where $A(\theta) = [a(\theta_1), \dots, a(\theta_M)]$ is the steering matrix formed by the steering vectors $a(\theta_m), 1 \le m \le M, s_{\theta}(t) =$
$[s_1(t),...,s_M(t)]^T$ the complex envelopes of the emitted signals and $\mathbf{n}(t)$ a complex circular Gaussian noise, independent of $\mathbf{s}_{\theta}(t)$, with covariance matrix $\mathbb{E}[\mathbf{n}(t)\mathbf{n}^H(t)] = \sigma^2\mathbf{I}_N$ with $\mathbb{E}[\cdot]$ the temporal mean and $\mathbf{I}_N$ the identity matrix of size $N$. The covariance matrix of (1) is then:

$$ \mathbf{R}_x = \mathbb{E}[\mathbf{x}(t)\mathbf{x}^H(t)] = \mathbf{A}(\theta)\mathbf{R}_s\mathbf{A}^H(\theta) + \sigma^2\mathbf{I}_N \quad (2) $$

where $\mathbf{R}_s = \mathbb{E}[\mathbf{s}_{\theta}(t)\mathbf{s}_{\theta}^H(t)]$ denotes the sources covariance matrix.

As outlined in section I, the VA [7], which has higher resolution and maximal number of identifiable sources, can be accessed using the VCMM [10]. Under the hypothesis of temporally uncorrelated sources (ie. $\mathbf{R}_s$ is diagonal), the vectorized covariance matrix observation is:

$$ \mathbf{r} = \mathrm{vec}(\mathbf{R}_x - \sigma^2\mathbf{I}_N) = \sum_{m=1}^{M} \mathbf{b}(\theta_m)\gamma_m = \mathbf{B}(\theta)\gamma_{\theta} \quad (3) $$

with $\mathrm{vec}(\cdot)$ the column-wise vectorization operator, $\mathbf{B}(\theta)$ the VA steering matrix formed by the vectors $\mathbf{b}(\theta_m) = \mathbf{a}^*(\theta_m) \otimes \mathbf{a}(\theta_m)$ where $\otimes$ is the Kronecker product and $\gamma_{\theta} = \mathrm{diag}(\mathbf{R}_s) = [\gamma_1, ..., \gamma_M]^T$ the sources powers vector.

In practice, the covariance matrix $\mathbf{R}_x$ is estimated using $K$ identically and independently distributed array snapshots $\mathbf{x}(t_k), 1 \le k \le K$. Assuming temporally white noise $(\mathbb{E}[\mathbf{n}^H(t_i)\mathbf{n}(t_j)] = 0, i \ne j)$, the estimate $\hat{\mathbf{R}}_x$ of $\mathbf{R}_x$ can be decomposed as:

$$ \hat{\mathbf{R}}_x = \frac{1}{K} \sum_{k=1}^{K} \mathbf{x}(t_k)\mathbf{x}^H(t_k) = \mathbf{R}_x + \Delta\mathbf{R}_x \quad (4) $$

where $\Delta\mathbf{R}_x$ is a complex Wishart noise matrix due to finite number of samples [22]. Consequently, model (3) is corrupted by noise $\delta = \mathrm{vec}(\Delta\mathbf{R}_x)$ leading to:

$$ \mathbf{r} = \mathbf{B}(\theta)\gamma_{\theta} + \delta \quad (5) $$

As a result of the Central Limit Theorem [23], the complex Wishart distribution of $\delta$ (5) can be approximated by a complex Gaussian law $\mathbb{CN}(0_{N^2 \times 1}, \Gamma, C)$ for sufficiently large number of array snapshots with the following moments [24]:

$$ \Gamma = \mathbb{E}[\delta\delta^H] = \frac{1}{K} (\mathbf{R}_x^T \otimes \mathbf{R}_x) \quad (6) $$

$$ C = \mathbb{E}[\delta\delta^T] = K\Gamma $$

where $K$ denotes the permutation matrix such that $\mathrm{vec}(\mathbf{M}^T) = K\mathrm{vec}(\mathbf{M})$ for any square matrix $\mathbf{M} \in \mathbb{C}^{N^2 \times N^2}$ [25].

Following the approach of [16], a pre-whitening noise transform (7) is performed on (5) converting the initially coloured noise (as $\Gamma \ne \mathbf{I}_{N^2}$ (6)) into white Gaussian noise and thereby enabling equivalence with the ML under white Gaussian noise [16].

$$ \mathbf{y} = \hat{\mathbf{W}}\mathbf{r} = \hat{\mathbf{W}}\mathbf{B}(\theta)\gamma_{\theta} + \hat{\mathbf{W}}\delta = \mathbf{B}_w(\theta)\gamma + \delta_w \quad (7) $$

where $\hat{\mathbf{W}} = \hat{\Gamma}^{-1/2} = \sqrt{K}(\hat{\mathbf{R}}_x^{-T/2} \otimes \hat{\mathbf{R}}_x^{-1/2})$ is the whitening matrix, $\mathbf{B}_w(\theta) = \hat{\mathbf{W}}\mathbf{B}(\theta)$ denotes the whitened dictionary

and $\delta_w = \hat{\mathbf{W}}\delta$ the whitened noise satisfying $\mathbb{E}[\delta_w\delta_w^H] = \mathbf{I}_{N^2}$.

**Remark 1:** Using the vectorization property of the Kronecker product $\mathrm{vec}(\mathbf{ABC}) = (\mathbf{C}^T \otimes \mathbf{A}) \mathrm{vec}(\mathbf{B})$ [26], the whitened observation (7) can be rewritten as:

$$ \mathbf{y} = \sqrt{K}\mathrm{vec}(\mathbf{I}_N - \sigma^2\hat{\mathbf{R}}_x^{-1}) \quad (8) $$

showing that the pre-whitening noise transform preserves (7) the DoA information as it is contained within $\hat{\mathbf{R}}_x^{-1}$.

### III. SPARSE MODELING AND SPARSE ESTIMATION

#### A. Sparse modeling

Let us consider a grid of $G$ pre-defined directions $\varphi = \{\varphi_1, ..., \varphi_G\}$. Assuming that the sources directions $\theta$ lie within $\varphi$, a sparse equivalent of (7) can be obtained:

$$ \mathbf{y} = \mathbf{B}_w(\varphi)\gamma_0 + \delta_w \quad (9) $$

where $\mathbf{B}_w(\varphi) = \hat{\mathbf{W}}\mathbf{B}(\varphi)$ with $\mathbf{B}(\varphi) = [\mathbf{b}(\varphi_1), ..., \mathbf{b}(\varphi_G)]$ an overcomplete dictionary of size $N^2 \times G, G \gg N^2$. $\gamma_0$ is an $M$-sparse vector which has only $M$ non-zero components associated to sources directions.

#### B. Sparse estimation

A DoA estimator can be formulated from (9) as grid directions $\varphi_g$ corresponding to non-null entries of the sparse vector $\gamma_0$. Consequently, an estimate of $\gamma_0$ is needed to estimate $\theta$. Given that the $G$ unknown coefficients of $\gamma_0$ are estimated from an observation of length $N^2$, the problem is ill-posed and thus can not be resolved through classical least squares minimization. Hence, the sparsity prior must be exploited to ensure the uniqueness of the solution. As [15], [16] suggested, $\gamma_0$ is estimated through the minimization of the following $\ell_0$-regularized objective:

$$ \min_{\gamma \in \mathbb{C}^G} \left\{ \mathcal{J}_{\ell_0}(\lambda, \gamma) = \frac{1}{2} \|\mathbf{y} - \mathbf{B}_w(\varphi)\gamma\|_2^2 + \lambda \|\gamma\|_0 \right\} \quad (10) $$

where $\lambda > 0$ refers to the regularization parameter which balances the solution sparsity towards data fidelity. In [16], authors proposed a new regularization parameter choice ensuring that the global minimizer of (10) is the ML DoA solution. Thus, both estimators are said equivalent. As a by-product, this equivalence allows the theoretical characterization the sparse estimator by the way of ML performances [27], [28].

### IV. WHITENING EFFECTS ON THE DICTIONARY

To study the consequence of whitening on the dictionary $\mathbf{B}_w(\varphi) = \hat{\mathbf{W}}\mathbf{B}(\varphi)$ (7), let us introduce the spatial correlation function:

$$ r_{\mathbf{E}}(\varphi_i, \varphi_j) = \frac{|\mathbf{e}^H(\varphi_i)\mathbf{e}(\varphi_j)|}{\|\mathbf{e}(\varphi_i)\|_2\|\mathbf{e}(\varphi_j)\|_2} \quad (11) $$

which measures the spatial correlation between pairs of directions $\{\varphi_i, \varphi_j\}$ of a given dictionary $\mathbb{E} = [\mathbf{e}(\varphi_1), ..., \mathbf{e}(\varphi_G)]$.

Throughout this paper, a circular array with $N = 4$ antennas among which 3 are uniformly distributed around a circle of radius $0.5\lambda_0$ with $\lambda_0$ the wavelength and one
central antenna is considered. Fig.1 depicts the squared spatial correlation coefficients obtained for a scenario with $M = 2$ sources of directions $\theta_1 = 180^\circ$ and $\theta_2 = 195^\circ$ (leading to $|r_A(\theta_1, \theta_2)|^2 = 0.8$) with the same SNR of 10 dB and $K = 200$ array snapshots. Note that $K$ is sufficiently large so that $\hat{W} \approx W$ where $W = \sqrt{K}(R_x^{-T/2} \otimes R_x^{-1/2})$ is the true whitening matrix defined from (6). In the following, $|r_A|^2$, $|r_B|^2$, $|r_{B_w}|^2$ respectively denote the spatial correlation coefficients obtained for the classical array, the VA prior the noise transform and the VA after the noise transform.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Squared spatial correlation as a function of the steering vector direction $\varphi$. The sources positions $\{\theta_1 = 180^\circ, \theta_2 = 195^\circ\}$ are represented by dashed lines.</figcaption></figure>

As shown by Fig.1, the chosen array ($|r_A|^2$) exhibits many sidelobes and a large beamwidth. The corresponding VA ($|r_B|^2$) reduces both flaws since $|r_B| = |r_A|^2$ [7]. The VA after whitening ($|r_{B_w}|^2$) has almost no sidelobes and a reduced non-symmetric mainlobe. Furthermore, the whitening introduces a null value in the sources directions $\{\theta_1, \theta_2\}$.

The spatial decorrelation between sources directions observed on Fig.1 has remarkable effects on the problem conditioning. Indeed, the whitening modifies the eigenvalues of $\mathcal{H}_w = \mathbf{B}_w^H(\theta)\mathbf{B}_w(\theta)$, the sparse criterion projection on sources directions Hessian matrix, thus improving the problem conditioning as depicted by Fig.2.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Projections of the sparse criterion in directions $\{\theta_1, \theta_2\}$ before (left) and after (right) the noise pre-whitening transform. Dashed lines and the red circle represent the global minimum $\hat{\gamma} = [\hat{\gamma}_1, \hat{\gamma}_2]^T$ coordinates for each criterion. Both criteria are represented using the Bartlow colormap [29].</figcaption></figure>

The whitening refines the problem conditioning by reshaping elliptical level lines into circles thus facilitating the criterion optimization.

For this single noise realization, the whitening divides the problem conditioning by a factor 6 (the Hessian matrix eigenvalues before whitening, $\mathcal{H} = \mathbf{B}^H(\theta)\mathbf{B}(\theta)$, and after whitening, $\mathcal{H}_w$, are respectively $\{61.9, 10.1\}$ and $\{2, 1.8\}$ for this example) hence leading to almost circular contour lines which confirms the criteria behaviors on Fig.2.

In the following, the spatial correlation coefficient between sources direction $\{\theta_1, \theta_2\}$, on which depend the Hessian matrix eigenvalues, is derived to analyze the influence of the scenario parameter on the problem conditioning.

**Property 1:** Assuming that the number of samples is sufficiently large such that $\hat{W} \approx W$, the spatial correlation coefficient after the pre-whitening noise transform between sources of directions $\{\theta_1, \theta_2\}$ is:

$$r_{B_w}(\theta_1, \theta_2) = \frac{|\mathbf{Q}_{12}|^2}{\mathbf{Q}_{11}\mathbf{Q}_{22}} \quad (12)$$

where $\mathbf{Q} = (\mathbf{R}_s + \sigma^2 (\mathbf{A}^H(\theta)\mathbf{A}(\theta))^{-1})^{-1}$.

*Proof:* Let us first compute the inner product $\mathbf{b}_w^H(\theta_i)\mathbf{b}_w(\theta_j)$:

$$ \begin{aligned} & \mathbf{b}_w^H(\theta_i)\mathbf{b}_w(\theta_j) \\ &= (\mathbf{a}^*(\theta_i) \otimes \mathbf{a}(\theta_i))^H \mathbf{W}^H \mathbf{W} (\mathbf{a}^*(\theta_j) \otimes \mathbf{a}(\theta_j)) \\ &= K (\mathbf{a}^*(\theta_i) \otimes \mathbf{a}(\theta_i))^H (\mathbf{R}_x^{-T} \otimes \mathbf{R}_x^{-1}) (\mathbf{a}^*(\theta_j) \otimes \mathbf{a}(\theta_j)) \\ &= K (\mathbf{a}^T(\theta_i)\mathbf{R}_x^{-T}\mathbf{a}^*(\theta_j)) \otimes (\mathbf{a}^H(\theta_i)\mathbf{R}_x^{-1}\mathbf{a}(\theta_j)) \\ &= K |\mathbf{a}^H(\theta_i)\mathbf{R}_x^{-1}\mathbf{a}(\theta_j)|^2 \\ &= K |\mathbf{Q}_{ij}|^2 \end{aligned} \quad (13) $$

where we used $\mathbf{Q} = \mathbf{A}^H(\theta)\mathbf{R}_x^{-1}\mathbf{A}(\theta)$, (7) and $(\mathbf{A}\mathbf{C}) \otimes (\mathbf{B}\mathbf{D}) = (\mathbf{A} \otimes \mathbf{B})(\mathbf{C} \otimes \mathbf{D})$ the mixed-product property of the Kronecker product [30]. Finally, substituting (13) in (11) yields (12). Using both orthogonal projectors onto signal and noise subspaces of $\mathbf{R}_x$ (2) respectively denoted $\mathbf{\Pi}_s$ and $\mathbf{\Pi}_n = \mathbf{I}_N - \mathbf{\Pi}_s$, $\mathbf{R}_x$ (2) can be rewritten as:

$$ \begin{aligned} \mathbf{R}_x &= \mathbf{A}(\theta)\mathbf{R}_s\mathbf{A}^H(\theta) + \sigma^2 (\mathbf{\Pi}_s + \mathbf{\Pi}_n) \\ &= \mathbf{A}(\theta) (\mathbf{R}_s + \sigma^2 (\mathbf{A}^H(\theta)\mathbf{A}(\theta))^{-1}) \mathbf{A}^H(\theta) \\ &\quad + \sigma^2 \mathbf{\Pi}_n \end{aligned} \quad (14) $$

where $\mathbf{\Pi}_s = \mathbf{A}(\theta)\mathbf{A}^H(\theta)$ with $(\cdot)^H$ the Moore-Penrose pseudo-inverse. Finally, the orthogonality between signal and noise subspaces of $\mathbf{R}_x$ yields:

$$ \mathbf{R}_x^{-1} = (\mathbf{A}^H(\theta))^\# (\mathbf{R}_s + \sigma^2 (\mathbf{A}^H(\theta)\mathbf{A}(\theta))^{-1})^{-1} \\ \times \mathbf{A}^H(\theta) + \frac{1}{\sigma^2} \mathbf{\Pi}_n \quad (15) $$

Substituting (15) into (13) leads to:

$$ \begin{aligned} \mathbf{Q} &= \mathbf{A}^H(\theta) (\mathbf{A}^H(\theta))^\# (\mathbf{R}_s + \sigma^2 (\mathbf{A}^H(\theta)\mathbf{A}(\theta))^{-1})^{-1} \\ &\quad \times \mathbf{A}^H(\theta)\mathbf{A}(\theta) + \frac{1}{\sigma^2} \mathbf{A}^H(\theta)\mathbf{\Pi}_n\mathbf{A}(\theta) \\ &= (\mathbf{R}_s + \sigma^2 (\mathbf{A}^H(\theta)\mathbf{A}(\theta))^{-1})^{-1} \end{aligned} \quad (16) $$
<figure><img src="image_4.png" /><figcaption>Fig. 3. Spatial correlation coefficient prior (left) and after (right) the pre-whitening noise transform as a function of the angular spacing between sources $\Delta\theta = |\theta_1 - \theta_2|$ and the SNR.</figcaption></figure>

Applying the property (12) in the special case of $M = 2$ temporally decorrelated sources of directions $\{\theta_1, \theta_2\}$ with $\mathbf{R}_s = \gamma \mathbf{I}_M$ where $\gamma = \mathbb{E}[s_m(t)s_m^*(t)]$ is the source power gives:

$$r_{\mathbf{B}_w}(\theta_1, \theta_2) = \frac{|r_{\mathbf{A}}(\theta_1, \theta_2)|^2}{N^2 (1 - |r_{\mathbf{A}}(\theta_1, \theta_2)|)^2} \times \left( \rho + \frac{1}{N \left(1 - |r_{\mathbf{A}}(\theta_1, \theta_2)|^2\right)} \right)^{-2} \quad (17)$$

where $\rho = \frac{\gamma}{\sigma^2}$ is the SNR in linear units. $|r_{\mathbf{B}_w}(\theta_1, \theta_2)|^2$ is represented on Fig.3.

For sufficient SNR, the pre-whitening noise transform can achieve low spatial correlation even for highly correlated sources and consequently approach an orthogonal dictionary matrix.

# V. NUMERICAL SIMULATIONS

## A. Experimental setup

To illustrates the performance improvement due to whitening, both detection probability $P(\theta_m)$ and Root Mean Square Error RMSE($\theta_m$) of the sparse estimator defined in section III without and with whitening are estimated with respect to $\Delta\theta = |\theta_1 - \theta_2|$. The two sources are considered resolved if two peaks are detected and $\max\{|\hat{\theta}_1 - \theta_1|, |\hat{\theta}_2 - \theta_2|\} < 30^\circ$ where $30^\circ$ is the array half beamwidth. For the non-whitened observation $\mathbf{r}$ (5), $\lambda$ is selected using the framework of [15] whereas we rely on [16] for the whitened observation $\mathbf{y}$ (7). Note that an oracle grid containing the ML estimate is employed to seek the equivalence [16] with a stepsize of $1^\circ$. Array and all scenario parameters are identical to those of section IV.

The optimization of (10) is performed using the Forward-Backward Splitting algorithm [31]. To alleviate the flaws inherent to the $l_0$-norm, we replaced it with the CEL0 functional introduced by Soubies et al. [32] which is a continuous approximation of the $l_0$-norm. Furthermore, authors proved that the corresponding sparse criterion exhibits less local minima while having the same global minimizer as $\mathcal{J}_{l_0}$.

The CEL0 penalty was developed assuming an orthogonal dictionary ie. $\mathbf{B}_w^H(\varphi)\mathbf{B}_w(\varphi) = \mathbf{I}_{N^2}$. Although, the whitened dictionary does not satisfy this property, whitening makes it closer to an orthogonal dictionary by orthogonalizing the sources directions and so to the convex optimal of [32].

## B. Simulation results

Fig.4 presents the simulation results for direction $\theta_1$. As expected, the estimator obtained with the whitened dictionary outperforms the non-whitened one. Indeed, it reaches a detection probability of 1 for angular spacings greater than $13^\circ$ whereas without whitening, the required angular spacing is equal to $20^\circ$. Furthermore, the whitened estimator achieves a lower RSME for all angular spacings.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Top: probability of detection. Bottom: RMSE represented for angular spacing leading to a detection probability greater than 0.8. Performances are only represented for direction $\theta_1$ since the chosen array is $\theta$-invariant and both sources have equal power.</figcaption></figure>

The lack of resolution without whitening can additionally be explained by the behaviour of the CEL0 functional for highly correlated directions. As shown by simulation in [33] for closely spaced sources, the CEL0 penalized criterion exhibits large regions where the criterion value remains approximately constant. These regions contain solutions for which the few non-null components of the sparse vectors are associated to directions around the middle of the true directions $\frac{1}{2}(\theta_1 + \theta_2)$. By orthogonalizing the sources directions, the whitening transform reduces flat regions thus improving the resolution.

# VI. CONCLUSION

In this paper, we studied the effects of the noise pre-whitening transform on the VCMM for sparse estimators. The transform significantly simplifies the estimator implementation by bringing the dictionary matrix closer to an orthogonal matrix thereby enhancing the problem conditioning. To analyze our findings, an analytical expression of the spatial correlation coefficient between sources after pre-whitening is derived. Finally, numerical experiments confirmed the performance increase of sparse DoA estimators.

In a future paper, we will provide theoretical analysis of the connection between spatial correlation, problem conditioning and dictionary orthogonality.
REFERENCES

[1] H. Krim and M. Viberg, "Two decades of array signal processing research: The parametric approach," *Signal Processing Magazine, IEEE*, vol. 13, pp. 67–94, 08 1996.

[2] J. Capon, "High-resolution frequency-wavenumber spectrum analysis," *Proceedings of the IEEE*, vol. 57, no. 8, pp. 1408–1418, 1969.

[3] G. Bienvenu and L. Kopp, "Optimality of high resolution array processing using the eigensystem approach," *IEEE Transactions on Acoustics, Speech, and Signal Processing*, vol. 31, no. 5, pp. 1235–1248, 1983.

[4] R. Schmidt, "Multiple emitter location and signal parameter estimation," *IEEE Transactions on Antennas and Propagation*, vol. 34, no. 3, pp. 276–280, 1986.

[5] R. Roy, A. Paulraj, and T. Kailath, "Estimation of signal parameters via rotational invariance techniques - esprit," in *MILCOM 1986 - IEEE Military Communications Conference: Communications-Computers: Teamed for the 90's*, vol. 3, 1986, pp. 41.6.1–41.6.5.

[6] B. Ottersten, M. Viberg, P. Stoica, e. S. Nehorai, A.", J. Litva, and T. J. Shepherd, *Exact and Large Sample Maximum Likelihood Techniques for Parameter Estimation and Detection in Array Processing*. Berlin, Heidelberg: Springer Berlin Heidelberg, 1993, pp. 99–151.

[7] P. Chevalier, L. Albera, A. Ferreol, and P. Comon, "On the virtual array concept for higher order array processing," *IEEE Transactions on Signal Processing*, vol. 53, no. 4, pp. 1254–1271, 2005.

[8] B. Porat and B. Friedlander, "Direction finding algorithms based on high-order statistics," *IEEE Transactions on Signal Processing*, vol. 39, no. 9, pp. 2016–2024, 1991.

[9] P. Chevalier, A. Ferreol, and L. Albera, "High-resolution direction finding from higher order statistics: The 2q-music algorithm," *IEEE Transactions on Signal Processing*, vol. 54, no. 8, pp. 2986–2997, 2006.

[10] J. S. Picard and A. J. Weiss, "Direction finding of multiple emitters by spatial sparsity and linear programming," in *2009 9th International Symposium on Communications and Information Technology*, 2009, pp. 1258–1262.

[11] Z. Yang, J. Li, P. Stoica, and L. Xie, "Sparse methods for direction-of-arrival estimation," 2017.

[12] M. Pesavento, M. Trinh-Hoang, and M. Viberg, "Three more decades in array signal processing research: An optimization and structure exploitation perspective," *IEEE Signal Processing Magazine*, vol. 40, no. 4, pp. 92–106, 2023.

[13] Z. He, Q. Liu, L. Jin, and S. Ouyang, "Low complexity method for doa estimation using array covariance matrix sparse representation," *Electronics Letters*, vol. 49, no. 3, pp. 228–230, 2013.

[14] W. Cui, T. Qian, and J. Tian, "Enhanced covariances matrix sparse representation method for doa estimation," *Electronics Letters*, vol. 51, no. 16, pp. 1288–1290.

[15] A. Delmer, A. Ferréol, and P. Larzabal, "On regularization parameter for l0-sparse covariance fitting based doa estimation," in *ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2020, pp. 4552–4556.

[16] T. Aussaguès, A. Ferréol, A. Delmer, and P. Larzabal, "Looking for equivalence between maximum likelihood and sparse doa estimators," in *2024 32nd European Signal Processing Conference (EUSIPCO)*, 2024.

[17] F. Li and R. Vaccaro, "Performance degradation of doa estimators due to unknown noise fields," in [*Proceedings* ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing, 1991], pp. 1413–1416 vol.2.

[18] M. Viberg, "Sensitivity of parametric direction finding to colored noise fields and undermodeling," *Signal Processing*, vol. 34, no. 2, pp. 207–222, 1993.

[19] Q. Wu and K. M. Wong, "Un-music and un-cle: an application of generalized correlation analysis to the estimation of the direction of arrival of signals in unknown correlated noise," *IEEE Transactions on Signal Processing*, vol. 42, no. 9, pp. 2331–2343, 1994.

[20] V. Nagesha and S. Kay, "Maximum likelihood estimation for array processing in colored noise," *IEEE Transactions on Signal Processing*, vol. 44, no. 2, pp. 169–180, 1996.

[21] M. Pesavento and A. Gershman, "Maximum-likelihood direction-of-arrival estimation in the presence of unknown nonuniform noise," *IEEE Transactions on Signal Processing*, vol. 49, no. 7, pp. 1310–1324, 2001.

[22] H. Krim, P. Forster, and J. Proakis, "Operator approach to performance analysis of root-music and root-min-norm," *IEEE Transactions on Signal Processing*, vol. 40, no. 7, pp. 1687–1696, 1992.

[23] R. J. Muirhead, "Aspects of multivariate statistical theory," in *Wiley Series in Probability and Statistics*, 1982.

[24] M. Mahot, F. Pascal, P. Forster, and J.-P. Ovarlez, "Asymptotic properties of robust complex covariance matrix estimates," *IEEE Transactions on Signal Processing*, vol. 61, no. 13, pp. 3348–3356, 2013.

[25] J. Magnus and H. Neudecker, "The commutation matrix: Some properties and applications," *Annals of Statistics*, vol. 7, no. 2, pp. 381–394, 1979, pagination: 14.

[26] H. N. Jan R. Magnus, *Matrix Differential Calculus with Applications in Statistics and Econometric*, Third Edition.

[27] P. Stoica and A. Nehorai, "Music, maximum likelihood, and cramer-rao bound," *IEEE Transactions on Acoustics, Speech, and Signal Processing*, vol. 37, no. 5, pp. 720–741, 1989.

[28] A. Renaux, P. Forster, E. Chaumette, and P. Larzabal, "On the high-snr conditional maximum-likelihood estimator full statistical characterization," *IEEE Transactions on Signal Processing*, vol. 54, no. 12, pp. 4840–4843, 2006.

[29] F. Crameri, G. E. Shephard, and P. J. Heron, "The misuse of colour in science communication," *Nature Communications*, vol. 11, no. 1, p. 5444, 2020.

[30] J. R. Magnus and H. Neudecker, *Matrix Differential Calculus with Applications in Statistics and Econometrics*, third edition.

[31] P. L. Combettes and J.-C. Pesquet, "Proximal splitting methods in signal processing," 2010.

[32] E. Soubies, L. Blanc-Féraud, and G. Aubert, "A Continuous Exact l0 penalty (CEL0) for least squares regularized problem," *SIAM Journal on Imaging Sciences*, vol. 8, no. 3, pp. pp. 1607–1639 (33 pages), Jul. 2015.

[33] A. Delmer, A. Ferréol, and P. Larzabal, "On the complementarity of sparse l0 and cel0 regularized loss landscapes for doa estimation," *Sensors*, vol. 21, no. 18, 2021.