A Novel Robust Approach to Least Squares Problems with Bounded Data Uncertainties

In this correspondence, we introduce a minimax regret criteria to the least squares problems with bounded data uncertainties and solve it using semi-definite programming. We investigate a robust minimax least squares approach that minimizes a worst case difference regret. The regret is defined as the difference between a squared data error and the smallest attainable squared data error of a least squares estimator. We then propose a robust regularized least squares approach to the regularized least squares problem under data uncertainties by using a similar framework. We show that both unstructured and structured robust least squares problems and robust regularized least squares problem can be put in certain semi-definite programming forms. Through several simulations, we demonstrate the merits of the proposed algorithms with respect to the the well-known alternatives in the literature.

Related papers:

title: Structured Least Squares Problems and Robust Estimators
abstract: A novel approach is proposed to provide robust and accurate estimates for linear regression problems when both the measurement vector and the coefficient matrix are structured and subject to errors or uncertainty. A new analytic formulation is developed in terms of the gradient flow of the residual norm to analyze and provide estimates to the regression. The presented analysis enables us to establish theoretical performance guarantees to compare with existing methods and also offers a criterion to choose the regularization parameter autonomously. Theoretical results and simulations in applications such as blind identification, multiple frequency estimation and deconvolution show that the proposed technique outperforms alternative methods in mean-squared error for a significant range of signal-to-noise ratio values.

title: Robust Solutions to Least-Squares Problems with Uncertain Data
abstract: We consider least-squares problems where the coefficient matrices A,b are unknown but bounded. We minimize the worst-case residual error using (convex) second-order cone programming, yielding an algorithm with complexity similar to one singular value decomposition of A. The method can be interpreted as a Tikhonov regularization procedure, with the advantage that it provides an exact bound on the robustness of solution and a rigorous way to compute the regularization parameter. When the perturbation has a known (e.g., Toeplitz) structure, the same problem can be solved in polynomial-time using semidefinite programming (SDP). We also consider the case when A,b are rational functions of an unknown-but-bounded perturbation vector. We show how to minimize (via SDP) upper bounds on the optimal worst-case residual. We provide numerical examples, including one from robust identification and one from robust interpolation.

title: Structured Least Squares Problems and Robust Estimators
abstract: A novel approach is proposed to provide robust and accurate estimates for linear regression problems when both the measurement vector and the coefficient matrix are structured and subject to errors or uncertainty. A new analytic formulation is developed in terms of the gradient flow of the residual norm to analyze and provide estimates to the regression. The presented analysis enables us to establish theoretical performance guarantees to compare with existing methods and also offers a criterion to choose the regularization parameter autonomously. Theoretical results and simulations in applications such as blind identification, multiple frequency estimation and deconvolution show that the proposed technique outperforms alternative methods in mean-squared error for a significant range of signal-to-noise ratio values.

title: Recovery of sparse perturbations in Least Squares problems
abstract: We show that the exact recovery of sparse perturbations on the coefficient matrix in overdetermined Least Squares problems is possible for a large class of perturbation structures. The well established theory of Compressed Sensing enables us to prove that if the perturbation structure is sufficiently incoherent, then exact or stable recovery can be achieved using linear programming. We derive sufficiency conditions for both exact and stable recovery using known results of ℓ0/ℓ1 equivalence. However the problem turns out to be more complicated than the usual setting used in various sparse reconstruction problems. We propose and solve an optimization criterion and its convex relaxation to recover the perturbation and the solution to the Least Squares problem simultaneously. Then we demonstrate with numerical examples that the proposed method is able to recover the perturbation and the unknown exactly with high probability. The performance of the proposed technique is compared in blind identification of sparse multipath channels.

title: Linear minimax regret estimation of deterministic parameters with bounded data uncertainties
abstract: We develop a new linear estimator for estimating an unknown parameter vector x in a linear model in the presence of bounded data uncertainties. The estimator is designed to minimize the worst-case regret over all bounded data vectors, namely, the worst-case difference between the mean-squared error (MSE) attainable using a linear estimator that does not know the true parameters x and the optimal MSE attained using a linear estimator that knows x. We demonstrate through several examples that the minimax regret estimator can significantly increase the performance over the conventional least-squares estimator, as well as several other least-squares alternatives.

title: Minimax MSE-ratio estimation with signal covariance uncertainties
abstract: In continuation to an earlier work, we further consider the problem of robust estimation of a random vector (or signal), with an uncertain covariance matrix, that is observed through a known linear transformation and corrupted by additive noise with a known covariance matrix. While, in the earlier work, we developed and proposed a competitive minimax approach of minimizing the worst-case mean-squared error (MSE) difference regret criterion, here, we study, in the same spirit, the minimum worst-case MSE ratio regret criterion, namely, the worst-case ratio (rather than difference) between the MSE attainable using a linear estimator, ignorant of the exact signal covariance, and the minimum MSE (MMSE) attainable by optimum linear estimation with a known signal covariance. We present the optimal linear estimator, under this criterion, in two ways: The first is as a solution to a certain semidefinite programming (SDP) problem, and the second is as an expression that is of closed form up to a single parameter whose value can be found by a simple line search procedure. We then show that the linear minimax ratio regret estimator can also be interpreted as the MMSE estimator that minimizes the MSE for a certain choice of signal covariance that depends on the uncertainty region. We demonstrate that in applications, the proposed minimax MSE ratio regret approach may outperform the well-known minimax MSE approach, the minimax MSE difference regret approach, and the "plug-in" approach, where in the latter, one uses the MMSE estimator with an estimated covariance matrix replacing the true unknown covariance.

title: Competitive Linear Estimation Under Model Uncertainties
abstract: We investigate a linear estimation problem under model uncertainties using a competitive algorithm framework under mean square error (MSE) criteria. Here, the performance of a linear estimator is defined relative to the performance of the linear minimum MSE estimator tuned to the underlying unknown system model. We then find the linear estimator that minimizes this relative performance measure, i.e., the regret, for the worst possible system model. Two definitions of regret are given: first as a difference of MSEs and second as a ratio of MSEs. We demonstrate that finding the linear estimators that minimize these regret definitions can be cast as a semidefinite programming (SDP) problem and provide numerical examples.