the cramer rao bound for estimating a sparse parameter vector the goal of this contribution is to characterize the best achievable mean squared error mse in estimating a sparse deterministic parameter from measurements corrupted by gaussian noise to this end an appropriate definition of bias in the sparse setting is developed and the constrained cramer rao bound crb is obtained this bound is shown to equal the crb of an estimator with knowledge of the support set for almost all feasible parameter values consequently in the unbiased case our bound is identical to the mse of the oracle estimator combined with the fact that the crb is achieved at high signal to noise ratios signal to noise ratio snrs by the maximum likelihood technique our result provides a new interpretation for the common practice of using the oracle estimator as a gold standard against which practical approaches are compared

exploring estimator bias variance tradeoffs using the uniform cr bound we introduce a plane which we call the delta sigma plane that is indexed by the norm of the estimator bias gradient and the variance of the estimator the norm of the bias gradient is related to the maximum variation in the estimator bias function over a neighborhood of parameter space using a uniform cramer rao cr bound on estimator variance a delta sigma tradeoff curve is specified that defines an unachievable region of the delta sigma plane for a specified statistical model in order to place an estimator on this plane for comparison with the delta sigma tradeoff curve the estimator variance bias gradient and bias gradient norm must be evaluated we present a simple and accurate method for experimentally determining the bias gradient norm based on applying a bootstrap estimator to a sample mean constructed from the gradient of the log likelihood we demonstrate the methods developed in this paper for linear gaussian and nonlinear poisson inverse problems

mse bounds with affine bias dominating the cramer rao bound in continuation to an earlier work we further develop bounds on the mean squared error mse when estimating a deterministic parameter vector thetas0 in a given estimation problem as well as estimators that achieve the optimal performance the traditional cramer rao cr type bounds provide benchmarks on the variance of any estimator of thetas0 under suitable regularity conditions while requiring a priori specification of a desired bias gradient to circumvent the need to choose the bias which is impractical in many applications it was suggested in our earlier work to directly treat the mse which is the sum of the variance and the squared norm of the bias while previously we developed mse bounds assuming a linear bias vector here we study in the same spirit affine bias vectors we demonstrate through several examples that allowing for an affine transformation can often improve the performance significantly over a linear approach using convex optimization tools we show that in many cases we can choose an affine bias that results in an mse bound that is smaller than the unbiased cr bound for all values of thetas0 furthermore we explicitly construct estimators that achieve these bounds in cases where an efficient estimator exists by performing an affine transformation of the standard maximum likelihood ml estimator this leads to estimators that result in a smaller mse than ml for all possible values of thetas0

cram acute text e r rao bound for constrained parameter estimation using lehmann unbiasedness the constrained cram acute text e r rao bound ccrb is a lower bound on the mean squared error mse of estimators that satisfy some unbiasedness conditions although the ccrb unbiasedness conditions are satisfied asymptotically by the constrained maximum likelihood cml estimator in the non asymptotic region these conditions are usually too strict and the commonly used estimators such as the cml estimator do not satisfy them therefore the ccrb may not be a lower bound on the mse matrix of such estimators in this paper we propose a new definition for unbiasedness under constraints denoted by c unbiasedness which is based on using lehmann unbiasedness with a weighted mse wmse risk and taking into account the parametric constraints in addition to c unbiasedness a cram acute text e r rao type bound on the wmse of c unbiased estimators denoted as lehmann unbiased ccrb lu ccrb is derived this bound is a scalar bound that depends on the chosen weighted combination of estimation errors it is shown that c unbiasedness is less restrictive than the ccrb unbiasedness conditions thus the set of estimators that satisfy the ccrb unbiasedness conditions is a subset of the set of c unbiased estimators and the proposed lu ccrb may be an informative lower bound in cases where the corresponding ccrb is not in the simulations we examine linear and nonlinear estimation problems under nonlinear constraints in which the cml estimator is shown to be c unbiased and the lu ccrb is an informative lower bound on the wmse while the corresponding ccrb on the wmse is not a lower bound and is not informative in the non asymptotic region

a simple derivation of the constrained multiple parameter cramer rao bound j d gorman and a o hero 1988 obtained a remarkable extension to the classical multiple parameter cramer rao cr lower bound that accounts for deterministic nonlinear equality constraints on the parameters the virtue of gorman and hero s result is that the constrained cr bound on all of the parameters is obtained by subtracting an easily computed nonnegative definite correction matrix from the unconstrained cr bound matrix the author presents a new simple derivation of the constrained cr bound and a new necessary condition for an estimator to satisfy the constrained cr bound with equality

covariance subspace and intrinsic crame spl acute r rao bounds crame spl acute r rao bounds on estimation accuracy are established for estimation problems on arbitrary manifolds in which no set of intrinsic coordinates exists the frequently encountered examples of estimating either an unknown subspace or a covariance matrix are examined in detail the set of subspaces called the grassmann manifold and the set of covariance positive definite hermitian matrices have no fixed coordinate system associated with them and do not possess a vector space structure both of which are required for deriving classical crame spl acute r rao bounds intrinsic versions of the crame spl acute r rao bound on manifolds utilizing an arbitrary affine connection with arbitrary geodesics are derived for both biased and unbiased estimators in the example of covariance matrix estimation closed form expressions for both the intrinsic and flat bounds are derived and compared with the root mean square error rmse of the sample covariance matrix scm estimator for varying sample support k the accuracy bound on unbiased covariance matrix estimators is shown to be about 10 log 10 n k sup 1 2 db where n is the matrix order remarkably it is shown that from an intrinsic perspective the scm is a biased and inefficient estimator and that the bias term reveals the dependency of estimation accuracy on sample support observed in theory and practice the rmse of the standard method of estimating subspaces using the singular value decomposition svd is compared with the intrinsic subspace crame spl acute r rao bound derived in closed form by varying both the signal to noise ratio snr of the unknown p dimensional subspace and the sample support in the simplest case the crame spl acute r rao bound on subspace estimation accuracy is shown to be about p n p sup 1 2 1 2 snr sup 1 2 rad for p dimensional subspaces it is seen that the svd based method yields accuracies very close to the crame spl acute r rao bound establishing that the principal invariant subspace of a random sample provides an excellent estimator of an unknown subspace the analysis approach developed is directly applicable to many other estimation problems on manifolds encountered in signal processing and elsewhere such as estimating rotation matrices in computer vision and estimating subspace basis vectors in blind source separation

extended ziv zakai lower bound for vector parameter estimation the bayesian ziv zakai bound on the mean square error mse in estimating a uniformly distributed continuous random variable is extended for arbitrarily distributed continuous random vectors and for distortion functions other than mse the extended bound is evaluated for some representative problems in time delay and bearing estimation the resulting bounds have simple closed form expressions and closely predict the simulated performance of the maximum likelihood estimator in all regions of operation

cyclic barankin type bounds for non bayesian periodic parameter estimation in many practical periodic parameter estimation problems the appropriate performance criteria are periodic in the parameter space the existing mean square error mse lower bounds such as cramer rao bound crb and barankin type bounds do not provide valid lower bounds in such problems in this paper cyclic versions of the crb and the barankin type bounds hammersley chapman robbins and mcaulay seidman are derived for non bayesian parameter estimation the proposed bounds are lower bounds on the mean cyclic error mce of any cyclic unbiased estimator where the cyclic unbiasedness is defined by using lehmann unbiasedness these mce lower bounds can be readily obtained from existing mse lower bounds and thus can be easily calculated the cyclic barankin type bounds and the performance of the maximum likelihood ml estimator are compared in terms of mce in von mises distributed measurements problem and for frequency and amplitude estimation with gaussian noise in these problems the ml estimator is found to be cyclic unbiased

lower bounds for parametric estimation with constraints a chapman robbins form of the barankin bound is used to derive a multiparameter cramer rao cr type lower bound on estimator error covariance when the parameter theta in r sup n is constrained to lie in a subset of the parameter space a simple form for the constrained cr bound is obtained when the constraint set theta sub c can be expressed as a smooth functional inequality constraint it is shown that the constrained cr bound is identical to the unconstrained cr bound at the regular points of theta sub c i e where no equality constraints are active on the other hand at those points theta in theta sub c where pure equality constraints are active the full rank fisher information matrix in the unconstrained cr bound must be replaced by a rank reduced fisher information matrix obtained as a projection of the full rank fisher matrix onto the tangent hyperplane of the full rank fisher matrix onto the tangent hyperplane of the constraint set at theta a necessary and sufficient condition involving the forms of the constraint and the likelihood function is given for the bound to be achievable and examples for which the bound is achieved are presented in addition to providing a useful generalization of the cr bound the results permit analysis of the gain in achievable mse performance due to the imposition of particular constraints on the parameter space without the need for a global reparameterization

pattern recognition based on scale invariant discriminant functions some probability models for classifying individuals as belonging to one of two or more populations using scale invariant discriminant functions are considered the investigation is motivated by practical situations where the observed data on an individual are in the form of ratios of some basic measurements or measurements scaled by an unknown nonnegative number the probability models are obtained by considering a p vector random variable x with a known distribution and deriving the distribution of the random vector y g x 1 x where g x is a nonnegative measure of size such that g x g x for 0 explicit expressions are obtained for the densities of what are called angular gaussian compositional gaussian type 1 and compositional gaussian type 2 distributions

on intrinsic cramer rao bounds for riemannian submanifolds and quotient manifolds we study cramer rao bounds crb s for estimation problems on riemannian manifolds in s t smith covariance subspace and intrinsic cramer rao bounds ieee trans signal process vol 53 no 5 1610 1630 2005 the author gives intrinsic crb s in the form of matrix inequalities relating the covariance of estimators and the fisher information of estimation problems we focus on estimation problems whose parameter space bar cal p is a riemannian submanifold or a riemannian quotient manifold of a parent space cal p that is estimation problems on manifolds with either deterministic constraints or ambiguities the crb s in the aforementioned reference would be expressed w r t bases of the tangent spaces to bar cal p in some cases though it is more convenient to express covariance and fisher information w r t bases of the tangent spaces to cal p we give crb s w r t such bases expressed in terms of the geodesic distances on the parameter space the bounds are valid even for singular fisher information matrices in two examples we show how the crb s for synchronization problems including a type of sensor network localization problem differ in the presence or absence of anchors leading to bounds for estimation on either submanifolds or quotient manifolds with very different interpretations

regularized orthogonal least squares algorithm for constructing radial basis function networks the paper presents a regularized orthogonal least squares learning algorithm for radial basis function networks the proposed algorithm combines the advantages of both the orthogonal forward regression and regularization methods to provide an efficient and powerful procedure for constructing parsimonious network models that generalize well examples of nonlinear modelling and prediction are used to demonstrate better generalization performance of this regularized orthogonal least squares algorithm over the unregularized one

weighted normal plots abstract weighted normal plots are proposed as graphical checks on the normality of random effects in gaussian linear models the technique is illustrated using the one way comparisons model yi i i where the i i are independent pairs with i and i independent n 0 2 and n 0 2 i respectively for i 1 n when the variance components 2 and 2 i are known an unweighted normal plot of the standardized zi yi 2 2 i 1 2 provides a check of the overall adequacy of the model weighted normal plots involve a modification that gives the ith observation a sample weight of wi 2 2 i 1 under the null hypothesis the sample size must be larger by a factor of 1 v m 2 where m and v are the mean and variance of the weights to produce a weighted plot with approximately the same sampling variance as an unweighted normal plot despite this higher variability we show that weighted plots are more sensitive than unweighted plots to several departures from the assume

maximum likelihood estimation the cramer rao bound and the method of scoring with parameter constraints maximum likelihood ml estimation is a popular approach to solving many signal processing problems many of these problems cannot be solved analytically and so numerical techniques such as the method of scoring are applied however in many scenarios it is desirable to modify the ml problem with the inclusion of additional side information often this side information is in the form of parametric constraints which the ml estimate mle must now satisfy we unify the asymptotic constrained ml cml theory with the constrained cramer rao bound ccrb theory by showing the cml estimate cmle is asymptotically efficient with respect to the ccrb we also generalize the classical method of scoring using the ccrb to include the constraints satisfying the constraints after each iterate convergence properties and examples verify the usefulness of the constrained scoring approach as a particular example an alternative and more general cmle is developed for the complex parameter linear model with linear constraints a novel proof of the efficiency of this estimator is provided using the ccrb

fundamentals of statistical signal processing estimation theory minimum variance unbiased estimation cramer rao lower bound linear models general minimum variance unbiased estimation best linear unbiased estimators maximum likelihood estimation least squares method of moments the bayesian philosophy general bayesian estimators linear bayesian estimators kalman filters summary of estimators extension for complex data and parameters

universal weighted mse improvement of the least squares estimator since the seminal work of stein in the 1950s there has been continuing research devoted to improving the total mean squared error mse of the least squares ls estimator in the linear regression model however a drawback of these methods is that although they improve the total mse they do so at the expense of increasing the mse of some of the individual signal components here we consider a framework for developing linear estimators that outperform the ls strategy over bounded norm signals under all weighted mse measures this guarantees for example that both the total mse and the mse of each of the elements will be smaller than that resulting from the ls approach we begin by deriving an easily verifiable condition on a linear method that ensures ls domination for every weighted mse we then suggest a minimax estimator that minimizes the worst case mse over all weighting matrices and bounded norm signals subject to the universal weighted mse domination constraint

general classes of performance lower bounds for parameter estimation part i non bayesian bounds for unbiased estimators in this paper a new class of lower bounds on the mean square error mse of unbiased estimators of deterministic parameters is proposed derivation of the proposed class is performed by projecting each entry of the vector of estimation error on a hilbert subspace of l2 this hilbert subspace contains linear transformations of elements in the domain of an integral transform of the likelihood ratio function the integral transform generalizes the traditional derivative and sampling operators which are applied on the likelihood ratio function for computation of performance lower bounds such as cramer rao bhattacharyya and mcaulay seidman bounds it is shown that some well known lower bounds on the mse of unbiased estimators can be derived from this class by modifying the kernel of the integral transform a new lower bound is derived from the proposed class using the kernel of the fourier transform in comparison with other existing bounds the proposed bound is computationally manageable and provides better prediction of the threshold region of the maximum likelihood estimator in the problem of single tone estimation

new results on deterministic cramer rao bounds for real and complex parameters the cramer rao bounds crb is a lower bound of great interest for system analysis and design in the asymptotic region high signal to noise ratio snr and or large number of snapshots as it is simple to calculate and it is usually possible to obtain closed form expressions the first part of the paper is a generalization to complex parameters of the barankin rationale for deriving mse lower bounds that is the minimization of a norm under a set of linear constraints with the norm minimization approach the study of fisher information matrix fim singularity constrained crb and regularity conditions become straightforward corollaries of the derivation the second part provides new results useful for system analysis and design a general reparameterization inequality the equivalence between reparameterization and equality constraints and an explicit relationship between parameters unidentifiability and fim singularity

polyhedrization of an arbitrary 3d point set abstract given a 3d point set the problem of defining the volume associated dividing it into a set of regions elements and defining a boundary surface is tackled several physical problems need to define volume domains boundary surfaces and approximating functions from a given point distribution this is for instance the case of particle methods in which all the information is the particle positions and there are not boundary surfaces definition until recently all the fem mesh generators were limited to the generation of simple elements as tetrahedral or hexahedral elements or triangular and quadrangular in 2d problems the reason of this limitation was the lack of any acceptable shape function to be used in other kind of geometrical elements nowadays there are several acceptable shape functions for a very large class of polyhedra these new shape functions together with a generalization of the delaunay tessellation presented in this paper gives an optimal marriage and a powerful tool to solve a large variety of physical problems by numerical methods the domain partition into polyhedra presented here is not a standard mesh generation the problem here is for a given node distribution to find a suitable boundary surface and a suitable mesh to be used in the solution of a physical problem by a numerical method to include new nodes or change their positions is not allowed