proximal thresholding algorithm for minimization over orthonormal bases the notion of soft thresholding plays a central role in problems from various areas of applied mathematics in which the ideal solution is known to possess a sparse decomposition in some orthonormal basis using convex analytical tools we extend this notion to that of proximal thresholding and investigate its properties providing in particular several characterizations of such thresholders we then propose a versatile convex variational formulation for optimization over orthonormal bases that covers a wide range of problems and we establish the strong convergence of a proximal thresholding algorithm to solve it numerical applications to signal recovery are demonstrated

estimation of the mean of a multivariate normal distribution estimation of the means of independent normal random variables is considered using sum of squared errors as loss an unbiased estimate of risk is obtained for an arbitrary estimate and certain special classes of estimates are then discussed the results are applied to smoothing by use of moving averages and to trimmed analogs of the james stein estimate a suggestion is made for calculating approximate confidence sets for the mean vector centered at an arbitrary estimate

tikhonov regularization and the l curve for large discrete ill posed problems discretization of linear inverse problems generally gives rise to very ill conditioned linear systems of algebraic equations typically the linear systems obtained have to be regularized to make the computation of a meaningful approximate solution possible tikhonov regularization is one of the most popular regularization methods a regularization parameter specifies the amount of regularization and in general an appropriate value of this parameter is not known a priori we review available iterative methods and present new ones for the determination of a suitable value of the regularization parameter by the l curve criterion and the solution of regularized systems of algebraic equations

the generalized logit linear item response model for binary designed items this paper introduces the generalized logit linear item response model gllirm which represents the item solving process as a series of dichotomous operations or steps the gllirm assumes that the probability function of the item response is a logistic function of a linear composite of basic parameters which describe the operations and the coefficients depend on three design matrices x y and z the gllirm provides a tool for testing hypotheses on the item solving process and generalizes existing models an empirical application is included in which the model is applied to evaluate sources of difficulty and pairwise item interactions in a logical analysis test

an interior point method for large scale l 1 regularized logistic regression logistic regression with l1 regularization has been proposed as a promising method for feature selection in classification problems in this paper we describe an efficient interior point method for solving large scale l1 regularized logistic regression problems small problems with up to a thousand or so features and examples can be solved in seconds on a pc medium sized problems with tens of thousands of features and examples can be solved in tens of seconds assuming some sparsity in the data a variation on the basic method that uses a preconditioned conjugate gradient method to compute the search step can solve very large problems with a million features and examples e g the 20 newsgroups data set in a few minutes on a pc using warm start techniques a good approximation of the entire regularization path can be computed much more efficiently than by solving a family of problems independently

structural analysis of subjective categorical data a general approach to the analysis of subjective categorical data is considered in which agreement matrices of two or more raters are directly expressed in terms of error and agreement parameters the method provides focused analyses of ratings from several raters for whom ratings have measurement error distributions that may induce bias in the evaluation of substantive questions of interest each rater s judgment process is modeled as a mixture of two components an error variable that is unique for the rater in question as well as an agreement variable that operationalizes the true values of the units of observation the statistical problems of identification estimation and testing of such measurement models are discussed

shrinkage tuning parameter selection with a diverging number of parameters contemporary statistical research frequently deals with problems involving a diverging number of parameters for those problems various shrinkage methods e g the lasso and smoothly clipped absolute deviation are found to be particularly useful for variable selection nevertheless the desirable performances of those shrinkage methods heavily hinge on an appropriate selection of the tuning parameters with a fixed predictor dimension wang and co worker have demonstrated that the tuning parameters selected by a bayesian information criterion type criterion can identify the true model consistently in this work similar results are further extended to the situation with a diverging number of parameters for both unpenalized and penalized estimators consequently our theoretical results further enlarge not only the scope of applicability of the traditional bayesian information criterion type criteria but also that of those shrinkage estimation methods

on the discrepancy principle for some newton type methods for solving nonlinear inverse problems we consider the computation of stable approximations to the exact solution x dagger of nonlinear ill posed inverse problems f x y with nonlinear operators f x y between two hilbert spaces x and y by the newton type methods x k 1 delta x 0 g alpha k left f x k delta f x k delta right f x k delta left f x k delta y delta f x k delta x k delta x 0 right of y satisfying y delta y le delta with a given small noise level delta 0 we terminate the iteration by the discrepancy principle in which the stopping index k delta is determined as the first integer such that f x k delta delta y delta le tau delta f x k delta y delta quad 0 le k k delta converges to x dagger as delta rightarrow 0 and establish various order optimal convergence rate results it is remarkable that we even can show the order optimality under merely the lipschitz condition on the frechet derivative f of f if x 0 x dagger is smooth enough

em and beyond the basic theme of the em algorithm to repeatedly use complete data methods to solve incomplete data problems is also a theme of several more recent statistical techniques these techniques multiple imputation data augmentation stochastic relaxation and sampling importance resampling combine simulation techniques with complete data methods to attack problems that are difficult or impossible for em

a pseudo heuristic parameter selection rule for l1 regularized minimization problems abstract this paper considers the regularization parameter determination of l 1 regularized minimization problem we solve the l 1 regularized problem using iterative reweighted least squares irls which involves solving a linear system whose coefficient matrix has the form m 1 n 0 1 the aim of this paper is to find an efficient and computationally inexpensive algorithm to both choose the regularization parameter and solve the l 1 regularized problem in order to achieve this we propose an irls algorithm with adaptive regularization parameter selection based on a heuristic parameter determination rule de boor s parameter selection criterion compared with some of the state of the art algorithms and parameter selection rules the numerical experiments show the efficiency and robustness of the proposed method

generic solutions for some perturbed optimization problem in non reflexive banach spaces abstract let z be a closed boundedly relatively weakly compact nonempty subset of a banach space x and j z r a lower semicontinuous function bounded from below if x 0 is a convex subset in x and x 0 has approximatively z property k then the set of all points x in x 0 z for which there exists z 0 z such that j z 0 x z 0 x and every sequence z n z satisfying lim n j z n x z n x for x contains a subsequence strongly convergent to an element of z is a dense g subset of x 0 z moreover under the assumption that x 0 is approximatively z strictly convex we show more namely that the set of all points x in x 0 z for which there exists a unique point z 0 z such that j z 0 x z 0 x and every sequence z n z satisfying lim n j z n x z n x for x converges strongly to z 0 is a dense g subset of x 0 z here x inf j z x z z z these extend s cobzas s result j math anal appl 243 2000 344 356

the statistical analysis of general processing tree models with the em algorithm multinomial processing tree models assume that an observed behavior category can arise from one or more processing sequences represented as branches in a tree these models form a subclass of parametric multinomial models and they provide a substantively motivated alternative to loglinear models we consider the usual case where branch probabilities are products of nonnegative integer powers in the parameters 0 s 1 and their complements 1 s a version of the em algorithm is constructed that has very strong properties first the e step and the m step are both analytic and computationally easy therefore a fast pc program can be constructed for obtaining mles for large numbers of parameters second a closed form expression for the observed fisher information matrix is obtained for the entire class third it is proved that the algorithm necessarily converges to a local maximum and this is a stronger result than for the exponential family as a whole fourth we show how the algorithm can handle quite general hypothesis tests concerning restrictions on the model parameters fifth we extend the algorithm to handle the read and cressie power divergence family of goodness of fit statistics the paper includes an example to illustrate some of these results

bayesian estimation of multinomial processing tree models with heterogeneity in participants and items multinomial processing tree mpt models are theoretically motivated stochastic models for the analysis of categorical data here we focus on a crossed random effects extension of the bayesian latent trait pair clustering mpt model our approach assumes that participant and item effects combine additively on the probit scale and postulates multivariate normal distributions for the random effects we provide a winbugs implementation of the crossed random effects pair clustering model and an application to novel experimental data the present approach may be adapted to handle other mpt models

why simple shrinkage is still relevant for redundant representations shrinkage is a well known and appealing denoising technique introduced originally by donoho and johnstone in 1994 the use of shrinkage for denoising is known to be optimal for gaussian white noise provided that the sparsity on the signal s representation is enforced using a unitary transform still shrinkage is also practiced with nonunitary and even redundant representations typically leading to very satisfactory results in this correspondence we shed some light on this behavior the main argument in this work is that such simple shrinkage could be interpreted as the first iteration of an algorithm that solves the basis pursuit denoising bpdn problem while the desired solution of bpdn is hard to obtain in general we develop a simple iterative procedure for the bpdn minimization that amounts to stepwise shrinkage we demonstrate how the simple shrinkage emerges as the first iteration of this novel algorithm furthermore we show how shrinkage can be iterated turning into an effective algorithm that minimizes the bpdn via simple shrinkage steps in order to further strengthen the denoising effect

a duality based splitting method for ell 1 tv image restoration with automatic regularization parameter choice a novel splitting method is presented for the ell 1 tv restoration of degraded images subject to impulsive noise the functional is split into an ell 2 tv denoising part and an ell 1 ell 2 deblurring part the dual problem of the relaxed functional is smooth with convex constraints and can be solved efficiently by applying an arrow hurwicz type algorithm to the augmented lagrangian formulation the regularization parameter is chosen automatically based on a balancing principle the accuracy the fast convergence and the robustness of the algorithm and the use of the parameter choice rule are illustrated on some benchmark images and compared with an existing method

analysis of discrete ill posed problems by means of the l curve when discrete ill posed problems are analyzed and solved by various numerical regularization techniques a very convenient way to display information about the regularized solution is to plot the norm or seminorm of the solution versus the norm of the residual vector in particular the graph associated with tikhonov regularization plays a central role the main purpose of this paper is to advocate the use of this graph in the numerical treatment of discrete ill posed problems the graph is characterized quantitatively and several important relations between regularized solutions and the graph are derived it is also demonstrated that several methods for choosing the regularization parameter are related to locating a characteristic l shaped corner of the graph

fixed point continuation for ell_1 minimization methodology and convergence we present a framework for solving the large scale ell 1 regularized convex minimization problem min x 1 mu f x our approach is based on two powerful algorithmic ideas operator splitting and continuation operator splitting results in a fixed point algorithm for any given scalar mu continuation refers to approximately following the path traced by the optimal value of x as mu increases in this paper we study the structure of optimal solution sets prove finite convergence for important quantities and establish q linear convergence rates for the fixed point algorithm applied to problems with f x convex but not necessarily strictly convex the continuation framework motivated by our convergence results is demonstrated to facilitate the construction of practical algorithms

a fast algorithm for sparse reconstruction based on shrinkage subspace optimization and continuation we propose a fast algorithm for solving the ell 1 regularized minimization problem min x in mathbb r n mu x 1 ax b 2 2 for recovering sparse solutions to an undetermined system of linear equations ax b the algorithm is divided into two stages that are performed repeatedly in the first stage a first order iterative shrinkage method yields an estimate of the subset of components of x likely to be nonzero in an optimal solution restricting the decision variables x to this subset and fixing their signs at their current values reduces the ell 1 norm x 1 to a linear function of x the resulting subspace problem which involves the minimization of a smaller and smooth quadratic function is solved in the second phase our code fpc as embeds this basic two stage algorithm in a continuation homotopy approach by assigning a decreasing sequence of values to mu this code exhibits state of the art performance in terms of both its speed and its ability to recover sparse signals

using em to obtain asymptotic variance covariance matrices the sem algorithm abstract the expectation maximization em algorithm is a popular and often remarkably simple method for maximum likelihood estimation in incomplete data problems one criticism of em in practice is that asymptotic variance covariance matrices for parameters e g standard errors are not automatic byproducts as they are when using some other methods such as newton raphson in this article we define and illustrate a procedure that obtains numerically stable asymptotic variance covariance matrices using only the code for computing the complete data variance covariance matrix the code for em itself and code for standard matrix operations the basic idea is to use the fact that the rate of convergence of em is governed by the fractions of missing information to find the increased variability due to missing information to add to the complete data variance covariance matrix we call this supplemented em algorithm the sem algorithm theory and particular examples reinforce the conclusion that the sem alg

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

porosity of perturbed optimization problems in banach spaces let x be a banach space and z a nonempty closed subset of x l etj z r be a lower semicontinuous function bounded from below this paper is concerned with the perturbed optimization problem infz z j z x z denoted by x j inf for x x in the case when x is compactly fully 2 convex it is proved in the present paper that the set of all points x in x for which there does not exist z0 z such that j z0 x z0 infz z j z x z is a porous set in x f urthermore ifx is assumed additionally to be compactly locally uniformly convex we verify that the set of all points x x z 0 such that the problem x j inf fails to be approximately compact is a porous set in x z 0 w herez 0 denotes the set of all z z such that z pz z moreover a counterexample to which some results of ni r x ni generic solutions for some perturbed optimization problem in nonreflexive banach space j math anal appl 302 2005 417 424 fail is provided

finding the observed information matrix when using the em algorithm a procedure is derived for extracting the observed information matrix when the em algorithm is used to find maximum likelihood estimates in incomplete data problems the technique requires computation of a complete data gradient vector or second derivative matrix but not those associated with the incomplete data likelihood in addition a method useful in speeding up the convergence of the em algorithm is developed two examples are presented

extensions of estimation methods using the em algorithm abstract the em algorithm described by dempster laid and rubin 1977 is reviewed with the purpose of clarifying several misconceptions in the statistical and econometric literature the clarifications lead to several applications of the algorithm to models that have appeared to be less tractable the relationship between the em algorithm and the method of scoring is also explained providing estimators of the score and the information from the em algorithm the em algorithm is extended to missing data problems and an estimation method based on simulations

h consensus control with spectrum constraints for stochastic multi agent systems subject to x u v dependent noises abstract this paper is concerned with the problem of h consensus control with spectrum constraints for continuous time multi agent systems mass with x u v dependent noises by means of the spectrum technique the definition of stability is introduced and the relationship between the convergence rate of state responses and the stability is discussed for the closed loop system the aim of the addressed problem is to design a static output feedback controller such that the closed loop system is stable and the prescribed h performance requirement is satisfied in virtue of the stochastic analysis method and kronecker product sufficient conditions are obtained for the existence of h consensus controller two examples are provided to illustrate the availability of the proposed techniques

existence and porosity for a class of perturbed optimization problems in banach spaces abstract let x be a banach space and z a nonempty closed subset of x let j z r be an upper semicontinuous function bounded from above this paper is concerned with the perturbed optimization problem sup z z j z x z which is denoted by x j sup we shall prove in the present paper that if z is a closed boundedly relatively weakly compact nonempty subset then the set of all x x for which the problem x j sup has a solution is a dense g subset of x in the case when x is uniformly convex and j is bounded we will show that the set of all points x in x for which there does not exist z 0 z such that j z 0 x z 0 sup z z j z x z is a porous subset of x and the set of all points x x z 0 such that there exists a maximizing sequence of the problem x j sup which has no convergent subsequence is a porous subset of x z 0 where z 0 denotes the set of all z z such that z is in the solution set of z j sup

heuristic regularization methods for numerical differentiation in this paper we use smoothing splines to deal with numerical differentiation some heuristic methods for choosing regularization parameters are proposed including the l curve method and the de boor method numerical experiments are performed to illustrate the efficiency of these methods in comparison with other procedures

regularization parameter selection for nonlinear iterative image restoration and mri reconstruction using gcv and sure based methods regularized iterative reconstruction algorithms for imaging inverse problems require selection of appropriate regularization parameter values we focus on the challenging problem of tuning regularization parameters for nonlinear algorithms for the case of additive possibly complex gaussian noise generalized cross validation gcv and weighted mean squared error mse approaches based on stein s unbiased risk estimate sure need the jacobian matrix of the nonlinear reconstruction operator representative of the iterative algorithm with respect to the data we derive the desired jacobian matrix for two types of nonlinear iterative algorithms a fast variant of the standard iterative reweighted least squares method and the contemporary split bregman algorithm both of which can accommodate a wide variety of analysis and synthesis type regularizers the proposed approach iteratively computes two weighted sure type measures predicted sure and projected sure which require knowledge of noise variance 2 and gcv which does not need 2 for these algorithms we apply the methods to image restoration and to magnetic resonance image mri reconstruction using total variation and an analysis type l1 regularization we demonstrate through simulations and experiments with real data that minimizing predicted sure and projected sure consistently lead to near mse optimal reconstructions we also observe that minimizing gcv yields reconstruction results that are near mse optimal for image restoration and slightly suboptimal for mri theoretical derivations in this paper related to jacobian matrix evaluations can be extended in principle to other types of regularizers and reconstruction algorithms

improved iteratively reweighted least squares for unconstrained smoothed ell_q minimization in this paper we first study ell q minimization and its associated iterative reweighted algorithm for recovering sparse vectors unlike most existing work we focus on unconstrained ell q minimization for which we show a few advantages on noisy measurements and or approximately sparse vectors inspired by the results in daubechies et al comm pure appl math 63 2010 pp 1 38 for constrained ell q minimization we start with a preliminary yet novel analysis for unconstrained ell q minimization which includes convergence error bound and local convergence behavior then the algorithm and analysis are extended to the recovery of low rank matrices the algorithms for both vector and matrix recovery have been compared to some state of the art algorithms and show superior performance on recovering sparse vectors and low rank matrices

hierarchical multinomial processing tree models a latent trait approach multinomial processing tree models are widely used in many areas of psychology a hierarchical extension of the model class is proposed using a multivariate normal distribution of person level parameters with the mean and covariance matrix to be estimated from the data the hierarchical model allows one to take variability between persons into account and to assess parameter correlations the model is estimated using bayesian methods with weakly informative hyperprior distribution and a gibbs sampler based on two steps of data augmentation estimation model checks and hypotheses tests are discussed the new method is illustrated using a real data set and its performance is evaluated in a simulation study

generic existence of solutions for some perturbed optimization problems abstract let x be a real banach space z a closed nonvoid subset of x and j z r a lower semicontinuous function bounded from below if x is reflexive and has the kadets property then the set of all x x for which there exists z0 z such that j z0 x z0 inf j z x z z z is g and dense in x this extends a result of k s lau 1978 indiana univ math j 27 791 795 on nearest points the case j 0