some aspects of the spline smoothing approach to non parametric regression curve fitting non parametric regression using cubic splines is an attractive flexible and widely applicable approach to curve estimation although the basic idea was formulated many years ago the method is not as widely known or adopted as perhaps it should be the topics and examples discussed in this paper are intended to promote the understanding and extend the practicability of the spline smoothing methodology particular subjects covered include the basic principles of the method the relation with moving average and other smoothing methods the automatic choice of the amount of smoothing and the use of residuals for diagnostic checking and model adaptation the question of providing inference regions for curves and for relevant properties of curves is approached via a finite dimensional bayesian formulation

splines in statistics abstract this is a survey article that attempts to synthesize a broad variety of work on splines in statistics splines are presented as a nonparametric function estimating technique after a general introduction to the theory of interpolating and smoothing splines splines are treated in the nonparametric regression setting the method of cross validation for choosing the smoothing parameter is discussed and the general multivariate regression surface estimation problem is addressed an extensive discussion of splines as nonparametric density estimators is followed by a discussion of their role in time series analysis a comparison of the spline and isotonic regression methodologies leads to a formulation of a hybrid estimator the closing section provides a brief overall summary and formulates a number of open unsolved problems relating to splines in statistics

a two stage filter for smoothing multivariate noisy data on unstructured grids abstract experimental data as well as numerical simulations are very often affected by noise random fluctuations that distort the final output or the intermediate products of a numerical process in this paper a new method for smoothing data given on nonstructured grids is proposed it takes advantage of the smoothing properties of kernel weighted averaging and least squares techniques the weighted averaging is performed following a shepard like procedure where gaussian kernels are employed while least squares fitting is reduced to the use of very few basis functions so as to improve smoothness though at the price of interpolation accuracy once we have defined an n point grid and want to make a smoothed fit at a given grid point this method reduces to consideration of all m point stencils for a given value of m where m n that include that point and to make a least squares fitting for that particular point in each of those stencils finally the various results thus obtained are weight averaged the weights being inversely proportional to the distance of the point to the middle of the stencil then this process is repeated for all the grid points so as to obtain the smoothing of the input function though this method is generalized for the multivariate case one two and five dimensional test cases are shown as examples of the performance of this method

local polynomial variance function estimation the conditional variance function in a heteroscedastic nonparametric regression model is estimated by linear smoothing of squared residuals attention is focused on local polynomial smoothers both the mean and variance functions are assumed to be smooth but neither is assumed to be in a parametric family the biasing effect of preliminary estimation of the mean is studied and a degrees of freedom correction of bias is proposed the corrected method is shown to be adaptive in the sense that the variance function can be estimated with the same asymptotic mean and variance as if the mean function were known a proposal is made for using standard bandwidth selectors for estimating both the mean and variance functions the proposal is illustrated with data from the lidar method of measuring atmospheric pollutants and from turbulence model computations

exploring regression structure using nonparametric functional estimation abstract average derivative functionals of regression are proposed for nonparametric model selection and diagnostics the functionals are of the integral type which under certain conditions allows their estimation at the usual parametric rate of n 1 2 we analyze asymptotic properties of the estimators of these functionals based on kernel regression these estimators can then be used for assessing the validity of various restrictions imposed on the form of regression in particular we show how they could be used to reduce the dimensionality of the model assess the relative importance of predictors measure the extent of nonlinearity and nonadditivity and under certain conditions help identify projection directions in projection pursuit models and decide on the number of these directions

brief tracking time varying parameters with local regression this paper shows that the recursive least squares rls algorithm with forgetting factor is a special case of a varying coefficient model and a model which can easily be estimated via simple local regression this observation allows us to formulate a new method which retains the rls algorithm but extends the algorithm by including polynomial approximations simulation results are provided which indicates that this new method is superior to the classical rls method if the parameter variations are smooth

spline smoothing the equivalent variable kernel method on considere une approche spline pour la regression non parametrique et l estimation de courbe on montre que dans un certain sens le lissage spline correspond approximativement au lissage par une methode du noyau avec une largeur de bande dependant de la densite locale des points du plan de regression

nonlinear system identification via direct weight optimization a general framework for estimating nonlinear functions and systems is described and analyzed in this paper identification of a system is seen as estimation of a predictor function the considered predictor function estimate at a particular point is defined to be affine in the observed outputs and the estimate is defined by the weights in this expression for each given point the maximal mean square error or an upper bound of the function estimate over a class of possible true functions is minimized with respect to the weights which is a convex optimization problem this gives different types of algorithms depending on the chosen function class it is shown how the classical linear least squares is obtained as a special case and how unknown but bounded disturbances can be handled most of the paper deals with the method applied to locally smooth predictor functions it is shown how this leads to local estimators with a finite bandwidth meaning that only observations in a neighborhood of the target point will be used in the estimate the size of this neighborhood the bandwidth is automatically computed and reflects the noise level in the data and the smoothness priors the approach is applied to a number of dynamical systems to illustrate its potential

streamflow simulation a nonparametric approach in this paper kernel estimates of the joint and conditional probability density functions are used to generate synthetic streamflow sequences streamflow is assumed to be a markov process with time dependence characterized by a multivariate probability density function kernel methods are used to estimate this multivariate density function simulation proceeds by sequentially resampling from the conditional density function derived from the kernel estimate of the underlying multivariate probability density function this is a nonparametric method for the synthesis of streamflow that is data driven and avoids prior assumptions as to the form of dependence e g linear or nonlinear and the form of the probability density functions e g gaussian we show using synthetic examples with known underlying models that the nonparametric method presented is more flexible than the conventional models used in stochastic hydrology and is capable of reproducing both linear and nonlinear dependence the effectiveness of this model is illustrated through its application to simulation of monthly streamflow from the beaver river in utah

prediction and creation of smooth curves for temporally correlated longitudinal data abstract this article presents a method of obtaining smoothed curves for a sample of individuals that permits an arbitrary number and spacing of observations for each individual we consider the case where each individual s curve cannot be separately estimated because either the n i s are too small or no suitable parametric forms for the random effects are available the model assumes a parametric form for the population mean curve and the correlation of the repeated measures the assumed correlation structure is evaluated using the empirical semivariogram a function of the sum of the squared differences of within individual residuals a method is proposed to validate the form and stationarity of the correlation structure maximum likelihood estimates for the population mean parameters and variance components are obtained simultaneously these estimates may be used to create a semiparametric differentiable curve and to predict future values for each individual using a method called kriging this method

linear smoothers and additive models we study linear smoothers and their use in building nonparametric regression models in the first part of this paper we examine certain aspects of linear smoothers for scatterplots examples of these are the running mean and running line kernel and cubic spline smoothers the eigenvalue and singular value decompositions of the corresponding smoother matrix are used to describe qualitatively a smoother and several other topics such as the number of degrees of freedom of a smoother are discussed in the second part of the paper we describe how linear smoothers can be used to estimate the additive model a powerful nonparametric regression model using the backfitting algorithm we show that backfitting is the gauss seidel iterative method for solving a set of normal equations associated with the additive model we provide conditions for consistency and nondegeneracy and prove convergence for the backfitting and related algorithms for a class of smoothers that includes cubic spline smoothers

locally adaptive metric nearest neighbor classification nearest neighbor classification assumes locally constant class conditional probabilities this assumption becomes invalid in high dimensions with finite samples due to the curse of dimensionality severe bias can be introduced under these conditions when using the nearest neighbor rule we propose a locally adaptive nearest neighbor classification method to try to minimize bias we use a chi squared distance analysis to compute a flexible metric for producing neighborhoods that are highly adaptive to query locations neighborhoods are elongated along less relevant feature dimensions and constricted along most influential ones as a result the class conditional probabilities are smoother in the modified neighborhoods whereby better classification performance can be achieved the efficacy of our method is validated and compared against other techniques using both simulated and real world data

the l 1 method for robust nonparametric regression abstract consider the problem of estimating the regression function underlying a set of data that is contaminated by a heavy tailed error distribution the two standard approaches to such a problem are each flawed robust parametric least squares is appropriate only if there is good reason to believe that the underlying function has some particular form whereas most nonparametric regression methods are asymptotically equivalent to kernel regression methods which are not resistant against outliers existing algorithms for robustifying nonparametric regression procedures use either nonlinear optimization of an influence function or iterative solution of local polynomial fitting using reweighted least squares neither of these two approaches combines computational ease with asymptotic theoretical results furthermore application of the robust procedure has been limited almost exclusively to the case of a single explanatory variable with the response variable in this article a new hybrid method is propose

generalized wavelets design using kernel methods application to signal processing abstract multiresolution representations of data are powerful tools in signal processing in harten s framework multiresolution transforms are defined by predicting finer resolution levels of information from coarser ones using an operator called the prediction operator and defining details or wavelet coefficients that are the difference between the exact values and the predicted values in this paper we present a multiresolution scheme using local polynomial regression theory in order to design a more accurate prediction operator the stability of the scheme is proved and the order of the method is calculated finally some results are presented comparing our method with the classical methods

optimal geostatistical model selection in many fields of science predicting variables of interest over a study region based on noisy data observed at some locations is an important problem two popular methods for the problem are kriging and smoothing splines the former assumes that the underlying process is stochastic whereas the latter assumes it is purely deterministic kriging performs better than smoothing splines in some situations but is outperformed by smoothing splines in others however little is known regarding selecting between kriging and smoothing splines in addition how to perform variable selection in a geostatistical model has not been well studied in this article we propose a general methodology for selecting among arbitrary spatial prediction methods based on approximately unbiased estimation of mean squared prediction errors using a data perturbation technique the proposed method accounts for estimation uncertainty in both kriging and smoothing spline predictors and is shown to be optimal in terms of two mean sq

forced vibration of two beams joined with a non linear rotational joint clamped and simply supported end conditions abstract an analytical model is proposed which consists of two euler bernoulli beams joined by a torsional spring with linear and cubic stiffness the method of harmonic balance is used to find an approximate solution supported and clamped end conditions specifically a one term harmonic balance solution is developed for two beams with unsymmetric properties and symmetric distributed loading solutions are given for three cases no non linearity and two cases with different levels of non linearity for all cases frequency amplitude relations are given and the effects of the cubic non linearity are observed and discussed

variable selection in nonparametric regression with continuous covariates in a nonparametric regression setup where the covariates are continuous the problem of estimating the number of covariates will be discussed in this paper the kernel method is used to estimate the regression function and the selection criterion is based on minimizing the cross validation estimate of the mean squared prediction error we consider choosing both the bandwidth and the number of covariates based on the data unlike the case of linear regression it turns out that the selection is consistent and efficient even when the true model has only a finite number of covariates in addition we also observe the curse of dimensionality at work

local discriminative learning for pattern recognition abstract local discriminative learning methods approximate a target function a posteriori class probability function directly by partitioning the feature space into a set of local regions and appropriately modeling a simple input output relationship function in each one this paper presents a new method for judiciously partitioning the input feature space in order to accurately represent the target function the method accomplishes this by approximating not only the target function itself but also its derivatives as such the method partitions the input feature space along those dimensions for which the class probability function changes most rapidly thus minimizing bias the efficacy of the method is validated using a variety of simulated and real world data

testing for cubic smoothing splines under dependent data in most research on smoothing splines the focus has been on estimation while inference especially hypothesis testing has received less attention by defining design matrices for fixed and random effects and the structure of the covariance matrices of random errors in an appropriate way the cubic smoothing spline admits a mixed model formulation which places this nonparametric smoother firmly in a parametric setting thus nonlinear curves can be included with random effects and random coefficients the smoothing parameter is the ratio of the random coefficient and error variances and tests for linear regression reduce to tests for zero random coefficient variances we propose an exact f test for the situation and investigate its performance in a real pine stem data set and by simulation experiments under certain conditions the suggested methods can also be applied when the data are dependent

some diagnostics for binary logistic regression via smoothing summary the paper discusses some diagnostic tools for binary logistic regression which use smoothing techniques smoothed binary data are employed to devise a battery of diagnostic plots from simple procedures which indicate only the adequacy or inadequacy of a maximum likelihood logistic fit to partial residual plots and delta local deviance plots which reveal the type of model inadequacy suggesting perhaps the addition of nonlinear terms or interactions properties of the methods are studied by simulation the diagnostic tools are illustrated on a set of data in which the probability of having telephone service is related to the demographic variables income and education

a semiparametric approach to analysing dose response data in the analysis of a quantal dose response experiment with grouped data the most commonly used parametric procedure is logistic regression commonly referred to as logit analysis the adequacy of the fit by the logistic regression curve is tested using the chi square lack of fit test if the lack of fit test is not significant then the logistic model is assumed to be adequate and estimation of effective doses and confidence intervals on the effective doses can be made when the tolerance distribution of the dose response data is not known and cannot be assumed by the user one can use non parametric methods such as kernel regression or local linear regression to estimate the dose response curve effective doses and confidence intervals this research proposes another alternative based on semi parametric regression to analysing quantal dose response data called model robust quantal regression mrqr mrqr linearly combines the parametric and non parametric predictions with the use of a mixing parameter mrqr uses logistic regression as the parametric portion of the model and local linear regression as the non parametric portion of the model our research has shown that the mrqr procedure can improve the fit of the dose response curve by producing narrower confidence intervals for predictions while providing improved precision of estimates of the effective doses with respect to either logistic or local linear regression results

dimension reduction by local principal component analysis reducing or eliminating statistical redundancy between the components of high dimensional vector data enables a lower dimensional representation without significant loss of information recognizing the limitations of principal component analysis pca researchers in the statistics and neural network communities have developed nonlinear extensions of pca this article develops a local linear approach to dimension reduction that provides accurate representations and is fast to compute we exercise the algorithms on speech and image data and compare performance with pca and with neural network implementations of nonlinear pca we find that both nonlinear techniques can provide more accurate representations than pca and show that the local linear techniques outperform neural network implementations

an adaptable k nearest neighbors algorithm for mmse image interpolation we propose an image interpolation algorithm that is nonparametric and learning based primarily using an adaptive k nearest neighbor algorithm with global considerations through markov random fields the empirical nature of the proposed algorithm ensures image results that are data driven and hence reflect ldquoreal worldrdquo images well given enough training data the proposed algorithm operates on a local window using a dynamic k nearest neighbor algorithm where k differs from pixel to pixel small for test points with highly relevant neighbors and large otherwise based on the neighbors that the adaptable k provides and their corresponding relevance measures a weighted minimum mean squared error solution determines implicitly defined filters specific to low resolution image content without yielding to the limitations of insufficient training additionally global optimization via single pass markov approximations similar to cited nearest neighbor algorithms provides additional weighting for filter generation the approach is justified in using a sufficient quantity of training per test point and takes advantage of image properties for in depth analysis we compare to existing methods and draw parallels between intuitive concepts including classification and ideas introduced by other nearest neighbor algorithms by explaining manifolds in low and high dimensions

some comments on cp i study the typical configuration of a cp plot when the number of variables in the regression problem is large and there are many weak effects i show that a particular configuration that is very commonly seen can arise in a simple way i give a formula by means of which the risk incurred by the minimum cp rule can be estimated

locally weighted polynomial regression parameter choice and application to forecasts of the great salt lake 1 relationships between hydrologic variables are often nonlinear usually the functional form of such a relationship is not known a priori a multivariate nonparametric regression methodology is provided here for approximating the underlying regression function using locally weighted polynomials locally weighted polynomials consider the approximation of the target function through a taylor series expansion of the function in the neighborhood of the point of estimate cross validatory procedures for the selection of the size of the neighborhood over which this approximation should take place and for the order of the local polynomial to use are provided and shown for some simple situations the utility of this nonparametric regression approach is demonstrated through an application to nonparametric short term forecasts of the biweekly great salt lake volume blind forecasts up to 1 year in the future using the 1847 2004 time series of the great salt lake are presented

interpolation methods for adapting to sparse design in nonparametric regression abstract we suggest interpolation methods for overcoming the problem of sparse design in local linear smoothing these methods are based on simple rules determined by the kernel and bandwidth for deciding when and where pseudo design points should be added to augment the original design sequence new ordinates for the added design points are computed by simple interpolation then local linear smoothing is applied directly to the expanded dataset the method is competitive with alternatives e g those involving ridge regression in terms of both simplicity and performance

acoustic articulatory mapping in vowels by locally weighted regression a method for mapping between simultaneously measured articulatory and acoustic data is proposed the method uses principal components analysis on the articulatory and acoustic variables and mapping between the domains by locally weighted linear regression or loess cleveland w s 1979 j am stat assoc 74 829 836 the latter method permits local variation in the slopes of the linear regression assuming that the function being approximated is smooth the methodology is applied to vowels of four speakers in the wisconsin x ray microbeam speech production database with formant analysis results are examined in terms of 1 examples of forward articulation to acoustics mappings and inverse mappings 2 distributions of local slopes and constants 3 examples of correlations among slopes and constants 4 root mean square error and 5 sensitivity of formant frequencies to articulatory change it is shown that the results are qualitatively correct and that loess performs better than global regression the forward mappings show different root mean square error properties than the inverse mappings indicating that this method is better suited for the forward mappings than the inverse mappings at least for the data chosen for the current study some preliminary results on sensitivity of the first two formant frequencies to the two most important articulatory principal components are presented

regression by local fitting methods properties and computational algorithms abstract local regression is a procedure for estimating regression surfaces by the local fitting of linear or quadratic functions of the independent variables in a moving fashion that is analogous to how a moving average is computed for a time series the advantage of the methodology over the global fitting of parametric functions of the independent variables by least squares the current paradigm in regression studies is that a much wider class of regression functions can be estimated without distortion in this paper we discuss the methods their statistical properties and computational algorithms

locally weighted regression an approach to regression analysis by local fitting abstract locally weighted regression or loess is a way of estimating a regression surface through a multivariate smoothing procedure fitting a function of the independent variables locally and in a moving fashion analogous to how a moving average is computed for a time series with local fitting we can estimate a much wider class of regression surfaces than with the usual classes of parametric functions such as polynomials the goal of this article is to show through applications how loess can be used for three purposes data exploration diagnostic checking of parametric models and providing a nonparametric regression surface along the way the following methodology is introduced a a multivariate smoothing procedure that is an extension of univariate locally weighted regression b statistical procedures that are analogous to those used in the least squares fitting of parametric functions c several graphical methods that are useful tools for understanding loess estimates and checking the a

a tree structured adaptive network for function approximation in high dimensional spaces nonlinear function approximation is often solved by finding a set of coefficients for a finite number of fixed nonlinear basis functions however if the input data are drawn from a high dimensional space the number of required basis functions grows exponentially with dimension leading many to suggest the use of adaptive nonlinear basis functions whose parameters can be determined by iterative methods the author proposes a technique based on the idea that for most of the data only a few dimensions of the input may be necessary to compute the desired output function additional input dimensions are incorporated only where needed the learning procedure grows a tree whose structure depends upon the input data and the function to be approximated this technique has a fast learning algorithm with no local minima once the network shape is fixed and it can be used to reduce the number of required measurements in situations where there is a cost associated with sensing three examples are given controlling the dynamics of a simulated planar two joint robot arm predicting the dynamics of the chaotic mackey glass equation and predicting pixel values in real images from pixel values above and to the left

a unified model for probabilistic principal surfaces principal curves and surfaces are nonlinear generalizations of principal components and subspaces respectively they can provide insightful summary of high dimensional data not typically attainable by classical linear methods solutions to several problems such as proof of existence and convergence faced by the original principal curve formulation have been proposed in the past few years nevertheless these solutions are not generally extensible to principal surfaces the mere computation of which presents a formidable obstacle consequently relatively few studies of principal surfaces are available we previously 2000 proposed the probabilistic principal surface pps to address a number of issues associated with current principal surface algorithms pps uses a manifold oriented covariance noise model based on the generative topographical mapping gtm which can be viewed as a parametric formulation of kohonen s self organizing map building on the pps we introduce a unified covariance model that implements pps 0 1 by varying the clamping parameter spl alpha then we comprehensively evaluate the empirical performance of pps gtm and the manifold aligned gtm on three popular benchmark data sets it is shown in two different comparisons that the pps outperforms the gtm under identical parameter settings convergence of the pps is found to be identical to that of the gtm and the computational overhead incurred by the pps decreases to 40 percent or less for more complex manifolds these results show that the generalized pps provides a flexible and effective way of obtaining principal surfaces

testing for heterogeneity of evoked potential signals using an approximation to an exact permutation test in brain evoked potential research and various other fields a signal embedded in noise is observed in each of many repeated trials we propose a new test of the null hypothesis that the signal is the same in every trial the test has power against the alternative hypothesis that the signal is changing slowly from trial to trial the test statistic is the distance between the average of the trials and a parametric or nonparametric estimate of a slowly changing signal we derive the exact mean and variance of the test statistic under permutation of the trials and use these moments to approximate the exact permutation test we use simulated evoked potential data to evaluate the proposed test and to compare it to three previous tests

filtering out high frequencies in time series using f transform abstract in this paper we focus on application of fuzzy transform f transform to analysis of time series under the assumption that the latter can be additively decomposed into trend cycle seasonal component and noise we prove that when setting properly width of the basic functions the inverse f transform of the time series closely approximates its trend cycle this means that the f transform almost completely removes the seasonal component and noise the obtained theoretical results are demonstrated on two artificial time series whose trend cycle is precisely known and on three real time series at the same time comparison with three classical methods namely stl method ssa method and low pass butterworth filter is also provided

ats methods nonparametric regression for non gaussian data abstract ats methods provide an approach to fitting curves and surfaces to data using nonparametric regression when distributions are not necessarily gaussian first a small amount of local averaging the a in ats is carried out then a variance stabilizing transformation is applied t and finally the result is smoothed s using a nonparametric regression procedure ats methods are quite broad in terms of applications in this article we show how they can be used for fitting a surface when the response is binary for estimating density and for estimating the spectrum of a time series we also present some theoretical investigations that give guidance on how to choose the amount of averaging and how efficient the methods are

conditional parametric models for storm sewer runoff 1 the method of conditional parametric modeling is introduced for flow prediction in a sewage system it is a well known fact that in hydrological modeling the response runoff to input precipitation varies depending on soil moisture and several other factors consequently nonlinear input output models are needed the model formulation described in this paper is similar to the traditional linear models like final impulse response fir and autoregressive exogenous arx except that the parameters vary as a function of some external variables the parameter variation is modeled by local lines using kernels for local linear regression as such the method might be referred to as a nearest neighbor method the results achieved in this study were compared to results from the conventional linear methods fir and arx the increase in the coefficient of determination is substantial furthermore the new approach conserves the mass balance better hence this new approach looks promising for various hydrological models and analysis

application of orthonormal transforms to implementation of quasi lms newton algorithm an efficient implementation of the lms newton lmsn algorithm is proposed the scheme uses a quasi inverse of the correlation matrix of the input instead of its exact inverse the proposed algorithm which is an alternative formulation of a class of transform domain adaptive filters tdafs has some advantages over the tdaf a feature of the scheme that greatly simplifies its implementations is the possibility of coarse quantization of the stochastic gradient terms in the adjustment recursion to back this up an analysis of the lmsn algorithm which includes the effect of coarse quantization of the stochastic gradient terms is given for correlated gaussian data computer simulation results that support the developed theories are also presented

preaveraged localized orthogonal polynomial estimators for surface smoothing and partial differentiation abstract we propose a multivariate smoothing method based on products of localized orthogonal polynomial series estimators for a smooth regression surface in the fixed design regression model the estimation of partial derivatives is included the proposed method provides for automatic and efficient boundary modifications near the edges of the surface assuming that the boundary of the support of the regression function satisfies some regularity conditions by allowing for a preaveraging step the corresponding algorithms are speeded up considerably and are easy to implement computation of special boundary kernels as required by the kernel method to avoid edge effects is not necessary it is shown that under sufficient smoothness assumptions the global average mean squared error has the same optimal rate of convergence as the mean squared error at an interior point that is the boundary correction is asymptotically effective the method depends on two smoothing parameters one determining the amount

graphics for assessing the adequacy of regression models abstract graphical paradigms for assessing the adequacy of nearly any regression model are discussed the fundamental idea is to examine the fit of a model using a sequence of marginal model plots on each plot nonparametric estimates of fit derived from the model are compared to nonparametric fits based on the observed data several examples are given