neural networks for functional approximation and system identification we construct generalized translation networks to uniformly approximate a class of nonlinear continuous functionals defined on l p 1 1 s for integer s 1 1 p 1 or c 1 1 s we obtain lower bounds on the possible order of approximation for such functionals in terms of any approximation process depending continuously upon a given number of parameters our networks almost achieve this order of approximation in terms of the number of parameters neurons involved in the network the training is simple and noniterative in particular we avoid any optimization such as that involved in the usual back propagation

stability of numerical methods for delay differential equations abstract consider the following delay differential equation dde y t y t y t t t t 0 with the initial condition y t for t t 0 where and are such that 0 1 0 2 has a unique solution y t the author gives sufficient conditions for the asymptotic stability of the equation 0 1 for which he introduces new definitions of numerical stability the approach is reminiscent of that from the nonlinear stiff ordinary differential equation ode field he investigates stability properties of the class of one point collocation rules in particular the backward euler method turns out to be stable with respect to all the given definitions

learning with box kernels supervised examples and prior knowledge on regions of the input space have been profitably integrated in kernel machines to improve the performance of classifiers in different real world contexts the proposed solutions which rely on the unified supervision of points and sets have been mostly based on specific optimization schemes in which as usual the kernel function operates on points only in this paper arguments from variational calculus are used to support the choice of a special class of kernels referred to as box kernels which emerges directly from the choice of the kernel function associated with a regularization operator it is proven that there is no need to search for kernels to incorporate the structure deriving from the supervision of regions of the input space because the optimal kernel arises as a consequence of the chosen regularization operator although most of the given results hold for sets we focus attention on boxes whose labeling is associated with their propositional description based on different assumptions some representer theorems are given that dictate the structure of the solution in terms of box kernel expansion successful results are given for problems of medical diagnosis image and text categorization

a fully discrete spectral method for delay differential equations in this paper a new lanczos tau method for solving linear functional differential equations is introduced the scheme has infinite order of accuracy both in time and in the delayed argument the high accuracy in time is obtained without increasing the computational work and memory space which is needed for a one step explicit difference scheme the article demonstrates how to implement the algorithm in a robust and efficient manner and to treat problems with piecewise continuous initial function numerical results illustrating the behavior of the method when faced with difficult problems are presented and the numerical results are compared to those obtained by using two other methods

multivariate smoothing spline functions given data z i g t i varepsilon i 1 leqq i leqq n where g is the unknown function the t i are known d dimensional variables in a domain omega and the varepsilon i are i

spline smoothing the equivalent variable kernel method on considere une approche spline pour la regression non parametrique et l estimation de courbe on montre que dans un certain sens le lissage spline correspond approximativement au lissage par une methode du noyau avec une largeur de bande dependant de la densite locale des points du plan de regression

regularization theory and neural networks architectures we had previously shown that regularization principles lead to approximation schemes that are equivalent to networks with one layer of hidden units called regularization networks in particular s

generalized additive models some applications abstract generalized additive models have the form x fj x j where might be the regression function in a multiple regression or the logistic transformation of the posterior probability pr y 1 x in a logistic regression in fact these models generalize the whole family of generalized linear models x x where x g x is some transformation of the regression function we use the local scoring algorithm to estimate the functions fj xj nonparametrically using a scatterplot smoother as a building block we demonstrate the models in two different analyses a nonparametric analysis of covariance and a logistic regression the procedure can be used as a diagnostic tool for identifying parametric transformations of the covariates in a standard linear analysis a variety of inferential tools have been developed to aid the analyst in assessing the relevance and significance of the estimated functions these include confidence curves degrees of freedom estimates and approximat

neural networks for optimal approximation of smooth and analytic functions we prove that neural networks with a single hidden layer are capable of providing an optimal order of approximation for functions assumed to possess a given number of derivatives if the activation function evaluated by each principal element satisfies certain technical conditions under these conditions it is also possible to construct networks that provide a geometric order of approximation for analytic target functions the permissible activation functions include the squashing function 1 e x 1 as well as a variety of radial basis functions our proofs are constructive the weights and thresholds of our networks are chosen independently of the target function we give explicit formulas for the coefficients as simple continuous linear functionals of the target function

regression learning based on incomplete relationships between attributes abstract in recent years machine learning researchers have focused on methods to construct flexible and interpretable regression models however the method of obtaining complete knowledge from incomplete and fuzzy prior knowledge and the trade off between the generalization performance and the interpretability of the model are very important factors to consider in this paper we propose a new regression learning method complete relationships are obtained from the incomplete fuzzy relationships between attributes by using markov logic networks 29 the complete relationships are then applied to constrain the shape of the regression model in the optimization procedure to solve the trade off problem finally the benefits of our approach are illustrated on benchmark data sets and in real world experiments

linear smoothers and additive models we study linear smoothers and their use in building nonparametric regression models in the first part of this paper we examine certain aspects of linear smoothers for scatterplots examples of these are the running mean and running line kernel and cubic spline smoothers the eigenvalue and singular value decompositions of the corresponding smoother matrix are used to describe qualitatively a smoother and several other topics such as the number of degrees of freedom of a smoother are discussed in the second part of the paper we describe how linear smoothers can be used to estimate the additive model a powerful nonparametric regression model using the backfitting algorithm we show that backfitting is the gauss seidel iterative method for solving a set of normal equations associated with the additive model we provide conditions for consistency and nondegeneracy and prove convergence for the backfitting and related algorithms for a class of smoothers that includes cubic spline smoothers

information theoretic shape matching in this paper we describe two related algorithms that provide both rigid and non rigid point set registration with different computational complexity and accuracy the first algorithm utilizes a nonlinear similarity measure known as correntropy the measure combines second and high order moments in its decision statistic showing improvements especially in the presence of impulsive noise the algorithm assumes that the correspondence between the point sets is known which is determined with the surprise metric the second algorithm mitigates the need to establish a correspondence by representing the point sets as probability density functions pdf the registration problem is then treated as a distribution alignment the method utilizes the cauchy schwarz divergence to measure the similarity distance between the point sets and recover the spatial transformation function needed to register them both algorithms utilize information theoretic descriptors however correntropy works at the realizations level whereas cauchy schwarz divergence works at the pdf level this allows correntropy to be less computationally expensive and for correct correspondence more accurate the two algorithms are robust against noise and outliers and perform well under varying levels of distortion they outperform several well known and state of the art methods for point set registration

training a support vector machine in the primal most literature on support vector machines svms concentrates on the dual optimization problem in this letter we point out that the primal problem can also be solved efficiently for both linear and nonlinear svms and that there is no reason for ignoring this possibility on the contrary from the primal point of view new families of algorithms for large scale svm training can be investigated

the numerical stability of linear multistep methods for delay differential equations with many delays this paper deals with the stability analysis of linear multistep methods for the numerical solution of delay differential equations ddss with many delays we focus on the stability behaviour of such methods when they are applied to a scalar linear dde with many constant delays and complex coefficients the concept of gp m stability is introduced under a sufficient condition for the test equation to be stable the linear multistep method when combined with a certain lagrange interpolation must be stable for all stepsizes it is proven that a linear multistep method is gp m stable if and only if it is a stable for ordinary differential equations odes

robust full bayesian learning for radial basis networks we propose a hierarchical full bayesian model for radial basis networks this model treats the model dimension number of neurons model parameters regularization parameters and noise parameters as unknown random variables we develop a reversible jump markov chain monte carlo mcmc method to perform the bayesian computation we find that the results obtained using this method are not only better than the ones reported previously but also appear to be robust with respect to the prior specification in addition we propose a novel and computationally efficient reversible jump mcmc simulated annealing algorithm to optimize neural networks this algorithm enables us to maximize the joint posterior distribution of the network parameters and the number of basis function it performs a global search in the joint space of the parameters and number of parameters thereby surmounting the problem of local minima to a large extent we show that by calibrating the full hierarchical bayesian prior we can obtain the classical akaike information criterion bayesian information criterion and minimum description length model selection criteria within a penalized likelihood framework finally we present a geometric convergence theorem for the algorithm with homogeneous transition kernel and a convergence theorem for the reversible jump mcmc simulated annealing method

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

quantized kernel least mean square algorithm in this paper we propose a quantization approach as an alternative of sparsification to curb the growth of the radial basis function structure in kernel adaptive filtering the basic idea behind this method is to quantize and hence compress the input or feature space different from sparsification the new approach uses the redundant data to update the coefficient of the closest center in particular a quantized kernel least mean square qklms algorithm is developed which is based on a simple online vector quantization method the analytical study of the mean square convergence has been carried out the energy conservation relation for qklms is established and on this basis we arrive at a sufficient condition for mean square convergence and a lower and upper bound on the theoretical value of the steady state excess mean square error static function estimation and short term chaotic time series prediction examples are presented to demonstrate the excellent performance

learning with boundary conditions kernel machines traditionally arise from an elegant formulation based on measuring the smoothness of the admissible solutions by the norm in the reproducing kernel hilbert space rkhs generated by the chosen kernel it was pointed out that they can be formulated in a related functional framework in which the green s function of suitable differential operators is thought of as a kernel in this letter our own picture of this intriguing connection is given by emphasizing some relevant distinctions between these different ways of measuring the smoothness of admissible solutions in particular we show that for some kernels there is no associated differential operator the crucial relevance of boundary conditions is especially emphasized which is in fact the truly distinguishing feature of the approach based on differential operators we provide a general solution to the problem of learning from data and boundary conditions and illustrate the significant role played by boundary conditions with examples it turns out that the degree of freedom that arises in the traditional formulation of kernel machines is indeed a limitation which is partly overcome when incorporating the boundary conditions this likely holds true in many real world applications in which there is prior knowledge about the expected behavior of classifiers and regressors on the boundary

stability and convergence of the two parameter cubic spline collocation method for delay differential equations in this paper we propose the cubic spline collocation method with two parameters for solving delay differential equations ddes some results of the local truncation error and the convergence of the spline collocation method are given we also obtain some results of the linear stability and the nonlinear stability of the method for ddes in particular we design an algorithm to obtain the ranges of the two parameters a b which are necessary for the p stability of the collocation method some illustrative examples successfully verify our theoretical results

identification of non linear systems using empirical data and prior knowledge an optimization approach abstract the choice of a parametric model structure in empirical and semi empirical non linear modeling is usually viewed as an important and critical step however it is known that by augmenting the least squares identification criterion with a term that imposes a penalty on the non smoothness of the model an optimal non parametric model can be found explicitly the optimal non parametric model will depend on the particular form of the penalty which can be looked upon as a priori knowledge or the desired properties of the model in this paper these results are extended in several directions i we show how useful types of prior knowledge other than smoothness can be included as a term in the criterion or as a constraint and how this influences the optimal model ii dynamic models and a general prediction error penalty are considered iii we present a practical numerical procedure for the identification of a close to optimal semi parametric model the numerical approach is motivated by the difficulty of deriving the optimal non parametric model if there are complicated constraints or penalty terms in the criterion and finally iv we discuss determination of the appropriate model complexity through selecting weights on the different penalties on the basis of empirical data since the optimal non parametric model is identified using an augmented optimization criterion and all prior knowledge and desired model properties may be specified through the augmented optimization criterion it is the choice of this criterion that is critical in this approach this elevates the empirical and semi empirical modeling problems to a more transparent and engineering friendly level than is achieved by directly specifying a parametric model structure hence our results shed some more light on modern non linear empirical modeling approaches like radial basis functions splines and neural networks in addition to providing a fundamental insight into the role of different kinds of prior knowledge this high level formulation is also attractive from a practical point of view as it lies closer to engineering thinking and less guesswork is required the flexibility and power of this approach are illustrated with a semi realistic simulation example

numerical treatment of delay differential equations by hermite interpolation a class of numerical methods for the treatment of delay differential equations is developed these methods are based on the wellknown runge kutta fehlberg methods the retarded argument is approximated by an appropriate multipoint hermite interpolation the inherent jump discontinuities in the various derivatives of the solution are considered automatically problems with piecewise continuous right hand side and initial function are treated too real life problems are used for the numerical test and a comparison with other methods published in literature

using multirate architectures in realizing quadratic volterra kernels multirate architectures have been used for realizing linear fir digital filters with reduced computational complexity the volterra kernel can be represented as a generalized convolution it would thus be expected that multirate architectures could be used to advantage in realizing volterra kernels as well the quadratic volterra kernel may be realized in the form of an ldl structure the ldl structure includes a set of fir filters of increasing length which may be realized in a computationally efficient manner using multirate architectures

some extensions of radial basis functions and their applications in artificial intelligence abstract radial basis functions have recently found interesting applications in artificial intelligence and in particular in the problem of learning to perform a particular task from a set of examples however in many practical cases the radial basis functions method cannot be applied in a straight forward manner because it does not take into account some features that are typical of the problem of learning from examples in this paper we show some extensions of the standard theory introduced in order to deal with a large number of examples and with problems in which different variables play very different roles we present some examples and also point out some open problems

projection based approximation and a duality with kernel methods projection pursuit regression and kernel regression are methods for estimating a smooth function of several variables from noisy data obtained at scattered sites methods based on local averaging can perform poorly in high dimensions curse of dimensionality intuition and examples have suggested that projection based approaches can provide better fits for what sorts of regression functions is this true when and by how much do projection methods reduce the curse of dimensionality we make a start by focusing on the two dimensional problem and study the l 2 approximation error bias of the two procedures with respect to gaussian measure let ra stand for a certain ppr type approximation and ka for a particular kernel type approximation building on a simple but striking duality for polynomials we show that ra behaves significantly better than the minimax rate of approximation for radial functions while ka performs significantly better than the minimax rate for harmonic functions in fact the rate improvements carry over to large classes ra behaving very well for functions with enough angular smoothness oscillating slowly with angle while ka behaves very well for functions with enough laplacian smoothness oscillations averaging out locally the rate improvements matter they are equivalent to lowering the dimensionality of the problem for example for functions with nice tail behavior ra behaves as if the dimensionality of the problem were 1 5 rather than its nominal value 2 also ra and ka are complementary for a given function if one method offers a dimensionality reduction the other does not

the stability of difference formulas for delay differential equations a new simple geometric technique is presented for analyzing the stability of difference formulas for the model delay differential equation y t py t qy t 8 where p and q are complex constants and the delay 8 is a positive constant the technique is based on the argument principle and directly relates the region of absolute stability for ordinary differential equations corresponding to the py t term with the region corresponding to the delay term qy t 8 a sufficient condition for stability is that these regions be disjoint the technique is used to show that for each a stable a a stable or stiffly stable linear multistep formula for ordinary differential equations there is a correspond ing linear multistep formula for delay differential equations with analogous stability properties the analogy does not extend however to a stable one step formulas

on different facets of regularization theory this review provides a comprehensive understanding of regularization theory from different perspectives emphasizing smoothness and simplicity principles using the tools of operator theory and fourier analysis it is shown that the solution of the classical tikhonov regularization problem can be derived from the regularized functional defined by a linear differential integral operator in the spatial fourier domain state of the art research relevant to the regularization theory is reviewed covering occam s razor minimum length description bayesian theory pruning algorithms informational entropy theory statistical learning theory and equivalent regularization the universal principle of regularization in terms of kolmogorov complexity is discussed finally some prospective studies on regularization theory and beyond are suggested

a comparison of gcv and gml for choosing the smoothing parameter in the generalized spline smoothing problem the partially improper prior behind the smoothing spline model is used to obtain a generalization of the maximum likelihood gml estimate for the smoothing parameter then this estimate is compared with the generalized cross validation gcv estimate both analytically and by monte carlo methods the comparison is based on a predictive mean square error criteria it is shown that if the true unknown function being estimated is smooth in a sense to be defined then the gml estimate undersmooths relative to the gcv estimate and the predictive mean square error using the gml estimate goes to zero at a slower rate than the mean square error using the gcv estimate if the true function is rough then the gcv and gml estimates have asymptotically similar behavior a monte carlo experiment was designed to see if the asymptotic results in the smooth case were evident in small sample sizes mixed results were obtained for n 32 gcv was somewhat better than gml for n 64 and gcv was decidedly superior for n 128 in the n 32 case gcv was better for smaller sigma 2 and the comparison close for larger sigma 2 the theoretical results are shown to extend to the generalized spline smoothing model which includes the estimate of functions given noisy values of various integrals of them

grafting fast incremental feature selection by gradient descent in function space we present a novel and flexible approach to the problem of feature selection called grafting rather than considering feature selection as separate from learning grafting treats the selection of suitable features as an integral part of learning a predictor in a regularized learning framework to make this regularized learning process sufficiently fast for large scale problems grafting operates in an incremental iterative fashion gradually building up a feature set while training a predictor model using gradient descent at each iteration a fast gradient based heuristic is used to quickly assess which feature is most likely to improve the existing model that feature is then added to the model and the model is incrementally optimized using gradient descent the algorithm scales linearly with the number of data points and at most quadratically with the number of features grafting can be used with a variety of predictor model classes both linear and non linear and can be used for both classification and regression experiments are reported here on a variant of grafting for classification using both linear and non linear models and using a logistic regression inspired loss function results on a variety of synthetic and real world data sets are presented finally the relationship between grafting stagewise additive modelling and boosting is explored

polyphase representation of multirate nonlinear filters and its applications this paper proposes a polyphase representation for nonlinear filters especially for volterra filters to derive the new realizations the well known linear polyphase theory is extended to the nonlinear case both the upsampling and downsampling cases are considered as in the linear case finite impulse response filters neither the input signal nor the volterra kernels must fulfil constraints in order to be realized in polyphase form the computational complexity can be reduced significantly because of two reasons on the one hand all operations are performed at the low sampling rate and on the other hand a new null identity allows to remove many coefficients in the polyphase representation furthermore some applications involving a nonlinear filter an upsampler and or a downsampler are discussed to demonstrate the utility of the new approach to multirate nonlinear signal processing

spline collocation methods for solving delay differential equations the convergence and stability analysis of seventh c 3 spline collocation methods if applied to delay differential equations ddes will be considered letting the interior collocation points x i 1 c j x i 1 c jh j 1 3 be dependent on three parameters c 1 c 2 c email protected 0 1 it will be shown that these methods for ddes possess convergence rate of order seven if 0 7279115

hinging hyperplanes for regression classification and function approximation a hinge function y h x consists of two hyperplanes continuously joined together at a hinge in regression prediction classification pattern recognition and noiseless function approximation use of sums of hinge functions gives a powerful and efficient alternative to neural networks with computation times several orders of magnitude less than is obtained by fitting neural networks with a comparable number of parameters a simple and effective method for finding good hinges is presented

running fir and iir filtering using multirate filter banks the computational complexity of running fir finite impulse response and iir infinite impulse response filtering using multirate filter banks is considered no restrictions are put on signal filter or block lengths it is shown how to map long running convolutions into smaller ones by using filter banks based on aperiodic convolution algorithms and short time fourier transforms with the proposed approach good tradeoffs among computational complexity system architecture and input output delay are achieved

collocation methods for the computation of periodic solutions of delay differential equations in this paper we investigate collocation methods for the computation of periodic solutions of autonomous delay differential equations ddes periodic solutions are found by solving a periodic two point boundary value problem which is an infinite dimensional problem for ddes in contrast to the case of ordinary differential equations we investigate three collocation methods based on piecewise polynomials we discuss computational issues and show numerical orders of convergence using an extensive number of tests we compare our numerical results with known theoretical convergence results for initial value problems for ddes in particular we show how superconvergence at the mesh points can be lost or recovered depending on the dde model under consideration and on the choice of collocation discretization we end with a brief discussion of adaptive mesh selection

nonlinear knowledge based classification in this brief prior knowledge over general nonlinear sets is incorporated into nonlinear kernel classification problems as linear constraints in a linear program these linear constraints are imposed at arbitrary points not necessarily where the prior knowledge is given the key tool in this incorporation is a theorem of the alternative for convex functions that converts nonlinear prior knowledge implications into linear inequalities without the need to kernelize these implications effectiveness of the proposed formulation is demonstrated on publicly available classification data sets including a cancer prognosis data set nonlinear kernel classifiers for these data sets exhibit marked improvements upon the introduction of nonlinear prior knowledge compared to nonlinear kernel classifiers that do not utilize such knowledge

on the relationship between generalization error hypothesis complexity and sample complexity for radial basis functions feedforward networks together with their training algorithms are a class of regression techniques that can be used to learn to perform some task from a set of examples the question of generalization of network performance from a finite training set to unseen data is clearly of crucial importance in this article we first show that the generalization error can be decomposed into two terms the approximation error due to the insufficient representational capacity of a finite sized network and the estimation error due to insufficient information about the target function because of the finite number of samples we then consider the problem of learning functions belonging to certain sobolev spaces with gaussian radial basis functions using the above mentioned decomposition we bound the generalization error in terms of the number of basis functions and number of examples while the bound that we derive is specific for radial basis functions a number of observations deriving from it apply to any approximation technique our result also sheds light on ways to choose an appropriate network architecture for a particular problem and the kinds of problems that can be effectively solved with finite resources i e with a finite number of parameters and finite amounts of data

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

an introduction to schoenberg s approximation for a given function b and a non zero real number h schoenberg s approximation defines from some data jh yj j zd the function j zd yj b h j for people not used to this kind of approximation this paper intends to do a summary of the main definitions properties and utilizations of schoenberg s approximation we show that the main tool to handle schoenberg s approximation is the fourier transform of b and even more its modified version the transfer function of b we give conditions for convergence of j zd f jh b h j when h tends to zero and we give various ways to define various b as combinations of translates of some function usually is either some radial function or obtained by a tensor product of some radial function depending on the properties we want for the associated schoenberg s approximation last we show how multi resolution analysis subdivision techniques and wavelets techniques are nicely connected to schoenberg s approximation

learning with mixed hard soft pointwise constraints a learning paradigm is proposed and investigated in which the classical framework of learning from examples is enhanced by the introduction of hard pointwise constraints i e constraints imposed on a finite set of examples that cannot be violated such constraints arise e g when requiring coherent decisions of classifiers acting on different views of the same pattern the classical examples of supervised learning which can be violated at the cost of some penalization quantified by the choice of a suitable loss function play the role of soft pointwise constraints constrained variational calculus is exploited to derive a representer theorem that provides a description of the functional structure of the optimal solution to the proposed learning paradigm it is shown that such an optimal solution can be represented in terms of a set of support constraints which generalize the concept of support vectors and open the doors to a novel learning paradigm called support constraint machines the general theory is applied to derive the representation of the optimal solution to the problem of learning from hard linear pointwise constraints combined with soft pointwise constraints induced by supervised examples in some cases closed form optimal solutions are obtained

convergence analysis of the solution of retarded and neutral delay differential equations by continuous numerical methods we have recently developed a generic approach to solving retarded and neutral delay differential equations ddes the approach is based on the use of an explicit continuous runge kutta formula and employs defect control the approach can be applied to problems with state dependent delays and vanishing delays in this paper we determine convergence properties for the numerical solution associated with methods that implement this approach we first analyze such properties for retarded ddes and then the analysis is extended to the case of neutral ddes ndes such an extension is particularly important since ndes have received little attention in the literature the main result we establish is that the global error of the numerical solution can be efficiently and reliably controlled by directly monitoring the magnitude of the associated defect our analysis can also be applied directly to other numerical methods which use defect error control based on runge kutta formulas for ddes

hilbert space embeddings and metrics on probability measures a hilbert space embedding for probability measures has recently been proposed with applications including dimensionality reduction homogeneity testing and independence testing this embedding represents any probability measure as a mean element in a reproducing kernel hilbert space rkhs a pseudometric on the space of probability measures can be defined as the distance between distribution embeddings we denote this as k indexed by the kernel function k that defines the inner product in the rkhs we present three theoretical properties of k first we consider the question of determining the conditions on the kernel k for which k is a metric such k are denoted characteristic kernels unlike pseudometrics a metric is zero only when two distributions coincide thus ensuring the rkhs embedding maps all distributions uniquely i e the embedding is injective while previously published conditions may apply only in restricted circumstances e g on compact domains and are difficult to check our conditions are straightforward and intuitive integrally strictly positive definite kernels are characteristic alternatively if a bounded continuous kernel is translation invariant on d then it is characteristic if and only if the support of its fourier transform is the entire d second we show that the distance between distributions under k results from an interplay between the properties of the kernel and the distributions by demonstrating that distributions are close in the embedding space when their differences occur at higher frequencies third to understand the nature of the topology induced by k we relate k to other popular metrics on probability measures and present conditions on the kernel k under which k metrizes the weak topology

reproducing kernels of generalized sobolev spaces via a green function approach with distributional operators in this paper we introduce a generalized sobolev space by defining a semi inner product formulated in terms of a vector distributional operator p consisting of finitely or countably many distributional operators p n which are defined on the dual space of the schwartz space the types of operators we consider include not only differential operators but also more general distributional operators such as pseudo differential operators we deduce that a certain appropriate full space green function g with respect to l p t p now becomes a conditionally positive function in order to support this claim we ensure that the distributional adjoint operator p of p is well defined in the distributional sense under sufficient conditions the native space reproducing kernel hilbert space associated with the green function g can be embedded into or even be equivalent to a generalized sobolev space as an application we take linear combinations of translates of the green function with possibly added polynomial terms and construct a multivariate minimum norm interpolant s f x to data values sampled from an unknown generalized sobolev function f at data sites located in some set x subset mathbb r d we provide several examples such as matern kernels or gaussian kernels that illustrate how many reproducing kernel hilbert spaces of well known reproducing kernels are equivalent to a generalized sobolev space these examples further illustrate how we can rescale the sobolev spaces by the vector distributional operator p introducing the notion of scale as part of the definition of a generalized sobolev space may help us to choose the best kernel function for kernel based approximation methods

regularized principal manifolds many settings of unsupervised learning can be viewed as quantization problems the minimization of the expected quantization error subject to some restrictions this allows the use of tools such as regularization from the theory of supervised risk minimization for unsupervised learning this setting turns out to be closely related to principal curves the generative topographic map and robust coding we explore this connection in two ways 1 we propose an algorithm for finding principal manifolds that can be regularized in a variety of ways and 2 we derive uniform convergence bounds and hence bounds on the learning rates of the algorithm in particular we give bounds on the covering numbers which allows us to obtain nearly optimal learning rates for certain types of regularization operators experimental results demonstrate the feasibility of the approach