c0 is isometrically isomorphic to a subspace of cantor lebesgue functions abstract we prove what the title of this note states i e c 0 that is the set of all sequences converging to 0 is isometrically isomorphic to a subspace of continuous almost everywhere differentiable and non lipschitz functions on 0 1 having almost everywhere null derivative this in particular improves a recent result jimenez rodriguez et al 2012 9 on the lineability of this set of functions

discriminant analysis by gaussian mixtures fisher rao linear discriminant analysis lda is a valuable tool for multigroup classification lda is equivalent to maximum likelihood classification assuming gaussian distributions for each class in this paper we fit gaussian mixtures to each class to facilitate effective classification in non normal settings especially when the classes are clustered low dimensional views are an important by product of lda our new techniques inherit this feature we can control the within class spread of the subclass centres relative to the between class spread our technique for fitting these models permits a natural blend with nonparametric versions of lda

origin of exponential growth in nonlinear reaction networks exponentially growing systems are prevalent in nature spanning all scales from biochemical reaction networks in single cells to food webs of ecosystems how exponential growth emerges in nonlinear systems is mathematically unclear here we describe a general theoretical framework that reveals underlying principles of long term growth scalability of flux functions and ergodicity of the rescaled systems our theory shows that nonlinear fluxes can generate not only balanced growth but also oscillatory or chaotic growth modalities explaining nonequilibrium dynamics observed in cell cycles and ecosystems our mathematical framework is broadly useful in predicting long term growth rates from natural and synthetic networks analyzing the effects of system noise and perturbations validating empirical and phenomenological laws on growth rate and studying autocatalysis and network evolution

bayesian regularization and pruning using a laplace prior standard techniques for improved generalization from neural networks include weight decay and pruning weight decay has a bayesian interpretation with the decay function corresponding to a prior over weights the method of transformation groups and maximum entropy suggests a laplace rather than a gaussian prior after training the weights then arrange themselves into two classes 1 those with a common sensitivity to the data error and 2 those failing to achieve this sensitivity and that therefore vanish since the critical value is determined adaptively during training pruning in the sense of setting weights to exact zeros becomes an automatic consequence of regularization alone the count of free parameters is also reduced automatically as weights are pruned a comparison is made with results of mackay using the evidence framework and a gaussian regularizer

learning with box kernels supervised examples and prior knowledge on regions of the input space have been profitably integrated in kernel machines to improve the performance of classifiers in different real world contexts the proposed solutions which rely on the unified supervision of points and sets have been mostly based on specific optimization schemes in which as usual the kernel function operates on points only in this paper arguments from variational calculus are used to support the choice of a special class of kernels referred to as box kernels which emerges directly from the choice of the kernel function associated with a regularization operator it is proven that there is no need to search for kernels to incorporate the structure deriving from the supervision of regions of the input space because the optimal kernel arises as a consequence of the chosen regularization operator although most of the given results hold for sets we focus attention on boxes whose labeling is associated with their propositional description based on different assumptions some representer theorems are given that dictate the structure of the solution in terms of box kernel expansion successful results are given for problems of medical diagnosis image and text categorization

nonseparable spaceability and strong algebrability of sets of continuous singular functions abstract let cbv denote the banach algebra of all continuous real valued functions of bounded variation defined in 0 1 we show that the set of strongly singular functions in cbv is nonseparably spaceable we also prove that certain families of singular functions constitute strongly c algebrable sets the argument is based on a criterion of strong c algebrability using compositions with exponential like functions

classifier learning with a new locality regularization method it is well known that the generalization capability is one of the most important criterions to develop and evaluate a classifier for a given pattern classification problem the localized generalization error model r s m recently proposed by ng et al localized generalization error and its application to rbfnn training in proceedings of the international conference on machine learning and cybernetics china 2005 image classification with the use of radial basis function neural networks and the minimization of the localized generalization error pattern recognition 40 1 2007 4 18 provides a more intuitive look at the generalization error although r s m gives a brand new method to promote the generalization performance it is in nature equivalent to another type of regularization in this paper we first prove the essential relationship between r s m and regularization and demonstrate that the stochastic sensitivity measure in r s m exactly corresponds to a regularizing term then we develop a new generalization error bound from the regularization viewpoint which is inspired by the proved relationship between r s m and regularization moreover we derive a new regularization method called as locality regularization lr from the bound different from the existing regularization methods which artificially and externally append the regularizing term in order to smooth the solution lr is naturally and internally deduced from the defined expected risk functional and calculated by employing locality information through combining with spectral graph theory lr introduces the local structure information of the samples into the regularizing term and further improves the generalization capability in contrast with r s m which is relatively sensitive to the different sampling of the samples lr uses the discrete k neighborhood rather than the common continuous q neighborhood in r s m to differentiate the relative position of different training samples automatically and avoid the complex computation of q for various classifiers furthermore lr uses the regularization parameter to control the trade off between the training accuracy and the classifier stability experimental results on artificial and real world problems show that lr yields better generalization capability than both r s m and some traditional regularization methods

lineability spaceability and additivity cardinals for darboux like functions we introduce the concept of maximal lineability cardinal number ml m of a subset m of a topological vector space and study its relation to the cardinal numbers known as additivity a m homogeneous lineability hl m and lineability l m of m in particular we will describe in terms of l the lineability and spaceability of the families of the following darboux like functions on r n n 1 extendable jones and almost continuous functions

spline smoothing the equivalent variable kernel method on considere une approche spline pour la regression non parametrique et l estimation de courbe on montre que dans un certain sens le lissage spline correspond approximativement au lissage par une methode du noyau avec une largeur de bande dependant de la densite locale des points du plan de regression

regularization theory and neural networks architectures we had previously shown that regularization principles lead to approximation schemes that are equivalent to networks with one layer of hidden units called regularization networks in particular s

pattern recognition as a quest for minimum entropy abstract we shall show that many known algorithms of clustering and pattern recognition can be characterized as efforts to minimize entropy when suitably defined

on the application of orthogonal transformation for the design and analysis of feedforward networks orthogonal transformation which can lead to compaction of information has been used in two ways to optimize on the size of feedforward networks 1 through the selection of optimum set of time domain inputs and the optimum set of links and nodes within a neural network nn and 2 through the orthogonalization of the data to be used in nn s in case of processes with periodicity the proposed methods are efficient and are also extremely robust numerically the singular value decomposition svd and qr with column pivoting factorization qrcp are the transformations used svd mainly serves as the null space detector qrcp coupled with svd is used for subset selection which is one of the main operations on which the design of the optimal network is based svd has also been used to devise a new approach for the assessment of the convergence of the nn s which is an alternative to the conventional output error analysis

note on free lunches and cross validation the no free lunch theorems wolpert macready 1995 have sparked heated debate in the computational learning community a recent communication zhu rohwer 1996 attempts to demonstrate the inefficiency of cross validation on a simple problem we elaborate on this result by considering a broader class of cross validation when used more strictly cross validation can yield the expected results on simple examples

multik mhks a novel multiple kernel learning algorithm in this paper we develop a new effective multiple kernel learning algorithm first we map the input data into m different feature spaces by m empirical kernels where each generated feature space is taken as one view of the input space then through borrowing the motivating argument from canonical correlation analysis cca that can maximally correlate the m views in the transformed coordinates we introduce a special term called inter function similarity loss rifsi into the existing regularization framework so as to guarantee the agreement of multiview outputs in implementation we select the modification of ho kashyap algorithm with squared approximation of the misclassification errors mhks as the incorporated paradigm and the experimental results on benchmark data sets demonstrate the feasibility and effectiveness of the proposed algorithm named multik mhks

lineability and integrability in the sense of riemann lebesgue denjoy and khintchine abstract in this paper we continue the ongoing research on lineability related questions on this occasion we shall consider among others the classes of integrable functions in the sense of riemann lebesgue denjoy and khintchine improving some already known results and expanding the study of lineability to other famous integrable classes never considered before

the lack of a priori distinctions between learning algorithms this is the first of two papers that use off training set ots error to investigate the assumption free relationship between learning algorithms this first paper discusses the senses in which there are no a priori distinctions between learning algorithms the second paper discusses the senses in which there are such distinctions in this first paper it is shown loosely speaking that for any two algorithms a and b there are as many targets or priors over targets for which a has lower expected ots error than b as vice versa for loss functions like zero one loss in particular this is true if a is cross validation and b is anti cross validation choose the learning algorithm with largest cross validation error this paper ends with a discussion of the implications of these results for computational learning theory it is shown that one cannot say if empirical misclassification rate is low the vapnik chervonenkis dimension of your generalizer is small and the training set is large then with high probability your ots error is small other implications for membership queries algorithms and punting algorithms are also discussed

the effects of adding noise during backpropagation training on a generalization performance we study the effects of adding noise to the inputs outputs weight connections and weight changes of multilayer feedforward neural networks during backpropagation training we rigorously derive and analyze the objective functions that are minimized by the noise affected training processes we show that input noise and weight noise encourage the neural network output to be a smooth function of the input or its weights respectively in the weak noise limit noise added to the output of the neural networks only changes the objective function by a constant hence it cannot improve generalization input noise introduces penalty terms in the objective function that are related to but distinct from those found in the regularization approaches simulations have been performed on a regression and a classification problem to further substantiate our analysis input noise is found to be effective in improving the generalization performance for both problems however weight noise is found to be effective in improving the generalization performance only for the classification problem other forms of noise have practically no effect on generalization

strong c algebrability of strong sierpinski zygmund smooth nowhere analytic and other sets of functions abstract we present a useful technique of proving strong c algebrability as an outcome we obtain the strong c algebrability of the following sets of functions strong sierpinski zygmund nowhere holder bruckner garg nowhere monotone differentiable a certain baire class smooth and nowhere analytic functions

large deviations with diminishing rates the theory of large deviations for jump markov processes has been generally proved only when jump rates are bounded below away from zero dupuis and ellis 1995 the large deviations principle for a general class of queueing systems i trans amer math soc 347 2689 2751 ignatiouk robert 2002 sample path large deviations and convergence parameters ann appl probab 11 1292 1329 shwartz and weiss 1995 large deviations for performance analysis chapman hall yet various applications of interest do not satisfy this condition we describe several classes of models where jump rates diminish to zero in a lipschitz continuous way under appropriate conditions we prove that the sample path large deviations principle continues to hold under our conditions the rate function remains an integral over a local rate function which retains its standard representation

similarities of error regularization sigmoid gain scaling target smoothing and training with jitter the generalization performance of feedforward layered perceptrons can in many cases be improved either by smoothing the target via convolution regularizing the training error with a smoothing constraint decreasing the gain i e slope of the sigmoid nonlinearities or adding noise i e jitter to the input training data in certain important cases the results of these procedures yield highly similar results although at different costs training with jitter for example requires significantly more computation than sigmoid scaling

a sparse representation for function approximation we derive a new general representation for a function as a linear combination of local correlation kernels at optimal sparse locations and scales and characterize its relation to principal component analysis regularization sparsity principles and support vector machines

algebraic structures in the sets of surjective functions abstract in the paper we construct several algebraic structures vector spaces algebras and free algebras inside sets of different types of surjective functions among many results we prove that the set of everywhere but not strongly everywhere surjective complex functions is strongly c algebrable and that its 2 c algebrability is consistent with zfc under ch the set of everywhere surjective complex functions which are sierpinski zygmund in the sense of continuous but not borel functions is strongly c algebrable the set of jones complex functions is strongly 2 c algebrable

learning and generalization in radial basis function networks the two layer radial basis function network with fixed centers of the basis functions is analyzed within a stochastic training paradigm various definitions of generalization error are considered and two such definitions are employed in deriving generic learning curves and generalization properties both with and without a weight decay term the generalization error is shown analytically to be related to the evidence and via the evidence to the prediction error and free energy the generalization behavior is explored the generic learning curve is found to be inversely proportional to the number of training pairs presented optimization of training is considered by minimizing the generalization error with respect to the free parameters of the training algorithms finally the effect of the joint activations between hidden layer units is examined and shown to speed training

entropy based algorithms for best basis selection adapted waveform analysis uses a library of orthonormal bases and an efficiency functional to match a basis to a given signal or family of signals it permits efficient compression of a variety of signals such as sound and images the predefined libraries of modulated waveforms include orthogonal wavelet packets and localized trigonometric functions and have reasonably well controlled time frequency localization properties the idea is to build out of the library functions an orthonormal basis relative to which the given signal or collection of signals has the lowest information cost the method relies heavily on the remarkable orthogonality properties of the new libraries all expansions in a given library conserve energy and are thus comparable several cost functionals are useful one of the most attractive is shannon entropy which has a geometric interpretation in this context

an equivalence between sparse approximation and support vector machines in the first part of this paper we show a similarity between the principle of structural risk minimization principle srm vapnik 1982 and the idea of sparse approximation as defined in chen donoho and saunders 1995 and olshausen and field 1996 then we focus on two specific approximate implementations of srm and sparse approximation which have been used to solve the problem of function approximation for srm we consider the support vector machine technique proposed by v vapnik and his team at at t bell labs and for sparse approximation we consider a modification of the basis pursuit de noising algorithm proposed by chen donoho and saunders 1995 we show that under certain conditions these two techniques are equivalent they give the same solution and they require the solution of the same quadratic programming problem

computation with infinite neural networks for neural networks with a wide class of weight priors it can be shown that in the limit of an infinite number of hidden units the prior over functions tends to a gaussian process in this article analytic forms are derived for the covariance function of the gaussian processes corresponding to networks with sigmoidal and gaussian hidden units this allows predictions to be made efficiently using networks with an infinite number of hidden units and shows somewhat paradoxically that it may be easier to carry out bayesian prediction with infinite networks rather than finite ones

no free lunch for noise prediction no free lunch theorems have shown that learning algorithms cannot be universally good we show that no free funch exists for noise prediction as well we show that when the noise is additive and the prior over target functions is uniform a prior on the noise distribution cannot be updated in the bayesian sense from any finite data set we emphasize the importance of a prior over the target function in order to justify superior performance for learning systems

on tikhonov regularization bias and variance in nonlinear system identification abstract regularization is a general method for solving ill posed and ill conditioned problems traditionally ill conditioning in system identification problems is approached using regularization methods such as ridge regression and principal component regression in this work it is argued that the tikhonov regularization method is a powerful alternative for regularization of nonlinear system identification problems by introducing smoothness of the model as a prior its properties is discussed in terms of an analysis of bias and variance and illustrated by a semirealistic simulation example

generalization performance of regularization networks and support vector machines via entropy numbers of compact operators we derive new bounds for the generalization error of kernel machines such as support vector machines and related regularization networks by obtaining new bounds on their covering numbers the proofs make use of a viewpoint that is apparently novel in the field of statistical learning theory the hypothesis class is described in terms of a linear operator mapping from a possibly infinite dimensional unit ball in feature space into a finite dimensional space the covering numbers of the class are then determined via the entropy numbers of the operator these numbers which characterize the degree of compactness of the operator can be bounded in terms of the eigenvalues of an integral operator induced by the kernel function used by the machine as a consequence we are able to theoretically explain the effect of the choice of kernel function on the generalization performance of support vector machines

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

lineability and additivity in r r we give a condition for a family of functions to be lineable by means of its additivity the novelty presented here is that we solve the lineability problem using a technique that is not constructive as are most approaches to this problem we relate the notions of additivity and lineability and use this relation to give a general method to find the lineability of large families of functions we also study more examples of pathologically behaving functions in particular the class of jones functions which is a highly pathological subclass of perfectly everywhere surjective functions we work on the additivity lineability and main properties of this class

large free linear algebras of real and complex functions let x be a set of cardinality such that we prove that the linear algebra rx or cx contains a free linear algebra with 2 generators using this we prove several algebrability results for spaces cc and rr in particular we show that the set of all perfectly everywhere surjective functions f c c is strongly 2c algebrable we also show that the set of all functions f r r whose sets of continuity points equals some fixed g set g is strongly 2c algebrable if and only if r g is c dense in itself

algebraic genericity within the class of sup measurable functions abstract it is well known that there exist real valued non measurable functions on the plane whose restrictions to the graph of all measurable functions on the real line are measurable in this paper it is shown that there is in fact a continuum dimensional vector space all of whose members except for zero are functions on the plane satisfying the mentioned property versions of this fact are proved in the baire category setting and in the case where the graphs are generated by softer functions

some extensions of radial basis functions and their applications in artificial intelligence abstract radial basis functions have recently found interesting applications in artificial intelligence and in particular in the problem of learning to perform a particular task from a set of examples however in many practical cases the radial basis functions method cannot be applied in a straight forward manner because it does not take into account some features that are typical of the problem of learning from examples in this paper we show some extensions of the standard theory introduced in order to deal with a large number of examples and with problems in which different variables play very different roles we present some examples and also point out some open problems

additivity and lineability in vector spaces abstract gamez merino munoz fernandez and seoane sepulveda proved that if additivity a f c then f is a f lineable where f r r they asked if a f c can be weakened we answer this question in negative moreover we introduce and study the notions of homogeneous lineability number and lineability number of subsets of linear spaces

training with noise is equivalent to tikhonov regularization it is well known that the addition of noise to the input data of a neural network during training can in some circumstances lead to significant improvements in generalization performance previous work has shown that such training with noise is equivalent to a form of regularization in which an extra term is added to the error function however the regularization term which involves second derivatives of the error function is not bounded below and so can lead to difficulties if used directly in a learning algorithm based on error minimization in this paper we show that for the purposes of network training the regularization term can be reduced to a positive semi definite form that involves only first derivatives of the network mapping for a sum of squares error function the regularization term belongs to the class of generalized tikhonov regularizers direct minimization of the regularized error function provides a practical alternative to training with noise

regularized neural networks some convergence rate results in a recent paper poggio and girosi 1990 proposed a class of neural networks obtained from the theory of regularization regularized networks are capable of approximating arbitrarily well any continuous function on a compactum in this paper we consider in detail the learning problem for the one dimensional case we show that in the case of output data observed with noise regularized networks are capable of learning and approximating on compacta elements of certain classes of sobolev spaces known as reproducing kernel hilbert spaces rkhs at a nonparametric rate that optimally exploits the smoothness properties of the unknown mapping in particular we show that the total squared error given by the sum of the squared bias and the variance will approach zero at a rate of n 2m 2m 1 where m denotes the order of differentiability of the true unknown function on the other hand if the unknown mapping is a continuous function but does not belong to an rkhs then there still exists a unique regularized solution but this is no longer guaranteed to converge in mean square to a well defined limit further even if such a solution converges the total squared error is bounded away from zero for all n sufficiently large

reduced hyperbf networks regularization by explicit complexity reduction and scaled rprop based training hyper basis function hyperbf networks are generalized radial basis function neural networks where the activation function is a radial function of a weighted distance such generalization provides hyperbf networks with high capacity to learn complex functions which in turn make them susceptible to overfitting and poor generalization moreover training a hyperbf network demands the weights centers and local scaling factors to be optimized simultaneously in the case of a relatively large dataset with a large network structure such optimization becomes computationally challenging in this paper a new regularization method that performs soft local dimension reduction in addition to weight decay is proposed the regularized hyperbf network is shown to provide classification accuracy competitive to a support vector machine while requiring a significantly smaller network structure furthermore a practical training to construct hyperbf networks is presented hierarchal clustering is used to initialize neurons followed by a gradient optimization using a scaled version of the rprop algorithm with a localized partial backtracking step experimental results on seven datasets show that the proposed training provides faster and smoother convergence than the regular rprop algorithm

linear subsets of nonlinear sets in topological vector spaces for the last decade there has been a generalized trend in mathematics on the search for large algebraic structures linear spaces closed subspaces or infinitely generated algebras composed of mathematical objects enjoying certain special properties this trend has caught the eye of many researchers and has also had a remarkable influence in real and complex analysis operator theory summability theory polynomials in banach spaces hypercyclicity and chaos and general functional analysis this expository paper is devoted to providing an account on the advances and on the state of the art of this trend nowadays known as lineability and spaceability

analysis of discrete ill posed problems by means of the l curve when discrete ill posed problems are analyzed and solved by various numerical regularization techniques a very convenient way to display information about the regularized solution is to plot the norm or seminorm of the solution versus the norm of the residual vector in particular the graph associated with tikhonov regularization plays a central role the main purpose of this paper is to advocate the use of this graph in the numerical treatment of discrete ill posed problems the graph is characterized quantitatively and several important relations between regularized solutions and the graph are derived it is also demonstrated that several methods for choosing the regularization parameter are related to locating a characteristic l shaped corner of the graph

regularization networks fast weight calculation via kalman filtering regularization networks are nonparametric estimators obtained from the application of tychonov regularization or bayes estimation to the hypersurface reconstruction problem their main drawback back is that the computation of the weights scales as o n sup 3 where n is the number of data in this paper we show that for a class of monodimensional problems the complexity can be reduced to o n by a suitable algorithm based on spectral factorization and kalman filtering moreover the procedure applies also to smoothing splines

on bias plus variance this article presents several additive corrections to the conventional quadratic loss bias plus variance formula one of these corrections is appropriate when both the target is not fixed as in bayesian analysis and training sets are averaged over as in the conventional bias plus variance formula another additive correction casts conventional fixed trainingset bayesian analysis directly in terms of bias plus variance another correction is appropriate for measuring full generalization error over a test set rather than as with conventional bias plus variance error at a single point yet another correction can help explain the recent counterintuitive bias variance decomposition of friedman for zero one loss after presenting these corrections this article discusses some other loss function specific aspects of supervised learning in particular there is a discussion of the fact that if the loss function is a metric e g zero one loss then there is bound on the change in generalization error acco

kernel independent component analysis we present a class of algorithms for independent component analysis ica which use contrast functions based on canonical correlations in a reproducing kernel hilbert space on the one hand we show that our contrast functions are related to mutual information and have desirable mathematical properties as measures of statistical dependence on the other hand building on recent developments in kernel methods we show that these criteria and their derivatives can be computed efficiently minimizing these criteria leads to flexible and robust algorithms for ica we illustrate with simulations involving a wide variety of source distributions showing that our algorithms outperform many of the presently known algorithms

a framework for multiscale and hybrid rkhs based approximators a generalized framework for deriving multiscale and hybrid functionally expanded approximators that are linear in the adjustable weights is presented the basic idea here is to define one or more appropriate function spaces and then to impose a geometric structure on these to obtain reproducing kernel hilbert spaces rkhss the weight identification problem is formulated as a minimum norm optimization problem that produces an approximation network structure that comprises a linear weighted sum of displaced reproducing kernels fed by the input signals examples of the application of this framework are discussed results of numerical experiments are presented

on different facets of regularization theory this review provides a comprehensive understanding of regularization theory from different perspectives emphasizing smoothness and simplicity principles using the tools of operator theory and fourier analysis it is shown that the solution of the classical tikhonov regularization problem can be derived from the regularized functional defined by a linear differential integral operator in the spatial fourier domain state of the art research relevant to the regularization theory is reviewed covering occam s razor minimum length description bayesian theory pruning algorithms informational entropy theory statistical learning theory and equivalent regularization the universal principle of regularization in terms of kolmogorov complexity is discussed finally some prospective studies on regularization theory and beyond are suggested

adaptive regularization parameter selection method for enhancing generalization capability of neural networks a novel adaptive regularization parameter selection arps method is proposed in this paper to enhance the performance of the regularization method the proposed arps method enables a gradient descent type training to tunnel through some of the undesired sub optimal solutions on the composite error surface by means of changing the value of the regularization parameter undesired sub optimal solutions are introduced inherently from regularized objective functions hence the proposed arps method is capable of enhancing the regularization method without getting stuck at these sub optimal solutions

automatic generation of rbf networks using wavelets abstract learning can be viewed as mapping from an input space to an output space examples of these mappings are used to construct a continuous function that approximates the given data and generalizes for intermediate instances radial basis function rbf networks are used to formulate this approximating function a novel method is introduced that automatically constructs a generalized radial basis function grbf network for a given mapping and error bound this network is shown to be the smallest network within the error bound for the given mapping the integral wavelet transform is used to determine the parameters of the network simple one dimensional examples are used to demonstrate how the network constructed using the transform is superior to that constructed using standard ad hoc optimization techniques the paper concludes with the automatic generation of grbf networks for a multi dimensional problem namely real time 3d object recognition and pose estimation the results of this application are favorable

convergence analyses on sparse feedforward neural networks via group lasso regularization in this paper a new variant of feedforward neural networks has been proposed for a class of nonsmooth optimization problems the penalty term of the presented neural networks stems from the group lasso method which selects hidden variables in a grouped manner to deal with the non differentiability of the original penalty term l 1 l 2 norm and avoid oscillations smoothing techniques have been used to approximate the objective function it is assumed that the training samples are supplied to the networks in a specific incremental way during training that is in each cycle samples are supplied in a fixed order then under suitable assumptions on learning rate penalization coefficients and smoothing parameters the weak and strong convergence of the training process for the smoothing neural networks have been proved the convergence analysis shows that the gradient of the smoothing error function approaches zero and the weight sequence converges to a fixed point respectively we demonstrate how the smoothing approximation parameter can be updated in the training procedure so as to guarantee the convergence of the procedure to a clarke stationary point of the original optimization problem in addition we have proved that the original nonsmoothing algorithm with l 1 l 2 norm penalty converges consistently to the same optimum solution with the corresponding smoothed algorithm numerical simulations demonstrate the convergence and effectiveness of the proposed training algorithm

least third order cumulant method with adaptive regularization parameter selection for neural networks abstract this paper introduces an interesting property of the least third order cumulant objective function the property is that the solution is optimal when the gradients of mean squares error and third order cumulant error are zero vectors the optimal solutions are independent of the value of regularization parameter also an adaptive regularization parameter selection method is derived to control the convergences of mean squares error and the cumulant error terms the proposed selection method is able to tunnel through the sub optimal solutions of which the locations are controllable via changing the value of the regularization parameter consequently the least third order cumulant method with the adaptive regularization parameter selection method is theoretically capable of estimating an optimal solution when it is applied to regression problems

bias variance decompositions for likelihood based estimators the bias variance decomposition of mean squared error is well understood and relatively straightforward in this note a similar simple decomposition is derived valid for any kind of error measure that when using the appropriate probability model can be derived from a kullback leibler divergence or log likelihood

connected polynomials and continuity abstract as it is well known f r r is continuous if and only if f maps continua compact connected sets to continua the same holds for mappings between any two real or complex normed spaces however when we restrict ourselves to polynomials p e k where e is a k normed space then it was proved in 2012 that p is continuous if and only if it transforms compact sets into compact sets here we show that if k c p is continuous if and only if it transforms connected sets into connected sets although we provide some partial results for k r the general case in the real setting remains still open

on the geometry of chemical reaction networks lyapunov function and large deviations in an earlier paper we proved the validity of large deviations theory for the particle approximation of quite general chemical reaction networks in this paper we extend its scope and present a more geometric insight into the mechanism of that proof exploiting the notion of spherical image of the reaction polytope this allows to view the asymptotic behavior of the vector field describing the mass action dynamics of chemical reactions as the result of an interaction between the faces of this polytope in different dimensions we also illustrate some local aspects of the problem in a discussion of wentzell freidlin theory together with some examples

radial basis function network learning using localized generalization error bound training a classifier with good generalization capability is a major issue for pattern classification problems a novel training objective function for radial basis function rbf network using a localized generalization error model l gem is proposed in this paper the localized generalization error model provides a generalization error bound for unseen samples located within a neighborhood that contains all training samples the assumption of the same width for all dimensions of a hidden neuron in l gem is relaxed in this work the parameters of rbf network are selected via minimization of the proposed objective function to minimize its localized generalization error bound the characteristics of the proposed objective function are compared with those for regularization methods for weight selection rbf networks trained by minimizing the proposed objective function consistently outperform rbf networks trained by minimizing the training error tikhonov regularization weight decay or locality regularization the proposed objective function is also applied to select center width and weight in rbf network simultaneously rbf networks trained by minimizing the proposed objective function yield better testing accuracies when compared to those that minimizes training error only

intensity based image registration by minimizing residual complexity accurate definition of the similarity measure is a key component in image registration most commonly used intensity based similarity measures rely on the assumptions of independence and stationarity of the intensities from pixel to pixel such measures cannot capture the complex interactions among the pixel intensities and often result in less satisfactory registration performances especially in the presence of spatially varying intensity distortions we propose a novel similarity measure that accounts for intensity nonstationarities and complex spatially varying intensity distortions in mono modal settings we derive the similarity measure by analytically solving for the intensity correction field and its adaptive regularization the final measure can be interpreted as one that favors a registration with minimum compression complexity of the residual image between the two registered images one of the key advantages of the new similarity measure is its simplicity in terms of both computational complexity and implementation this measure produces accurate registration results on both artificial and real world problems that we have tested and outperforms other state of the art similarity measures in these cases

on lineability and additivity of real functions with finite preimages abstract we study lineability of real functions with finite preimages in particular we prove that the class of n to one functions contains a vector subspace of dimension n but not of dimension n 1 additionally we give examples of star like families of functions closed under multiplication by a non zero scalar for which lineability is less than additivity

point set registration coherent point drift point set registration is a key component in many computer vision tasks the goal of point set registration is to assign correspondences between two sets of points and to recover the transformation that maps one point set to the other multiple factors including an unknown nonrigid spatial transformation large dimensionality of point set noise and outliers make the point set registration a challenging problem we introduce a probabilistic method called the coherent point drift cpd algorithm for both rigid and nonrigid point set registration we consider the alignment of two point sets as a probability density estimation problem we fit the gaussian mixture model gmm centroids representing the first point set to the data the second point set by maximizing the likelihood we force the gmm centroids to move coherently as a group to preserve the topological structure of the point sets in the rigid case we impose the coherence constraint by reparameterization of gmm centroid locations with rigid parameters and derive a closed form solution of the maximization step of the em algorithm in arbitrary dimensions in the nonrigid case we impose the coherence constraint by regularizing the displacement field and using the variational calculus to derive the optimal transformation we also introduce a fast algorithm that reduces the method computation complexity to linear we test the cpd algorithm for both rigid and nonrigid transformations in the presence of noise outliers and missing points where cpd shows accurate results and outperforms current state of the art methods

discriminant component pruning regularization and interpretation of multilayered backpropagation networks neural networks are often employed as tools in classification tasks the use of large networks increases the likelihood of the task s being learned although it may also lead to increased complexity pruning is an effective way of reducing the complexity of large networks we present discriminant components pruning dcp a method of pruning matrices of summed contributions between layers of a neural network attempting to interpret the underlying functions learned by the network can be aided by pruning the network generalization performance should be maintained at its optimal level following pruning we demonstrate dcp s effectiveness at maintaining generalization performance applicability to a wider range of problems and the usefulness of such pruning for network interpretation possible enhancements are discussed for the identification of the optimal reduced rank and inclusion of nonlinear neural activation functions in the pruning algorithm

minimum description length induction bayesianism and kolmogorov complexity the relationship between the bayesian approach and the minimum description length approach is established we sharpen and clarify the general modeling principles minimum description length mdl and minimum message length mml abstracted as the ideal mdl principle and defined from bayes s rule by means of kolmogorov complexity the basic condition under which the ideal principle should be applied is encapsulated as the fundamental inequality which in broad terms states that the principle is valid when the data are random relative to every contemplated hypothesis and also these hypotheses are random relative to the universal prior the ideal principle states that the prior probability associated with the hypothesis should be given by the algorithmic universal probability and the sum of the log universal probability of the model plus the log of the probability of the data given the model should be minimized if we restrict the model class to finite sets then application of the ideal principle turns into kolmogorov s minimal sufficient statistic in general we show that data compression is almost always the best strategy both in model selection and prediction

mlpnn training via a multiobjective optimization of training error and stochastic sensitivity the training of a multilayer perceptron neural network mlpnn concerns the selection of its architecture and the connection weights via the minimization of both the training error and a penalty term different penalty terms have been proposed to control the smoothness of the mlpnn for better generalization capability however controlling its smoothness using for instance the norm of weights or the vapnik chervonenkis dimension cannot distinguish individual mlpnns with the same number of free parameters or the same norm in this paper to enhance generalization capabilities we propose a stochastic sensitivity measure st sm to realize a new penalty term for mlpnn training the st sm determines the expectation of the squared output differences between the training samples and the unseen samples located within their q neighborhoods for a given mlpnn it provides a direct measurement of the mlpnns output fluctuations i e smoothness we adopt a two phase pareto based multiobjective training algorithm for minimizing both the training error and the st sm as biobjective functions experiments on 20 uci data sets show that the mlpnns trained by the proposed algorithm yield better accuracies on testing data than several recent and classical mlpnn training methods

an undecidable case of lineability in rr abstract recently it has been proved that assuming that there is an almost disjoint family of cardinality 2 c in c which is assured for instance by either martin s axiom or the continuum hypothesis or even 2 c c one has that the set of sierpinski zygmund functions is 2 c strongly algebrable and thus 2 c lineable here we prove that these two statements are actually equivalent and moreover that they both are undecidable this would be the first time in which one encounters an undecidable proposition in the recently coined theory of lineability and spaceability

simplifying neural networks by soft weight sharing one way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of nonzero weights we propose a more complicated penalty term in which the distribution of weight values is modeled as a mixture of multiple gaussians a set of weights is simple if the weights have high probability density under the mixture model this can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values since we do not know the appropriate means or variances of the clusters in advance we allow the parameters of the mixture model to adapt at the same time as the network learns simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms

on the relationship between generalization error hypothesis complexity and sample complexity for radial basis functions feedforward networks together with their training algorithms are a class of regression techniques that can be used to learn to perform some task from a set of examples the question of generalization of network performance from a finite training set to unseen data is clearly of crucial importance in this article we first show that the generalization error can be decomposed into two terms the approximation error due to the insufficient representational capacity of a finite sized network and the estimation error due to insufficient information about the target function because of the finite number of samples we then consider the problem of learning functions belonging to certain sobolev spaces with gaussian radial basis functions using the above mentioned decomposition we bound the generalization error in terms of the number of basis functions and number of examples while the bound that we derive is specific for radial basis functions a number of observations deriving from it apply to any approximation technique our result also sheds light on ways to choose an appropriate network architecture for a particular problem and the kinds of problems that can be effectively solved with finite resources i e with a finite number of parameters and finite amounts of data

no free lunch for early stopping we show that with a uniform prior on models having the same training error early stopping at some fixed training error above the training error minimum results in an increase in the expected generalization error

large structures in certain subsets of orlicz spaces abstract assume that and n n 1 are young functions in this paper we prove that under sharp conditions on a measure space and the young functions and n s the subset of functions in the corresponding orlicz space l which are not in l n for all n n admits various large linear structures

on a class of support vector kernels based on frames in function hilbert spaces there has been an increasing interest in kernel based techniques such as support vector techniques regularization networks and gaussian processes there are inner relationships among those techniques with the kernel function playing a central role this article discusses a new class of kernel functions derived from the so called frames in a function hilbert space

sparse on line gaussian processes we develop an approach for sparse representations of gaussian process gp models which are bayesian types of kernel machines in order to overcome their limitations for large data sets the method is based on a combination of a bayesian on line algorithm together with a sequential construction of a relevant subsample of the data that fully specifies the prediction of the gp model by using an appealing parameterization and projection techniques in a reproducing kernel hilbert space recursions for the effective parameters and a sparse gaussian approximation of the posterior process are obtained this allows for both a propagation of predictions and bayesian error measures the significance and robustness of our approach are demonstrated on a variety of experiments

centennial history of hilbert s 16th problem the second part of hilbert s 16th problem deals with polynomial differential equations in the plane it remains unsolved even for quadratic polynomials there were several attempts to solve it that failed yet the problem inspired significant progress in the geometric theory of planar differential equations as well as bifurcation theory normal forms foliations and some topics in algebraic geometry the dramatic history of the problem as well as related developments are presented below 1 the problem and its counterparts what may be said about the number and location of limit cycles of a planar polynomial vector field of degree n the limit cycle is an isolated closed orbit of a vector field this second part of hilbert s 16th problem appears to be one of the most persistent in the famous hilbert list h second only to the riemann function conjecture traditionally hilbert s question is split into three each one requiring a stronger answer problem 1 is it true that a planar polynomial vector field has but a finite number of limit cycles problem 2 is it true that the number of limit cycles of a planar polynomial vector field is bounded by a constant depending on the degree of the polynomials only the bound on the number of limit cycles in problem 2 is denoted by h n and known as the hilbert number linear vector fields have no limit cycles hence h 1 0 it is still unknown whether or not h 2 exists problem 3 give an upper bound for h n a solution to any of these problems implies a solution for the previous ones only the first problem is solved now the positive answer was established in e92 i91 there are analytic counterparts of problems 1 and 2 received by the editors december 2001 2000 mathematics subject classification primary 34cxx 34mxx 37f75

lp 0 1 q plq 0 1 is spaceable for every p 0 abstract in this short note we prove the result stated in the title that is for every p 0 there exists an infinite dimensional closed linear subspace of l p 0 1 every nonzero element of which does not belong to q p l q 0 1 this answers in the positive a question raised in 2010 by r m aron on the spaceability of the above sets for both the banach and quasi banach cases we also complete some recent results from botelho et al 2011 3 for subsets of sequence spaces

holomorphic functions on non runge domains and related problems let u be a domain in cn that is not a runge domain we study the topological and algebraic properties of the family of holomorphic functions on u which cannot be approximated by polynomials

infinite dimensional holomorphic non extendability and algebraic genericity abstract in this note the linear structure of the family h e g of holomorphic functions in a domain g of a complex banach space that are not holomorphically continuable beyond the boundary of g is analyzed more particularly we prove that h e g contains except for zero a closed and a dense vector space having maximal dimension as well as a maximally generated free algebra the results obtained complete a number of previous ones by several authors

discriminatively regularized least squares classification over the past decades regularization theory is widely applied in various areas of machine learning to derive a large family of novel algorithms traditionally regularization focuses on smoothing only and does not fully utilize the underlying discriminative knowledge which is vital for classification in this paper we propose a novel regularization algorithm in the least squares sense called discriminatively regularized least squares classification drlsc method which is specifically designed for classification inspired by several new geometrically motivated methods drlsc directly embeds the discriminative information as well as the local geometry of the samples into the regularization term so that it can explore as much underlying knowledge inside the samples as possible and aim to maximize the margins between the samples of different classes in each local area furthermore by embedding equality type constraints in the formulation the solutions of drlsc can follow from solving a set of linear equations and the framework naturally contains multi class problems experiments on both toy and real world problems demonstrate that drlsc is often superior in classification performance to the classical regularization algorithms including regularization networks support vector machines and some of the recent studied manifold regularization techniques

some results and open questions on spaceability in function spaces a subset m of a topological vector space x is called lineable respectively spaceable in x if there exists an infinite dimensional linear space respectively an infinite dimensional closed linear space y subset of m boolean or 0 in this article we prove that for every infinite dimensional closed subspace x of c 0 1 the set of functions in x having infinitely many zeros in 0 1 is spaceable in x we discuss problems related to these concepts for certain subsets of some important classes of banach spaces such as c 0 1 or muntz spaces we also propose several open questions in the field and study the properties of a new concept that we call the oscillating spectrum of subspaces of c 0 1 as well as oscillating and annulling properties of subspaces of c 0 1