on the estimation of a probability density function by the maximum penalized likelihood method abstract a class of probability density estimates can be obtained by penalizing the likelihood by a functional which depends on the roughness of the logarithm of the density the limiting case of the estimates as the amount of smoothing increasing has a natural form which makes the method attractive for data analysis and which provides a rationale for a particular choice of roughness penalty the estimates are shown to be the solution of an unconstrained convex optimization problem and mild natural conditions are given for them to exist rates of consistency in various norms and conditions for asymptotic normality and approximation by a gaussian process are given thus breaking new ground in the theory of maximum penalized likelihood density estimation author

local asymptotics for polynomial spline regression in this paper we develop a general theory of local asymptotics for least squares estimates over polynomial spline spaces in a regression problem the polynomial spline spaces we consider include univariate splines tensor product splines and bivariate or multivariate splines on triangulations we establish asymptotic normality of the estimate and study the magnitude of the bias due to spline approximation the asymptotic normality holds uniformly over the points where the regression function is to be estimated and uniformly over a broad class of design densities error distributions and regression functions the bias is controlled by the minimum l norm of the error when the target regression function is approximated by a function in the polynomial spline space that is used to define the estimate the control of bias relies on the stability in l norm of l 2 projections onto polynomial spline spaces asymptotic normality of least squares estimates over polynomial or trigonometric polynomial spaces is also treated by the general theory in addition a preliminary analysis of additive models is provided

a putnam fuglede commutativity property for hilbert space operators abstract given hilbert space operators a b b h define a b and a b in b b h by a b x a x x b and a b x a x b x for each x b h an operator a b h satisfies the putnam fuglede properties respectively notation a pf respectively a pf if for every isometry v b h for which the equation a v x 0 respectively a v x 0 has a non trivial solution x b h the solution x also satisfies a v x 0 respectively a v x 0 we prove that an operator a b h is in pf if and only if it is in pf

generalization in a multi state neural network the generalization ability of an extremely dilute feedback neural network with multi state neurons is studied by means of a deterministic noiseless parallel dynamics the overlap with any one of a macroscopic number of binary full activity concepts is determined when the network is trained with examples of variable activity according to a hebbian learning algorithm that favours stable symmetric mixture states explicit results about the phase diagram and the generalization error are obtained for a network with three state neurons which remain inactive below a threshold it is shown that the generalization ability can be considerably enhanced either by training the network with low activity examples or by means of a moderate increase in

on wold type decomposition abstract the main result of this paper is some new equivalent condition for a contraction to be a direct sum of a unitary operator and a c 0 completely nonunitary contraction called wold type decomposition it is applied for p k quasihyponormal k paranormal k paranormal and of class q operators this decomposition is connected with the putnam fuglede property additionally we show an example of operator of class q with norm 1 but without svep so not paranormal finally we remark that each k paranormal operator is k 1 paranormal

rate of convergence in density estimation using neural networks given n i i d observations xi ni 1 taking values in a compact subset of rd such that p denotes their common probability density function we estimate p from an exponential family of densities based on single hidden layer sigmoidal networks using a certain minimum complexity density estimation scheme assuming that p possesses a certain exponential representation we establish a rate of convergence independent of the dimension d for the expected hellinger distance between the proposed minimum complexity density estimator and the true underlying density p

thermodynamic properties of fully connected q ising neural networks the thermodynamic and retrieval properties of fully connected q ising networks are studied in the replica symmetric mean field approximation in particular capacity gain parameter and capacity temperature phase diagrams are derived for q 3 4 and q infinity and different distributions of the stored patterns furthermore the optimal gain function is determined in order to obtain the best performance where appropriate the results are compared with the diluted and layered versions of these models

minimum complexity regression estimation with weakly dependent observations the minimum complexity regression estimation framework barron 1991 barron and cover 1991 and rissanen 1989 is a general data driven methodology for estimating a regression function from a given list of parametric models using independent and identically distributed i i d observations we extend barron s regression estimation framework to m dependent observations and to strongly mixing observations in particular we propose abstract minimum complexity regression estimators for dependent observations which may be adapted to a particular list of parametric models and establish upper bounds on the statistical risks of the proposed estimators in terms of certain deterministic indices of resolvability assuming that the regression function satisfies a certain fourier transform type representation we examine minimum complexity regression estimators adapted to a list of parametric models based on neural networks and by using the upper bounds for the abstract estimators we establish rates of convergence for the statistical risks of these estimators also as a key tool we extend the classical bernstein inequality from i i d random variables to m dependent processes and to strongly mixing processes

density estimation and random variate generation using multilayer networks in this paper we consider two important topics density estimation and random variate generation we present a framework that is easily implemented using the familiar multilayer neural network first we develop two new methods for density estimation a stochastic method and a related deterministic method both methods are based on approximating the distribution function the density being obtained by differentiation in the second part of the paper we develop new random number generation methods our methods do not suffer from some of the restrictions of existing methods in that they can be used to generate numbers from any density provided that certain smoothness conditions are satisfied one of our methods is based on an observed inverse relationship between the density estimation process and random number generation we present two variants of this method a stochastic and a deterministic version we propose a second method that is based on a novel control formulation of the problem where a controller network is trained to shape a given density into the desired density we justify the use of all the methods that we propose by providing theoretical convergence results in particular we prove that the l sub spl infin convergence to the true density for both the density estimation and random variate generation techniques occurs at a rate o log log n n sup 1 spl epsi 2 where n is the number of data points and spl epsi can be made arbitrarily small for sufficiently smooth target densities this bound is very close to the optimally achievable convergence rate under similar smoothness conditions also for comparison the l sub 2 root mean square rms convergence rate of a positive kernel density estimator is o n sup 2 5 when the optimal kernel width is used we present numerical simulations to illustrate the performance of the proposed density estimation and random variate generation methods in addition we present an extended introduction and bibliography that serves as an overview and reference for the practitioner

projection estimation in multiple regression with application to functional anova models a general theory on rates of convergence of the least squares projection estimate in multiple regression is developed the theory is applied to the functional anova model where the multivariate regression function is modeled as a specified sum of a constant term main effects functions of one variable and selected interaction terms functions of two or more variables the least squares projection is onto an approximating space constructed from arbitrary linear spaces of functions and their tensor products respecting the assumed anova structure of the regression function the linear spaces that serve as building blocks can be any of the ones commonly used in practice polynomials trigonometric polynomials splines wavelets and finite elements the rate of convergence result that is obtained reinforces the intuition that low order anova modeling can achieve dimension reduction and thus overcome the curse of dimensionality moreover the components of the projection estimate in an appropriately defined anova decomposition provide consistent estimates of the corresponding components of the regression function when the regression function does not satisfy the assumed anova form the projection estimate converges to its best approximation of that form

optimal capacity of graded response perceptrons optimal capacities of perceptrons with graded input output relations are computed within the gardner approach 1988 the influence of desired output precision stability with respect to input errors and output pattern statistics are analysed and discussed

improving point and interval estimators of monotone functions by rearrangement suppose that a target function is monotonic and an available original estimate of this target function is not monotonic rearrangements univariate and multivariate transform the original estimate to a monotonic estimate that always lies closer in common metrics to the target function furthermore suppose an original confidence interval which covers the target function with probability at least 1 is defined by an upper and lower endpoint functions that are not monotonic then the rearranged confidence interval defined by the rearranged upper and lower endpoint functions is monotonic shorter in length in common norms than the original interval and covers the target function with probability at least 1 we illustrate the results with a growth chart example

on the overlap dynamics of multi state neural networks with a finite number of patterns neural networks with multi state neurons are studied in the case of low loading for symmetric couplings satisfying a certain positivity condition a lyapunov function is shown to exist in the space of overlaps between the instantaneous microscopic state of the system and the learned patterns furthermore an algorithm is derived for zero temperature to determine all the fixed points as an illustration the three state model is worked out explicitly for hebbian couplings for finite temperature the time evolution of the overlap is studied for couplings which need not be symmetric the stability properties are discussed in detail for the three state model for asymmetric couplings limit cycle behaviour is shown to be possible

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

density estimation by stochastic complexity the results by p hall and e j hannan 1988 on optimization of histogram density estimators with equal bin widths by minimization of the stochastic complexity are extended and sharpened in two separate ways as the first contribution two generalized histogram estimators are constructed the first has unequal bin widths which together with the number of the bins are determined by minimization of the stochastic complexity using dynamic programming the other estimator consists of a mixture of equal bin width estimators each of which is defined by the associated stochastic complexity as the main contribution in the present work two theorems are proved which together extend the universal coding theorems to a large class of data generating densities the first gives an asymptotic upper bound for the code redundancy in the order of magnitude achieved with a special predictive type of histogram estimator which sharpens a related bound the second theorem states that this bound cannot be improved upon by any code whatsoever

statistical mechanics for neural networks with continuous time dynamics the present paper is intended to summariz our rent knowledge about the long time behaviour of nehwrks of graded response neurons with continuous time dynamics we demonstrate the workings of our previously developed statistical mechanical approach to continuous time dynamics by applying it to networks with various forms of synaptic organization leaming rules and neural composition neuron types as encoded in gain functions as well as to networks varying with respect to the ensemble of stored data unbiased and low activity patterns we present phase diagrams and compute distributions of local fields for a variety of examples local field distributions are found to deviate from the gaussian form obtained for stochastic neumns in the wntexi of the replica approach a solution to the low firing rates problem within the framework of nets of analogue neurons is also briefly discussed finally the statisticalmechanical approach to the analysis of continuowtime dynamics is extended to include effects of fast stochastic noise detailed balance solutions are shown to be unique and of canonical form governed by hamiltonians which exhibit a reciprocity relation between potential dynamics and fuing rate dynamics for the potential dynamics the hamiltonian is given by the lyapounw function of the system expressed in terms of the fuing rates and it generates a gibbs distribution over fuing rate space for the noisy firinerate dynamics the same lyapounov function now expressed in terms of neural potentials generates a gibbs distribution over the space of these potentials as a consequence the firing rate dynamics will freeze in configurations saturating the neural input xtput relations whenever such saturation levels exist both types of stationary distribution are shown to exist only under unrealistic assumptions about the noise in the system

projection pursuit density estimation abstract the projection pursuit methodology is applied to the multivariate density estimation problem the resulting nonparametric procedure is often less biased than the kernel and near neighbor methods in addition graphical information is produced that can be used to help gain geometric insight into the multivariate data distribution

learning grey toned patterns in neural networks the problem of learning multi state patterns in neural networks is investigated an analysis of the space of couplings gardner approach yields the distribution of local fields the critical storage capacity alpha c and the minimum number of errors for an overloaded network for noisy local fields the classification error is minimized if the local fields of the patterns are allowed to lie in intervals of finite width a fast converging adaptive learning algorithm is presented which finds the coupling matrix of optimal stability

storing an extensive number of grey toned patterns in a neural network using multistate neurons grey toned patterns are pictures composed of pixels of several shades of grey the ability of neural networks using multistate neurons to store such patterns is systematically investigated if conventional generalizations of hopfield networks using analogue or soft neurons are considered it is impossible to stabilize these grey tones nevertheless it is shown that it can be done with networks that use neurons which have only a discrete set of possible activities this is demonstrated for the pseudo inverse rule for the synaptic couplings where only the stability of the patterns shrinks with increasing number q of grey tones one wants to store if the patterns are uncorrelated one can use the hebb rule and in this case the mean field theory is presented applying this rule the storage capacity decreases as q 2 with the number of grey tones

on the parallel dynamics of the q state potts and q ising neural networks using a probabilistic approach the parallel dynamics of theq state potts andq ising neural networks are studied at zero and at nonzero temperatures evolution equations are derived for the first time step and arbitraryq these formulas constitute recursion relations for the exact parallel dynamics of the extremely diluted asymmetric versions of these networks an explicit analysis including dynamical capacity temperature diagrams and the temperature dependence of the overlap is carried out forq 3 both types of models are compared

nonparametric density estimation the l 1 view differentiation of integrals consistency lower bounds for rates of convergence rates of convergence in l1 the automatic kernel estimate l1 and pointwise convergence estimates related to the kernel estimate and the histogram estimate simulation inequalities and random variate generation the transformed kernel estimate applications in discrimination operations on density estimates estimators based on orthogonal series index

information processing in three state neural networks we introduce networks of three state neurons where the additional state embodies the absence of information their dynamical behavior is studied from the standpoint of information processing these networks display strong pattern completion capabilities moreover inference naturally occurs between coherent patterns

parallel dynamics of fully connected q ising neural networks using a probabilistic approach the parallel dynamics of fully connected q ising neural networks is studied for arbitrary q a novel recursive scheme is set up to determine the time evolution of the order parameters through the evolution of the distribution of the local field taking into account all feedback correlations in contrast to extremely diluted and layered network architectures the local field is no longer normally distributed but contains a discrete part as an illustrative example an explicit analysis is carried out for the first four time steps for the case of the q 2 and q 3 model the results are compared with extensive numerical simulations and excellent agreement is found finally equilibrium fixed point equations are derived and compared with the thermodynamic approach based upon the replica symmetric mean field approximation

using specially designed exponential families for density estimation i this paper have y being portions of the real line or of the plane but the methodology applies just as well to higher dimensionalities and to more complicated spaces estimates of g y are traditionally constructed in two quite different ways by maximum likelihood fitting within some parametric family such as the normal or by nonparametric methods such as kernel density estimation these two methods can be combined by putting an exponential family through a nonparametric estimator the resulting hybrid estimators are the specially designed exponential families of the title figure 1 shows a simple example of this methodology the y are pain i scores for n s 67 women each obtained by averaging the results from a questionnaire administered after an operation the scale runs from y s 0 s w x

generalization ability of perceptrons with continuous outputs in this paper we examine the influence of different input output relations on the generalization ability of a single layer perceptron the input output relations can be linear binary or sigmoid with this choice we take into account most of the cases which are of present interest the generalization problem will be realizable or unrealizable if the input output relations for teacher and student are identical or not we show that sometimes it can have a positive effect on the generalization ability if one learns with errors

estimation of a regression function by maxima of minima of linear functions in this paper estimation of a regression function from independent and identically distributed random variables is considered estimates are defined by minimization of the empirical l2 risk over a class of functions which are defined as maxima of minima of linear functions results concerning the rate of convergence of the estimates are derived in particular it is shown that for smooth regression functions satisfying the assumption of single index models the estimate is able to achieve up to some logarithmic factor the corresponding optimal one dimensional rate of convergence hence under these assumptions the estimate is able to circumvent the so called curse of dimensionality the small sample behavior of the estimates is illustrated by applying them to simulated data

bishop s property β a commutativity theorem and the dynamics of class a s t operators abstract given a hilbert space operator a b h with polar decomposition a u a the class a s t 0 s t 1 consists of operators a b h such that a 2 t a t a 2 s a t t t s every class a s t operator is paranormal prominent amongst the subclasses of a s t operators are the class a 1 2 1 2 consisting of w hyponormal operators and the class a 1 1 consisting of semi quasihyponormal 16 p 93 or class a operators our aim here is threefold we prove that a s t operators satisfy i bishop s property thereby providing a proof of 6 theorem 3 1 and ii a putnam fuglede commutativity theorem thereby answering a question posed in 18 conjecture 2 4 we prove also an extension of 3 theorem 3 4 to prove that iii if an a s t operator is weakly supercyclic then it is a scalar multiple of a unitary operator

approximation of density functions by sequences of exponential families 1 introduction consider the estimation of a probability density func tion p x defined on a bounded interval we approximate the logarithm of the density by a basis function expansion consisting of polynomials splines or trigonometric series the expansion yields a regular exponential family within which we estimate the density by the method of maximum likelihood this method of density estimation arises by application of the principle of maxi mum entropy or minimum relative entropy subject to empirical constraints we show that if the logarithm of the density has r square integrable deriva tives f idr log p12 ox then the sequence of density estimators p converges to p in the sense of relative entropy kullback leibler fp log p pn at rate opr 1 m2r m n as m x c and m2 n 0 in the spline and trigonometric cases and m3 n o 0 in the polynomial case where m is the dimension of the family and n is the sample size boundary conditions are assumed for the density in the trigonometric case this convergence rate specializes to op r n 2r 2r 1 by setting m nl 2r 1 when the log density is known to have degree of smoothness at least r analogous convergence results for the relative entropy are shown to hold in general for any class of log density functions and sequence of finite dimensional linear spaces having l2 and l approximation properties the approximation of log densities using polynomials has previously been considered by neyman 1937 to define alternatives for goodness of fit tests by good 1963 as an application of the method of maximum entropy or minimum relative entropy by crain 1974 1976a b 1977 who demonstrates existence and consistency of the maximum likelihood estimator and by mead and

neural networks that use three state neurons the dynamics of a neural network which uses three state neurons 1 0 and 1 is solved exactly in the limit of non symmetric and highly dilute synapses recursion relations for the activity the fraction of non zero neurons and overlap of the network with a given pattern are derived which have three generic kinds of fixed points a retrieval fixed point a chaotic fixed point which corresponds to non zero activity but no overlap and a zero fixed point where all the neurons go to the 0 state as the non retrieval fixed points both have activities different from the retrieval fixed point one can easily tell whether a pattern has been recovered an analysis of which fixed points occur as a function of the thresholds and the storage ratio of the system yields remarkably rich phase diagrams optimising the threshold level can be very important especially when low activity patterns are stored a similar analysis can be applied to biased networks using two state 1 0 neurons finally one finds that mixture states which have an overlap with two patterns can be stabilised by a threshold in the networks using three state neurons this property allows larger higher activity memories to be naturally constructed out of smaller ones

a finite size scaling investigation for q state hopfield models storage capacity and basins of attraction the storage capacity of a q state hopfield network is determined via finite size scaling for parallel dynamics and q or 8 the results are in good agreement with theoretical predictions by rieger the basins of attraction and other associative memory properties are discussed for q 4 6 a self controlling q state model with improved basins of attraction is proposed

synchronous versus sequential updating in the three state ising neural network with variable dilution abstract the q 3 state ising neural network with synchronous updating and variable dilution is discussed starting from the appropriate hamiltonians the thermodynamic and retrieval properties are examined using replica mean field theory the appearance and properties of two cycles are discussed capacity temperature phase diagrams are derived for several values of the pattern activity and different gradations of dilution and the information content is calculated it is found that the asymptotic behaviour is rather similar to that for sequential updating the retrieval region is enhanced marginally but the spin glass region is visibly enlarged only the presence of self coupling can enlarge the retrieval region substantially the dynamics of the network is studied for general q and both synchronous and sequential updating using an extension of the generating function technique the differences with the signal to noise approach are outlined typical flow diagrams for the q 3 overlap order parameter are presented

gardner optimal capacity of the diluted blume emery griffiths neural network abstract the optimal capacity of a diluted blume emery griffiths neural network is studied as a function of the pattern activity and the embedding stability using the gardner entropy approach annealed dilution is considered cutting some of the couplings referring to the ternary patterns themselves and some of the couplings related to the active patterns both simultaneously synchronous dilution or independently asynchronous dilution through the de almeida thouless criterion it is found that the replica symmetric solution is locally unstable as soon as there is dilution the distribution of the couplings shows the typical gap with a width depending on the amount of dilution but this gap persists even in cases where a particular type of coupling plays no role in the learning process

minimum complexity density estimation the authors introduce an index of resolvability that is proved to bound the rate of convergence of minimum complexity density estimators as well as the information theoretic redundancy of the corresponding total description length the results on the index of resolvability demonstrate the statistical effectiveness of the minimum description length principle as a method of inference the minimum complexity estimator converges to true density nearly as fast as an estimator based on prior knowledge of the true subclass of densities interpretations and basic properties of minimum complexity estimators are discussed some regression and classification problems that can be examined from the minimum description length framework are considered

improved rates and asymptotic normality for nonparametric neural network estimators we obtain an improved approximation rate in sobolev norm of r sup 1 2 spl alpha d 1 for a large class of single hidden layer feedforward artificial neural networks ann with r hidden units and possibly nonsigmoid activation functions when the target function satisfies certain smoothness conditions here d is the dimension of the domain of the target function and spl alpha spl isin 0 1 is related to the smoothness of the activation function when applying this class of anns to nonparametrically estimate train a general target function using the method of sieves we obtain new root mean square convergence rates of op n log n sup 1 2 spl alpha d 1 4 1 spl alpha d 1 op n sup 1 4 by letting the number of hidden units spl tau sub n increase appropriately with the sample size number of training examples n these rates are valid for i i d data as well as for uniform mixing and absolutely regular spl beta mixing stationary time series data in addition the rates are fast enough to deliver root n asymptotic normality for plug in estimates of smooth functionals using general ann sieve estimators as interesting applications to nonlinear time series we establish rates for ann sieve estimators of four different multivariate target functions a conditional mean a conditional quantile a joint density and a conditional density we also obtain root n asymptotic normality results for semiparametric model coefficient and average derivative estimators

mean field theory of em algorithm for bayesian grey scale image restoration the em algorithm for the bayesian grey scale image restoration is investigated in the framework of the mean field theory our model system is identical to the infinite range random field ising model the maximum marginal likelihood method is applied to the determination of hyper parameters we calculate both the data averaged mean square error between the original image and its maximizer of posterior marginal estimate and the data averaged marginal likelihood function exactly after evaluating the hyper parameter dependence of the data averaged marginal likelihood function we derive the em algorithm which updates the hyper parameters to obtain the maximum likelihood estimate analytically the time evolutions of the hyper parameters and so called q function are obtained the relation between the speed of convergence of the hyper parameters and the shape of the q function is explained from the viewpoint of dynamics

large sample inference for log spline models let f be a continuous and positive unknown density on a known compact interval mathscr y let f denote the distribution function of f and let q f 1 denote its quantile function a finite parameter exponential family model based on b splines is constructed maximum likelihood estimation of the parameters of the model based on a random sample of size n from f yields estimates hat f f and hat q of f f and q respectively under mild conditions if the number of parameters tends to infinity in a suitable manner as n rightarrow infty these estimates achieve the optimal rate of convergence the asymptotic behavior of the corresponding confidence bounds is also investigated in particular it is shown that the standard errors of hat f and hat q are asymptotically equal to those of the usual empirical distribution function and empirical quantile function

commutativity theorems for normaloid hilbert space operators given hilbert space operators a b b h let a b x la rb x ax xb and a b x larb x axb for all x b h the pair a b satisfies the putnam fuglede commutativity property respectively if ab 1 0 a b 1 0 respectively ab 1 1 0 a b 1 1 0 normaloid operators do not satisfy either of the properties or this paper considers commutativity properties a b 1 0 a b 1 0 and a b 1 0 a b 1 0 for some choices of scalars and normaloid operators a b starting with normaloid a b b h such that the isolated points of their spectrum are normal eigenvalues of the operator we prove that a if 0 iso larb then a b 1 0 a b 1 0 b if 0 p a p b and 0 iso la r b then a b 1 0 a b 1 0 let t denote the peripheral spectrum of the operator t if a b are normaloid then i either dim b h a b b h for all a b or there exists a a b p a b ii if x is hilbert schmidt and axb x 0 for some a b then a xb x 0 iii if v b h is an isometry a a 1 0 a 1 0 and axv x 0 or ax xv 0 for some x b h then a xv x 0 resp a x xv 0

thermodynamics of fully connected blume emery griffiths neural networks the thermodynamic and retrieval properties of fully connected blume emery griffiths networks are studied using replica mean field theory these networks can be considered as generalizations of the hopfield model to the storage of ternary patterns capacity temperature phase diagrams are derived for several values of the pattern activity it is found that the retrieval phase is the largest in comparison with other three state neuron models furthermore the meaning and stability of the so called quadrupolar phase is discussed as a function of both the temperature and the pattern activity where appropriate the results are compared with the diluted version of the model