how initial conditions affect generalization performance in large networks generalization is one of the most important problems in neural network research it is influenced by several factors in the network design such as network size weight decay factor and others we show here that the initial weight distribution for gradient decent training algorithms is one other factor that influences generalization the initial conditions guide the training algorithm to search particular places of the weight space for instance small initial weights tend to result in low complexity networks and therefore can effectively act as a regularization factor we propose a novel network complexity measure which is helpful in shedding insight into the phenomenon as well as in studying other aspects of generalization

the computation of the principal square roots of centrosymmetric h matrices in this paper two algorithms for computing the principal square root of a centrosymmetric h matrix with positive diagonal entries are proposed it is showed that our algorithms ensure significant savings in computational costs as compared to the case of an arbitrary h matrix a with positive diagonal entries

vapnik chervonenkis bounds for generalization the authors review the vapnik and chervonenkis theorem as applied to the problem of generalization by combining some of the technical modifications proposed in the literature they derive tighter bounds and a new version of the theorem bounding the accuracy in the estimation of generalization probabilities from finite samples a critical discussion and comparison with the results from statistical mechanics is given

weakly convergent expansions of a plane wave and their use in fourier integrals the fourier transform of an irreducible spherical tensor is normally computed with the help of the rayleigh expansion of a plane wave in terms of spherical bessel functions and spherical harmonics the angular integrations are then trivial however the remaining radial integral containing a spherical bessel function may be so complicated that the applicability of fourier transformation is severely restricted as an alternative the use of weakly convergent expansions of a plane wave in terms of complete orthonormal sets of functions is suggested the weakly convergent expansions of a plane wave are constructed in such a way that their application in fourier integrals leads to expansions of the fourier or inverse fourier transform that converge with respect to the norm of either the hilbert space l2 r3 or the sobolev space w 1 2 r3 accordingly these weakly convergent expansions may be viewed as distributions that are defined on either l2 r3 or w 1 2 r3 the properties of some complete orthonormal se

four types of learning curves if machines are learning to make decisions given a number of examples the generalization error e t is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples the generalization error decreases as t increases and the curve e t is called a learning curve the present paper uses the bayesian approach to show that given the annealed approximation learning curves can be classified into four asymptotic types if the machine is deterministic with noiseless teacher signals then 1 e at 1 when the correct machine parameter is unique and 2 e at 2 when the set of the correct parameters has a finite measure if the teacher signals are noisy then 3 e at 1 2 for a deterministic machine and 4 e c at 1 for a stochastic machine

linear readout of object manifolds objects are represented in sensory systems by continuous manifolds due to sensitivity of neuronal responses to changes in physical features such as location orientation and intensity what makes certain sensory representations better suited for invariant decoding of objects by downstream networks we present a theory that characterizes the ability of a linear readout network the perceptron to classify objects from variable neural responses we show how the readout perceptron capacity depends on the dimensionality size and shape of the object manifolds in its input neural representation

learning of higher order perceptrons with tunable complexities we study learning from examples by higher order perceptrons which realize polynomially separable rules the model complexities of the networks are made tunable by varying the relative orders of different monomial terms we analyse the learning curves of higher order perceptrons when the gibbs algorithm is used for training it is found that learning occurs in a stepwise manner this is because the number of examples needed to constrain the corresponding phase space component scales differently

computing the square roots of matrices with central symmetry for computing square roots of a nonsingular matrix a which are functions of a two well known fast and stable algorithms which are based on the schur decomposition of a were proposed by bjork and hammarling a bjork s hammarling a schur method for the square root of a matrix linear alg appl 52 53 1983 127 140 for square roots of general complex matrices and by higham n j higham computing real square roots of a real matrix linear alg appl 88 89 1987 405 430 for real square roots of real matrices in this paper we further consider the computation of the square roots of matrices with central symmetry we first investigate the structure of the square roots of these matrices and then develop several algorithms for computing the square roots we show that our algorithms ensure significant savings in computational costs as compared to the use of standard algorithms for arbitrary matrices

finite size effects and bounds for perceptron models ablncl in lhis paper we consider two main aspsln of the binary perccptron problem the maximal capacity when random paltem are stored model a and its generalhation ability model b we have extended previous numerical estimates of critical capacilies and studied thermal properties of systems of small sizes to lest recent replica predictions we have also considered some simpler versions of thest models the discrete spherical vemions can be salved exactly using gardner s replica calculation for the spherical model and are shown to give a rigorous upper bound and lower bound on the capacities of models a and b respectively iby vemions of models a and b are soivcd in detail and provide information which is useful for interpreting the bnile size eeeciects present in the numerical studies of models a and b

necessary and sufficient conditions for constructing orthonormal wavelet bases this paper proves a previous conjecture of the author characterizing sequences h l2 z that yield orthonormal wavelet bases of l2 r in terms of the multiplicity of the eigenvalue 1 of an operator associated to h the proof utilizes a result of cohen characterizing these sequences in terms of the real zeros of their fourier transforms the mapping from sequences to wavelets is shown to define a continuous mapping from a subset of l2 z into l2 r related conjectures are discussed

anti pairing in learning of a neural network with redundant hidden units we study the statistical mechanics of learning from examples between the two layered committee machines with different numbers of hidden units using the replica theory the number m of hidden units of the student network is larger than the number mt of those of the target network called the teacher we choose the networks to have binary synaptic weights 1 which makes it possible to compare the calculation with the monte carlo simulation we propose an effective teacher as a virtual target network which has the same m hidden units as the student and gives identical outputs with those of the original teacher this is a way of making a conjecture for a ground state of a thermodynamic system given by the weights of the effective teacher in our study we suppose that the weights on mt hidden units of the effective teacher are the same as those of the original teacher while those on m mt redundant hidden units are composed of anti pairs 1 1 with probability 1 p in the limit p 0 for p 0 exact there are no terms related to the effective teacher in the calculation for the contributions of anti pairs to outputs are exactly cancelled in the limit p 0 however we find that the learnt weights of the student are actually equivalent to those of the suggested effective teacher which is not possible from the calculation for p 0 p plays the role of a symmetry breaking parameter for anti pairing ordering which is analogous to the magnetic field for the ising model a first order phase transition is found to be signalled by breaking of symmetry in permuting hidden units above a critical number of examples the student is shown to learn perfectly the effective teacher anti pairing can be measured by a set of order parameters zero in the permutation symmetric phase and nonzero in the permutation symmetry breaking phase results from the monte carlo simulation are shown to be in good agreement with those from the replica calculation

some properties of centrosymmetric matrices in this paper some properties of centrosymmetric matrices which often appear in the construction of orthonormal wavelet basis in wavelet analysis are investigated

approximation dimension reduction and nonconvex optimization using linear superpositions of gaussians this paper concerns neural network approaches to function approximation and optimization using linear superposition of gaussians or what are popularly known as radial basis function rbf networks the problem of function approximation is one of estimating an underlying function f given samples of the form y sub i x sub i i 1 2 n with y sub i f x sub i when the dimension of the input is high and the number of samples small estimation of the function becomes difficult due to the sparsity of samples in local regions the authors find that this problem of high dimensionality can be overcome to some extent by using linear transformations of the input in the gaussian kernels such transformations induce intrinsic dimension reduction and can be exploited for identifying key factors of the input and for the phase space reconstruction of dynamical systems without explicitly computing the dimension and delay they present a generalization that uses multiple linear projections onto scalars and successive rbf networks mlprbf that estimate the function based on these scaler values they derive some key properties of rbf networks that provide suitable grounds for implementing efficient search strategies for nonconvex optimization within the same framework

how tight are the vapnik chervonenkis bounds we describe a series of numerical experiments that measure the average generalization capability of neural networks trained on a variety of simple functions these experiments are designed to test the relationship between average generalization performance and the worst case bounds obtained from formal learning theory using the vapnik chervonenkis vc dimension blumer et al 1989 haussler et al 1990 recent statistical learning theories tishby et al 1989 schwartz et al 1990 suggest that surpassing these bounds might be possible if the spectrum of possible generalizations has a gap near perfect performance we indeed find that in some cases the average generalization is significantly better than the vc bound the approach to perfect performance is exponential in the number of examples m rather than the 1 m result of the bound however in these cases we have not found evidence of the gap predicted by the above statistical theories in other cases we do find the 1 m behavior of the vc bound

symmetric multisplitting of a symmetric positive definite matrix abstract a parallel symmetric multisplitting method for solving a symmetric positive system ax b is presented here the s p d symmetric positive definite matrix a need not be assumed in a special form e g the dissection form r e white siam j matrix anal appl 11 1990 69 82 the main tool for deriving our method is the diagonally compensated reduction cf o axelsson l kolotilina numer linear algebra appl 1 1994 155 177 the convergence of the presented parallel symmetric multisplitting method is also discussed by using this tool

the iterative methods for centrosymmetric matrices in this paper we give some new centrosymmetric splittings of a centrosymmetric matrix a and obtain corresponding iterative methods for a linear system ax b these iterative schemes are particularly based on the centrosymmetric property of a and aimed at reduction of the cost of computation and storage

generalization from educated teachers the authors study the generalization ability of a perceptron as a function of the structural similarity between teacher and student to this end they consider a teacher perceptron b designed to implement a certain number of random input output mappings and a student perceptron bold j trained with input output examples generated by the teacher if the pattern sets of teacher and student overlap different combinations of teacher and student learning rules result in different generalization probabilities generalization can be facilitated or can be hindered depending on the combination of learning rules and the storage levels of teacher and student analytical calculations of the generalization ability for several combinations of teacher and student learning rules are in very good agreement with numerical simulations

perfect learning and power law in learning from stochastic examples by ising perceptrons analysis under one step replica symmetry breaking ansatz we study learning from stochastic examples by perceptrons with ising weights in the framework of statistical mechanics under the one step replica symmetry breaking ansatz the behaviour of learning curves are classified according to some local property of the rules by which examples are drawn the conditions for the existence of perfect learning together with other behaviours of the learning curves are given precisely the results agree with those obtained by seung 1995 using a refined annealled approximation

the analytic structure of oseen flow past a sphere as a function of reynolds number the feasibility of van dyke s 28 proposal that one go from zero to infinite reynolds number r by computer extension of a stokes series in powers of r depends on the analytic structure of the flow as a function of r this structure is investigated in detail for oseen flow past a sphere we find that there is a doubly infinite array of simple pole singularities in the left half complex r plane each of which lies close to one of the n zeros of the modified bessel functions k n 1 2 r for some positive integer n consequently r infty is an accumulation point of poles and hence a nonisolated singularity we have extended the stokes series for the drag coefficient c d r to 66 terms in double precision and have sought the asymptotic behavior of c d r as r to infty neither pade approximants nor an euler transformation formed from our long series give good convergence we suggest that the asymptotic behavior of c d r is primarily as an expansion in powers of r 2 3 though

on line learning in the ising perceptron on line learning of both binary and continuous rules in an ising space is studied learning is achieved by using an artificial parameter a weight vector which is constrained to the surface of a hypersphere spherical constraint in the case of a binary rule the generalization error decays to zero super exponentially as exp c 2 where is the number of examples divided by n the size of the input vector and c 0 much faster learning is obtained in the case of continuous activation functions where the generalization error decays as exp e the number of steps required for perfect learning is estimated for both scenarios and compared with simulations

on learning perceptrons with binary weights we present an algorithm that pac learns any perceptron with binary weights and arbitrary threshold under the family of product distributions the sample complexity of this algorithm is of o n e 4 ln n and its running time increases only linearly with the number of training examples the algorithm does not try to find an hypothesis that agrees with all of the training examples rather it constructs a binary perceptron based on various probabilistic estimates obtained from the training examples we show that under the restricted case of the uniform distribution and zero threshold the algorithm reduces to the well known clipped hebb rule we calculate exactly the average generalization rate i e the learning curve of the algorithm under the uniform distribution in the limit of an infinite number of dimensions we find that the error rate decreases exponentially as a function of the number of training examples hence the average case analysis gives a sample complexity of o n ln 1 e a large improvement over the pac learning analysis the analytical expression of the learning curve is in excellent agreement with the extensive numerical simulations in addition the algorithm is very robust with respect to classification noise

learning from examples in fully connected committee machines the problem of learning from examples in two layer neural networks is studied within the framework of statistical mechanics a fully connected committee machine is trained to implement a task which is not linearly separable the generalization error as a function of the number of training examples per adjustable weight is calculated in the annealed approximation for both binary and continuous weights we find a first order transition with a discontinuous drop in the generalization error the transitions occur due to a specialization of the hidden units from a symmetric state to one with each hidden unit in the student network specialized on a corresponding unit in the teacher network the symmetric states of poor generalization remain metastable even for large training sets

relations between k s 1 potent matrices and different classes of complex matrices in this paper k s 1 potent matrices are considered a matrix a cn n is called k s 1 potent when kas 1k a where k is an involutory matrix and s 1 2 3 specifically k s 1 potent matrices are analyzed considering their relations to different classes of complex matrices these classes of matrices are s 1 generalized projectors k hermitian matrices normal matrices and matrices b cn n anti commuting with k or such that kb is involutory hermitian or normal in addition some new relations for k generalized centrosymmetric matrices have been derived

on the oseen drag on a sphere the method of induced forces is employed to evaluate the oseen drag on a sphere the accuracy of the values obtained for the drag coefficient for large reynolds numbers is increased by application of shanks s transformation on the basis of the improved values we obtain as numerical approximations for the exponent of the first term in the asymptotic expansion values approaching 0 660 for reynolds numbers between 10 2 and 10 6

towards more practical average bounds on supervised learning in this paper we describe a method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities the resulting theory provides a unified viewpoint of average case learning curves of concept learning and regression in realistic learning problems not necessarily within the bayesian framework the advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the vapnik chervonenkis vc theory and its alike and provides general insights into generalization besides the bounds on learning curves are directly related to the number of adjustable system weights although the theory is based on an approximation assumption and cannot apply to the worst case learning setting the precondition of the assumption is mild and the approximation itself is only a sufficient condition for the validity of the theory we illustrate the results with numerical simulations and apply the theory to examining the generalization ability of combination of neural networks

dynamics of learning and generalization in perceptrons with constraints abstract depending on the kind of constraints imposed on the weights of a perceptron learning can be a combinatorially hard problem as an example of this type i discuss a perception with binary weights comparing results obtained from replica theory dynamic mean field theory and simulated annealing contrary to the replica calculation dynamics yields information about the performance of a polynomial algorithm in a situation where the best solution cannot be found in polynomial time i also discuss improved learning algorithms and results for finite size perceptrons

temporal evolution of generalization during learning in linear networks we study generalization in a simple framework of feedforward linear networks with n inputs and n outputs trained from examples by gradient descent on the usual quadratic error function we derive analytical results on the behavior of the validation function corresponding to the lms error function calculated on a set of validation patterns we show that the behavior of the validation function depends critically on the initial conditions and on the characteristics of the noise under certain simple assumptions if the initial weights are sufficiently small the validation function has a unique minimum corresponding to an optimal stopping time for training for which simple bounds can be calculated there exists also situations where the validation function can have more complicated and somewhat unexpected behavior such as multiple local minima at most n of variable depth and long but finite plateau effects additional results and possible extensions are briefly discussed

generalized analytic functions in axially symmetric oseen flows a class of generalized analytic functions arising from the oseen equations in the axially symmetric case has been identified for this class of functions the generalized cauchy integral formula has been obtained and a series representation for the region exterior to a sphere has been constructed the velocity field of the axially symmetric oseen flow has been represented in terms of two generalized analytic functions and it has been shown that for an exterior oseen flow problem those functions are uniquely determined provided that they both vanish at infinity also the pressure and vorticity have been determined as the real and imaginary parts of the two functions representing the velocity field and the drag exerted on a solid body of revolution in the axially symmetric oseen flow has been expressed in terms of one of the involved generalized analytic functions the problem of the axially symmetric oseen flow past a solid body of revolution has been reduced to an integral equation based on the gener

on line learning in a discrete state space on line learning of a rule given by an n dimensional ising perceptron is considered for the case when the student is constrained to take values in a discrete state space of size for l 2 no on line algorithm can achieve a finite overlap with the teacher in the thermodynamic limit however if l is on the order of hebbian learning does achieve a finite overlap

some properties of generalized k centrosymmetric h matrices every nxn generalized k centrosymmetric matrix a can be reduced into a 2x2 block diagonal matrix see z liu h cao h chen a note on computing matrix vector products with generalized centrosymmetric centrohermitian matrices appl math comput 169 2 2005 1332 1345 this block diagonal matrix is called the reduced form of the matrix a in this paper we further investigate some properties of the reduced form of these matrices and discuss the square roots of these matrices finally exploiting these properties the development of structure preserving algorithms for certain computations for generalized k centrosymmetric h matrices is discussed

on a variational principle for the drag in linear hydrodynamics it has been demonstrated several times that the method of induced forces is a suitable tool to calculate the drag on a submerged body in various linear approximations of the navier stokes equation in this paper we show that this method may be derived from a variational principle the stationary value of the appropriate functional is the drag the derivation of the set of equations for the induced force moments is given explicitly for two hydrodynamically relevant problems namely a sphere moving slowly along the axis of a rotating viscous fluid and a sphere in oseen flow moreover the variational scheme is employed to study the influence of momentum convection on the drag on a sphere for small reynolds and taylor numbers already with a simple approximation for the induced force density the results are in reasonably good agreement with experimental data

generalization in a linear perceptron in the presence of noise the authors study the evolution of the generalization ability of a simple linear perceptron with n inputs which learns to imitate a teacher perceptron the system is trained on p alpha n example inputs drawn from some distribution and the generalization ability is measured by the average agreement with the teacher on test examples drawn from the same distribution the dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at alpha 1 when there are no errors static noise in the training examples if the examples are produced by an erroneous teacher overfitting is observed i e the generalization error starts to increase after a finite time of training it is shown that a weight decay of the same size as the variance of the noise errors on the teacher improves on the generalization and suppresses the overfitting the generalization error as a function of time is calculated numerically for various values of the parameters finally dynamic noise in the training is considered white noise on the input corresponds on average to a weight decay and can thus improve generalization whereas white noise on the weights or the output degrades generalization generalization is particularly sensitive to noise on the weights for alpha 1 where it makes the error constantly increase with time but this effect is also shown to be damped by a weight decay weight noise and output noise acts similarly above the transition at alpha 1

storage capacities of committee machines with overlapping and non overlapping receptive fields we present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number m of hidden units and spherical weights difficulties arise in the solution of this problem in the limit of large m in the case of overlapping receptive fields as the number of patterns increases both permutation symmetry and replica symmetry are broken which leads to the appearance of many order parameters and causes additional difficulty we observe that the relations among these order parameters yield a set of quantities which are small in the limit of large m making the asymptotic calculation tractable using the one step replica symmetry breaking scheme we compute the asymptotic value of the storage capacity per input unit in the limit of large m we find that the shift to the case of non overlapping receptive fields can be made easily we then find both values satisfy the bound of mitchison and durbin

no free lunch for early stopping we show that with a uniform prior on models having the same training error early stopping at some fixed training error above the training error minimum results in an increase in the expected generalization error

on the hss iteration methods for positive definite toeplitz linear systems we study the hss iteration method for large sparse non hermitian positive definite toeplitz linear systems which first appears in bai golub and ng s paper published in 2003 z z bai g h golub m k ng hermitian and skew hermitian splitting methods for non hermitian positive definite linear systems siam j matrix anal appl 24 2003 603 626 and hss stands for the hermitian and skew hermitian splitting of the coefficient matrix a in this note we use the hss iteration method based on a special case of the hss splitting where the symmetric part h 12 a a t is a centrosymmetric matrix and the skew symmetric part s 12 a a t is a skew centrosymmetric matrix for a given toeplitz matrix hence fast methods are available for computing the two half steps involved in the hss and ihss iteration methods some numerical results illustrate their effectiveness

how to use expert advice we analyze algorithms that predict a binary value by combining the predictions of several prediction strategies called experts our analysis is for worst case situations i e we make no assumptions about the way the sequence of bits to be predicted is generated we measure the performance of the algorithm by the difference between the expected number of mistakes it makes on the bit sequence and the expected number of mistakes made by the best expert on this sequence where the expectation is taken with respect to the randomization in the predictins we show that the minimum achievable difference is on the order of the square root of the number of mistakes of the best expert and we give efficient algorithms that achieve this our upper and lower bounds have matching leading constants in most cases we then show how this leads to certain kinds of pattern recognition learning algorithms with performance bounds that improve on the best results currently know in this context we also compare our analysis to the case in which log loss is used instead of the expected number of mistakes

an algorithm for matrix extension and wavelet construction this paper gives a practical method of extending an n x r matrix p z r n with laurent polynomial entries in one complex variable z to a square matrix also with laurent polynomial entries if p z has orthonormal columns when z is restricted to the torus t it can be extended to a paraunitary matrix if p z has rank r for each z t it can be extended to a matrix with nonvanishing determinant on t the method is easily implemented in the computer it is applied to the construction of compactly supported wavelets and prewavelets from multiresolutions generated by several univariate scaling functions with an arbitrary dilation parameter

analytical approach to the oseen drag on a sphere at infinite reynolds number this paper analytically evaluates the oseen drag coefficient c d of a sphere at infinite reynolds number for this purpose the method of induced forces is used this method enables the obtainment of a series of rationale converging to the desired value for c d the best value for c d on the basis of this series truncated to its first 50 terms has only a three decimal accuracy the use of the shanks transformation in accelerated form on the truncated rational series improves the accuracy to nine decimals yielding c d 1 058250936

neural network training without spurious minima physics department national university la plata casilla de correro 727 1900 la plata argentina received 2 january 1996 in training student perceptions recourse to information theory concepts allows one to select the best workinghypothesis and obtain an exact solution for the associated probability distribution we apply this trainingscheme to perceptions with binary weights and show that no phase transition ensues by recourse to ourapproach fast learning is guaranteed and trapping by spurious local minima is avoided s1063 651x 96 07505 8 pacs number s 87 10 1e 05 20 2y 02 70 2ci introduction

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state

average case learning curves for radial basis function networks the application of statistical physics to the study of the learning curves of feedforward connectionist networks has to date been concerned mostly with perceptron like networks recent work has extended the theory to networks such as committee machines and parity machines and an important direction for current and future research is the extension of this body of theory to further connectionist networks in this article we use this formalism to investigate the learning curves of gaussian radial basis function networks rbfns having fixed basis functions these networks have also been called generalized linear regression models we address the problem of learning linear and nonlinear realizable and unrealizable target rules from noise free training examples using a stochastic training algorithm expressions for the generalization error defined as the expected error for a network with a given set of parameters are derived for general gaussian rbfns for which all parameters including centers and sp