separation of unistochastic matrices from the double stochastic ones recovery of a 3 3 unitary matrix from experimental data the aim of the paper is to provide a constructive method for recovering a unitary matrix from experimental data since there is a natural immersion of unitary matrices within the set of double stochastic ones the problem to solve is to find necessary and sufficient criteria that separate the two sets a complete solution is provided for the three dimensional case accompanied by a 2 test necessary for the reconstruction of a unitary matrix from error affected data

the non linear equations of motion of pipes conveying fluid abstract the non linear equations of motion of pipes conveying fluid are derived in simple and accessible terms by energy and newtonian methods different derivations are made for cantilevered pipes where the centreline is assumed inextensible and for pipes with both ends fixed it is shown that the derivations the origin of the various terms and the structure of the equations are distinctly different in these two cases the equations of motion are then compared with those already in existence e g by holmes rousselet and herrmann and lundgren sethna and bajaj at the same time clarifying the derivations and assumptions made and discussing the validity and completeness of the final equations some of the equations are found to be fully correct indeed superior to those derived here while others are found to be deficient because of assumptions made or inconsistencies in the derivations for pipes with both ends fixed the equations given in this paper are considered to be the most complete and correct

generalization properties of multilayered neural networks generalization properties of multilayered neural networks with binary couplings are studied in the high temperature limit the transition to the perfect generalization phase is evaluated for systems with an arbitrary number of layers it is found that the thermodynamic transition occurs for a number of examples lower than for the perceptron but the opposite occurs for the transition in which the poor generalization solution disappears the generalization error is also decomposed according to the contributions coming from different numbers of hidden neurons that have a wrong sign in the internal field this allows the authors to describe the generalization behaviour of the hidden neurons

time series based bifurcation diagram reconstruction abstract we consider the problem of reconstructing bifurcation diagrams bds of maps using time series this study goes along the same line of ideas presented by tokunaga et al physica d 79 1994 348 and tokuda et al physica d 95 1996 380 the aim is to reconstruct the bd of a dynamical system without the knowledge of its functional form and its dependence on the parameters instead time series at different parameter values assumed to be available are used a three layer fully connected neural network is employed in the approximation of the map the task of the network is to learn the dynamics of the system as function of the parameters from the available time series we determine a class of maps for which one can always find a linear subspace in the weight space of the network where the network s bifurcation structure is qualitatively the same as the bifurcation structure of the map we discuss a scheme in locating this subspace using the time series we further discuss how to recognize time series generated by this class of maps finally we propose an algorithm in reconstructing the bds of this class of maps using predictor functions obtained by neural network this algorithm is flexible so that other classes of predictors apart from neural networks can be used in the reconstruction

inferring hidden states in langevin dynamics on large networks average case performance we present average performance results for dynamical inference problems in large networks where a set of nodes is hidden while the time trajectories of the others are observed examples of this scenario can occur in signal transduction and gene regulation networks we focus on the linear stochastic dynamics of continuous variables interacting via random gaussian couplings of generic symmetry we analyze the inference error given by the variance of the posterior distribution over hidden paths in the thermodynamic limit and as a function of the system parameters and the ratio ensuremath alpha between the number of hidden and observed nodes by applying kalman filter recursions we find that the posterior dynamics is governed by an effective drift that incorporates the effect of the observations we present two approaches for characterizing the posterior variance that allow us to tackle respectively equilibrium and nonequilibrium dynamics the first appeals to random matrix theory and reveals average spectral properties of the inference error and typical posterior relaxation times the second is based on dynamical functionals and yields the inference error as the solution of an algebraic equation

generalization errors of the simple perceptron to find an exact form for the generalization error of a learning machine is an open problem even in the simplest case simple perceptron learning we introduce a new approach to tackle the problem the generalization error of the simple perceptron is expressed as a linear combination of extreme values of inputs with the help of extreme value theory in statistics we then obtain an exact form of the generalization error of the simple perceptron in the case of the worst learning generalization errors of the higher order perceptron taking the form of an inverse power law in the number of examples are also considered

on biunimodular vectors for unitary matrices abstract a biunimodular vector of a unitary matrix a u n is a vector v t n c n such that av t n as well over the last 30 years the sets of biunimodular vectors for fourier matrices have been the object of extensive research in various areas of mathematics and applied sciences here we broaden this basic harmonic analysis perspective and extend the search for biunimodular vectors to arbitrary unitary matrices this search can be motivated in various ways the main motivation is provided by the fact that the existence of biunimodular vectors for an arbitrary unitary matrix allows for a natural understanding of the structure of all unitary matrices

perturbative treatments and learning techniques the replica trick rt of statistical mechanics has become a very useful tool for the investigation of complex systems in general and in particular for studying learning and generalization processes in neural networks nn 1 2 the trick overcomes the difficulty of performing an ensemble average of the logarithm of the partition function z by that of averaging z over n replicas of the original network with n a very large integer for feedforward single layered nn 3 4 the rt ansatz has been proved to be a valuable tool for the learning of a rule on the basis of a suitable set of examples in particular the full quenched theory has been carefully studied within the framework of a replica symmetry approximation 5 however as the temperature drops symmetry breakings may invalidate this approximation so that an approach to the full quenched theory that incorporates disorder effects due to improper examples may be of some interest in the present effort we wish to study in the spirit of a second order approximation noise effects in the training set unavoidable in any realistic setting the noise will here be the result of letting just part of the examples to be produced by the perceptron teacher pt the rest are to be randomly selected bad examples two types of situations are to be confronted learnable and unlearnable rules 6 for the former there is at least a vector in the concomitant weight space that can learn the rule in an exact fashion the latter arises mostly in cases of architectural mismatch in such a situation the training error can never vanish the question to be answered is can a perceptron trained under these circumstances correctly respond to queries posed by a pt in other words is the rule underlying the good examples a learnable one we will show here that these questions can be adequately dealt with the paper is organized as follows section ii is devoted to a brief recapitulation of basic concepts concerning the rt while sec iii deals with the thermodynamics of the situation that interests us here a second order high temperature approximation is derived boolean perceptrons with ising weights on the learning curves are the subject of sec iv finally some conclusions are drawn in sec v ii the replica method

some formulas for invariant phases of unitary matrices by jarlskog we describe a calculation of jarlskog s determinant in the case of n 4 in detail next we investigate some formulas for invariant phases of unitary matrices and derive some explicit relations for them

vapnik chervonenkis bounds for generalization the authors review the vapnik and chervonenkis theorem as applied to the problem of generalization by combining some of the technical modifications proposed in the literature they derive tighter bounds and a new version of the theorem bounding the accuracy in the estimation of generalization probabilities from finite samples a critical discussion and comparison with the results from statistical mechanics is given

four types of learning curves if machines are learning to make decisions given a number of examples the generalization error e t is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples the generalization error decreases as t increases and the curve e t is called a learning curve the present paper uses the bayesian approach to show that given the annealed approximation learning curves can be classified into four asymptotic types if the machine is deterministic with noiseless teacher signals then 1 e at 1 when the correct machine parameter is unique and 2 e at 2 when the set of the correct parameters has a finite measure if the teacher signals are noisy then 3 e at 1 2 for a deterministic machine and 4 e c at 1 for a stochastic machine

a numerical study on learning curves in stochastic multilayer feedforward networks the universal asymptotic scaling laws proposed by amari et al are studied in large scale simulations using a cm5 small stochastic multilayer feedforward networks trained with backpropagation are investigated in the range of a large number of training patterns t the asymptotic generalization error scales as 1 t as predicted for a medium range t a faster 1 t2 scaling is observed this effect is explained by using higher order corrections of the likelihood expansion it is shown for small t that the scaling law changes drastically when the network undergoes a transition from strong overfitting to effective learning

linear readout of object manifolds objects are represented in sensory systems by continuous manifolds due to sensitivity of neuronal responses to changes in physical features such as location orientation and intensity what makes certain sensory representations better suited for invariant decoding of objects by downstream networks we present a theory that characterizes the ability of a linear readout network the perceptron to classify objects from variable neural responses we show how the readout perceptron capacity depends on the dimensionality size and shape of the object manifolds in its input neural representation

learning of higher order perceptrons with tunable complexities we study learning from examples by higher order perceptrons which realize polynomially separable rules the model complexities of the networks are made tunable by varying the relative orders of different monomial terms we analyse the learning curves of higher order perceptrons when the gibbs algorithm is used for training it is found that learning occurs in a stepwise manner this is because the number of examples needed to constrain the corresponding phase space component scales differently

exponential or polynomial learning curves case based studies learning curves exhibit a diversity of behaviors such as phase transition however the understanding of learning curves is still extremely limited and existing theories can give the impression that without empirical studies e g cross validation one can probably do nothing more than qualitative interpretations in this note we propose a theory of learning curves based on the idea of reducing learning problems to hypothesistesting ones this theory provides a simple approach that is potentially useful for predicting and interpreting a diversity of learning curve behaviors qualitatively and quantitatively and it applies to finite training sample size and finite learning machine and for learning situations not necessarily within the bayesian framework we illustrate the results by examining some exponential learning curve behaviors observed in cohn and tesauro 1992 s experiment

how bad may learning curves be in this paper we motivate the need for estimating bounds on learning curves of average case learning algorithms when they perform the worst on training samples we then apply the method of reducing learning problems to hypothesis testing ones to investigate the learning curves of a so called ill disposed learning algorithm in terms of a system complexity the boolean interpolation dimension since the ill disposed algorithm behaves worse than ordinal ones and the boolean interpolation dimension is generally bounded by the number of system weights the results can apply to interpreting or to bounding the worst case learning curve in real learning situations this study leads to a new understanding of the worst case generalization in real learning situations which differs significantly from that in the uniform learnable setting via vapnik chervonenkis vc dimension analysis we illustrate the results with some numerical simulations

finite size effects and bounds for perceptron models ablncl in lhis paper we consider two main aspsln of the binary perccptron problem the maximal capacity when random paltem are stored model a and its generalhation ability model b we have extended previous numerical estimates of critical capacilies and studied thermal properties of systems of small sizes to lest recent replica predictions we have also considered some simpler versions of thest models the discrete spherical vemions can be salved exactly using gardner s replica calculation for the spherical model and are shown to give a rigorous upper bound and lower bound on the capacities of models a and b respectively iby vemions of models a and b are soivcd in detail and provide information which is useful for interpreting the bnile size eeeciects present in the numerical studies of models a and b

on the generalization ability of diluted perceptrons a linearly separable boolean function is teamed by a diluted perceptron with optimal stability a different level of dilution is allowed for teacher and student perceptron the learning algorithms used were the optimal annealed dilution and hebbian dilution the generalization ability i e the probability to recognize a pattern which has not been learned before is calculated in replica symmetry

on a generalized entropic uncertainty relation in the case of the qubit we revisit generalized entropic formulations of the uncertainty principle for an arbitrary pair of quantum observables in two dimensional hilbert space r nyi entropy is used as an uncertainty measure associated with the distribution probabilities corresponding to the outcomes of the observables we derive a general expression for the tight lower bound of the sum of r nyi entropies for any couple of positive entropic indices thus we have overcome the h lder conjugacy constraint imposed on the entropic indices by riesz thorin theorem in addition we present an analytical expression for the tight bound inside the square 0 1 2 2 in the plane and a semi analytical expression on the line it is seen that previous results are included as particular cases moreover we present a semi analytical suboptimal bound for any couple of indices in all cases we provide the minimizing states

anti pairing in learning of a neural network with redundant hidden units we study the statistical mechanics of learning from examples between the two layered committee machines with different numbers of hidden units using the replica theory the number m of hidden units of the student network is larger than the number mt of those of the target network called the teacher we choose the networks to have binary synaptic weights 1 which makes it possible to compare the calculation with the monte carlo simulation we propose an effective teacher as a virtual target network which has the same m hidden units as the student and gives identical outputs with those of the original teacher this is a way of making a conjecture for a ground state of a thermodynamic system given by the weights of the effective teacher in our study we suppose that the weights on mt hidden units of the effective teacher are the same as those of the original teacher while those on m mt redundant hidden units are composed of anti pairs 1 1 with probability 1 p in the limit p 0 for p 0 exact there are no terms related to the effective teacher in the calculation for the contributions of anti pairs to outputs are exactly cancelled in the limit p 0 however we find that the learnt weights of the student are actually equivalent to those of the suggested effective teacher which is not possible from the calculation for p 0 p plays the role of a symmetry breaking parameter for anti pairing ordering which is analogous to the magnetic field for the ising model a first order phase transition is found to be signalled by breaking of symmetry in permuting hidden units above a critical number of examples the student is shown to learn perfectly the effective teacher anti pairing can be measured by a set of order parameters zero in the permutation symmetric phase and nonzero in the permutation symmetry breaking phase results from the monte carlo simulation are shown to be in good agreement with those from the replica calculation

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state

tight bounds on transition to perfect generalization in perceptrons sudden transition to perfect generalization in binary perceptrons is investigated building on recent theoretical works of gardner and derrida 1989 and baum and lyuu 1991 we show the followi

the dynamics of a genetic algorithm for a simple learning problem a formalism for describing the dynamics of genetic algorithms gas using method s from statistical mechanics is applied to the problem of generalization in a perceptron with binary weights the dynamics are solved for the case where a new batch of training patterns is presented to each population member each generation which considerably simplifies the calculation the theory is shown to agree closely to simulations of a real ga averaged over many runs accurately predicting the mean best solution found for weak selection and large problem size the difference equations describing the dynamics can be expressed analytically and we find that the effects of noise due to the finite size of each training batch can be removed by increasing the population size appropriately if this population resizing is used one can deduce the most computationally efficient size of training batch each generation for independent patterns this choice also gives the minimum total number of training patterns used although using independent patterns is a very inefficient use of training patterns in general this work may also prove useful for determining the optimum batch size in the case where patterns are recycled

learnability and the vapnik chervonenkis dimension valiant s learnability model is extended to learning classes of concepts defined by regions in euclidean space e n the methods in this paper lead to a unified treatment of some of valiant s results along with previous results on distribution free convergence of certain pattern recognition algorithms it is shown that the essential condition for distribution free learnability is finiteness of the vapnik chervonenkis dimension a simple combinatorial parameter of the class of concepts to be learned using this parameter the complexity and closure properties of learnable classes are analyzed and the necessary and sufficient conditions are provided for feasible learnability

a note on factoring unitary matrices abstract the note concerns three factorizations of unitary matrices a recent factorization allowing to obtain any unitary matrix of dimension 2n from four unitary matrices of dimension n the cosine sine decomposition and a new factorization decomposing a given unitary matrix into a product of two orthogonal matrices and a diagonal unitary matrix

how tight are the vapnik chervonenkis bounds we describe a series of numerical experiments that measure the average generalization capability of neural networks trained on a variety of simple functions these experiments are designed to test the relationship between average generalization performance and the worst case bounds obtained from formal learning theory using the vapnik chervonenkis vc dimension blumer et al 1989 haussler et al 1990 recent statistical learning theories tishby et al 1989 schwartz et al 1990 suggest that surpassing these bounds might be possible if the spectrum of possible generalizations has a gap near perfect performance we indeed find that in some cases the average generalization is significantly better than the vc bound the approach to perfect performance is exponential in the number of examples m rather than the 1 m result of the bound however in these cases we have not found evidence of the gap predicted by the above statistical theories in other cases we do find the 1 m behavior of the vc bound

dynamic proximity of spatio temporal sequences recurrent networks can generate spatio temporal neural sequences of very large cycles having an apparent random behavior nonetheless a proximity measure between these sequences may be defined through comparison of the synaptic weight matrices that generate them following the dynamic neural filter dnf formalism we demonstrate this concept by comparing teacher and student recurrent networks of binary neurons we show that large sequences providing a training set well exceeding the cover limit allow for good determination of the synaptic matrices alternatively assuming the matrices to be known very fast determination of the biases can be achieved thus a spatio temporal sequence may be regarded as spatio temporal encoding of the bias vector we introduce a linear support vector machine svm variant of the dnf in order to specify an optimal weight matrix this approach allows us to deal with noise spatio temporal sequences generated by different dnfs with the same number of neurons may be compared by calculating correlations of the synaptic matrices of the reconstructed dnfs other types of spatio temporal sequences need the introduction of hidden neurons and or the use of a kernel variant of the svm approach the latter is being defined as a recurrent support vector network rsvn

the transition to perfect generalization in perceptrons several recent papers gardner and derrida 1989 gyrgyi 1990 sompolinsky et al 1990 have found using methods of statistical physics that a transition to perfect generalization occurs in training a simple perceptron whose weights can only take values 1 we give a rigorous proof of such a phenomena that is we show for 2 0821 that if at least n examples are drawn from the uniform distribution on 1 1 n and classified according to a target perceptron wt 1 1 n as positive or negative according to whether wtx is nonnegative or negative then the probability is 2 n that there is any other such perceptron consistent with the examples numerical results indicate further that perfect generalization holds for as low as 1 5

application of the differential transformation method to vibration analysis of pipes conveying fluid abstract in this paper a relatively new semi analytical method called differential transformation method dtm is generalized to analyze the free vibration problem of pipes conveying fluid with several typical boundary conditions the natural frequencies and critical flow velocities are obtained using dtm the results are compared with those predicted by the differential quadrature method dqm and with other results reported in the literature it is demonstrated that the dtm has high precision and computational efficiency in the vibration analysis of pipes conveying fluid

independent component analysis to enhance performances of karhunen loeve expansions for non gaussian stochastic processes application to uncertain systems abstract in the context of engineering systems an essential step in uncertainty quantification is the development of accurate and efficient representation of the random input parameters for such input parameters modeled as stochastic processes karhunen loeve expansion is a classical approach providing efficient representations using a set of uncorrelated but generally statistically dependent random variables the dependence structure among these random variables may be difficult to estimate statistically and is thus ignored in many practical applications this simplifying assumption of independence may lead to considerable errors in estimating the variability in the system state thus limiting the effectiveness of karhunen loeve expansion in certain cases in this paper independent component analysis is exploited to linearly transform the random variables used in karhunen loeve expansion resulting into a set of random variables exhibiting higher order decorrelation the stochastic wave equation is investigated for numerical illustration whereby the random stiffness coefficient is modeled as a non gaussian stochastic process under the assumption of independence among the random variables used in the karhunen loeve expansion and independent component analysis representations the latter provides more accurate statistical characterization of the output process for the specific cases examined

chaotic vibrations of circular cylindrical shells galerkin versus reduced order models via the proper orthogonal decomposition method abstract the geometric nonlinear response of a water filled simply supported circular cylindrical shell to harmonic excitation in the spectral neighbourhood of the fundamental natural frequency is investigated the response is investigated for a fixed excitation frequency by using the excitation amplitude as bifurcation parameter for a wide range of variation bifurcation diagrams of poincare maps obtained from direct time integration and calculation of the lyapunov exponents and lyapunov dimension have been used to study the system by increasing the excitation amplitude the response undergoes i a period doubling bifurcation ii subharmonic response iii quasi periodic response and iv chaotic behaviour with up to 16 positive lyapunov exponents hyperchaos the model is based on donnell s nonlinear shallow shell theory and the reference solution is obtained by the galerkin method the proper orthogonal decomposition pod method is used to extract proper orthogonal modes that describe the system behaviour from time series response data these time series have been obtained via the conventional galerkin approach using normal modes as a projection basis with an accurate model involving 16 degrees of freedom dofs validated in previous studies the pod method in conjunction with the galerkin approach permits to build a lower dimensional model as compared to those obtainable via the conventional galerkin approach periodic and quasi periodic response around the fundamental resonance for fixed excitation amplitude can be very successfully simulated with a 3 dof reduced order model however in the case of large variation of the excitation even a 5 dof reduced order model is not fully accurate results show that the pod methodology is not as robust as the galerkin method

generalization from educated teachers the authors study the generalization ability of a perceptron as a function of the structural similarity between teacher and student to this end they consider a teacher perceptron b designed to implement a certain number of random input output mappings and a student perceptron bold j trained with input output examples generated by the teacher if the pattern sets of teacher and student overlap different combinations of teacher and student learning rules result in different generalization probabilities generalization can be facilitated or can be hindered depending on the combination of learning rules and the storage levels of teacher and student analytical calculations of the generalization ability for several combinations of teacher and student learning rules are in very good agreement with numerical simulations

perfect learning and power law in learning from stochastic examples by ising perceptrons analysis under one step replica symmetry breaking ansatz we study learning from stochastic examples by perceptrons with ising weights in the framework of statistical mechanics under the one step replica symmetry breaking ansatz the behaviour of learning curves are classified according to some local property of the rules by which examples are drawn the conditions for the existence of perfect learning together with other behaviours of the learning curves are given precisely the results agree with those obtained by seung 1995 using a refined annealled approximation

on an asymptotically optimal adaptive classifier design criterion a new approach for estimating classification errors is presented in the model there are two types of classification error empirical and generalization error the first is the error observed over the training samples and the second is the discrepancy between the error probability and empirical error in this research the vapnik and chervonenkis dimension vcdim is used as a measure for classifier complexity based on this complexity measure an estimate for generalization error is developed an optimal classifier design criterion the generalized minimum empirical error criterion gmee is used the gmee criterion consists of two terms the empirical and the estimate of generalization error as an application the criterion is used to design the optimal neural network classifier a corollary to the gamma optimality of neural network based classifiers is proven thus the approach provides a theoretic foundation for the connectionist approach to optimal classifier design experimental results to validate this approach

a horizontal fluid conveying cantilever spatial coherent structures beam modes and jumps in stability diagram the proper orthogonal decomposition method is used to obtain a low dimensional model for the planar non linear dynamics of a horizontal fluid conveying cantilever undergoing a limit cycle oscillation the finite dimensional approximation of the non linear partial differential equation pde describing the oscillation is carried out by a galerkin projection scheme using both the cantilever beam modes and proper orthogonal modes coherent structures as projection bases which leads to a finite set of coupled ordinary differential equations the proper orthogonal modes are obtained semi analytically using the cantilever beam modes as a basis a systematic study is then carried out focusing on the jumps in the linear stability diagram of a horizontal fluid conveying cantilever vis a vis the order of the finite dimensional model obtained using either the beam modes or the proper orthogonal modes depending on the mass ratio of the cantilever while the order of the finite dimensional model using the beam mode basis increases steadily up to 10 the corresponding order when the proper orthogonal modes are used to span the solution of the pde remains unaltered only two

on line learning in the ising perceptron on line learning of both binary and continuous rules in an ising space is studied learning is achieved by using an artificial parameter a weight vector which is constrained to the surface of a hypersphere spherical constraint in the case of a binary rule the generalization error decays to zero super exponentially as exp c 2 where is the number of examples divided by n the size of the input vector and c 0 much faster learning is obtained in the case of continuous activation functions where the generalization error decays as exp e the number of steps required for perfect learning is estimated for both scenarios and compared with simulations

on learning perceptrons with binary weights we present an algorithm that pac learns any perceptron with binary weights and arbitrary threshold under the family of product distributions the sample complexity of this algorithm is of o n e 4 ln n and its running time increases only linearly with the number of training examples the algorithm does not try to find an hypothesis that agrees with all of the training examples rather it constructs a binary perceptron based on various probabilistic estimates obtained from the training examples we show that under the restricted case of the uniform distribution and zero threshold the algorithm reduces to the well known clipped hebb rule we calculate exactly the average generalization rate i e the learning curve of the algorithm under the uniform distribution in the limit of an infinite number of dimensions we find that the error rate decreases exponentially as a function of the number of training examples hence the average case analysis gives a sample complexity of o n ln 1 e a large improvement over the pac learning analysis the analytical expression of the learning curve is in excellent agreement with the extensive numerical simulations in addition the algorithm is very robust with respect to classification noise

chaotic dynamics of coherent structures abstract a variety of chaotic flows evolving in relatively high dimensional spaces are considered it is shown through the use of an optimal choice of basis functions which are consequence of the karhunen loeve procedure that an accurate description can be given in a relatively low dimensional space particular examples of this procedure which are presented are the ginzburg landau equation turbulent convection in an unbounded domain and turbulent convection in a bounded domain

a recursive parametrization of unitary matrices a simple recursive scheme for parametrization of n by n unitary matrices is presented the n by n matrix is expressed as a product containing the n 1 by n 1 matrix and a unitary matrix that contains the additional parameters needed to go from n 1 to n the procedure is repeated to obtain recursion formulas for n by n unitary matrices

learning from examples in fully connected committee machines the problem of learning from examples in two layer neural networks is studied within the framework of statistical mechanics a fully connected committee machine is trained to implement a task which is not linearly separable the generalization error as a function of the number of training examples per adjustable weight is calculated in the annealed approximation for both binary and continuous weights we find a first order transition with a discontinuous drop in the generalization error the transitions occur due to a specialization of the hidden units from a symmetric state to one with each hidden unit in the student network specialized on a corresponding unit in the teacher network the symmetric states of poor generalization remain metastable even for large training sets

a composite parameterization of unitary groups density matrices and subspaces unitary transformations and density matrices are central objects in quantum physics and various tasks require to introduce them in a parameterized form in this paper we present a parameterization of the unitary group of arbitrary dimension d which is constructed in a composite way we show explicitly how any element of can be composed of matrix exponential functions of generalized anti symmetric matrices and one dimensional projectors the specific form makes it considerably easy to identify and discard redundant parameters in several cases in this way redundancy free density matrices of arbitrary rank k can be formulated our construction can also be used to derive an orthonormal basis of any k dimensional subspaces of with the minimal number of parameters as an example it is shown that this feature leads to a significant reduction of parameters in the case of investigating distillability of quantum states via lower bounds of an entanglement measure the m concurrence

a model for nonpolynomial decrease in error rate with increasing sample size much theoretical evidence exists for an inverse proportionality between the error rate of a classifier and the number of samples used to train it cohn and tesauro 1992 have however discovered various problems which experimentally display an approximately exponential decrease in error rate we present evidence that the observed exponential decrease is caused by the finite nature of the problems studied a simple model classification problem is presented which demonstrates how the error rate approaches zero exponentially or faster when sufficiently many training samples are used

composite parameterization and haar measure for all unitary and special unitary groups we adopt the concept of the composite parameterization of the unitary group u d to the special unitary group su d furthermore we also consider the haar measure in terms of the introduced parameters we show that the well defined structure of the parameterization leads to a concise formula for the normalized haar measure on u d and su d with regard to possible applications of our results we consider the computation of high order integrals over unitary groups

theory and practice of vector quantizers trained on small training sets examines how the performance of a memoryless vector quantizer changes as a function of its training set size specifically the authors study how well the training set distortion predicts test distortion when the training set is a randomly drawn subset of blocks from the test or training image s using the vapnik chervonenkis vc dimension the authors derive formal bounds for the difference of test and training distortion of vector quantizer codebooks the authors then describe extensive empirical simulations that test these bounds for a variety of codebook sizes and vector dimensions and give practical suggestions for determining the training set size necessary to achieve good generalization from a codebook the authors conclude that by using training sets comprising only a small fraction of the available data one can produce results that are close to the results obtainable when all available data are used

towards more practical average bounds on supervised learning in this paper we describe a method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities the resulting theory provides a unified viewpoint of average case learning curves of concept learning and regression in realistic learning problems not necessarily within the bayesian framework the advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the vapnik chervonenkis vc theory and its alike and provides general insights into generalization besides the bounds on learning curves are directly related to the number of adjustable system weights although the theory is based on an approximation assumption and cannot apply to the worst case learning setting the precondition of the assumption is mild and the approximation itself is only a sufficient condition for the validity of the theory we illustrate the results with numerical simulations and apply the theory to examining the generalization ability of combination of neural networks

dynamics of learning and generalization in perceptrons with constraints abstract depending on the kind of constraints imposed on the weights of a perceptron learning can be a combinatorially hard problem as an example of this type i discuss a perception with binary weights comparing results obtained from replica theory dynamic mean field theory and simulated annealing contrary to the replica calculation dynamics yields information about the performance of a polynomial algorithm in a situation where the best solution cannot be found in polynomial time i also discuss improved learning algorithms and results for finite size perceptrons

on the practical applicability of vc dimension bounds this article addresses the question of whether some recent vapnik chervonenkis vc dimension based bounds on sample complexity can be regarded as a practical design tool specifically we are interested in bounds on the sample complexity for the problem of training a pattern classifier such that we can expect it to perform valid generalization early results using the vc dimension while being extremely powerful suffered from the fact that their sample complexity predictions were rather impractical more recent results have begun to improve the situation by attempting to take specific account of the precise algorithm used to train the classifier we perform a series of experiments based on a task involving the classification of sets of vowel formant frequencies the results of these experiments indicate that the more recent theories provide sample complexity predictions that are significantly more applicable in practice than those provided by earlier theories however we also find that the recent theories still have significant shortcomings

one step replica symmetry breaking solution for a perceptron learning with weight mismatch we investigate the properties of the one step replica symmetry breaking 1rsb solution for a perceptron learning from examples with weight mismatch where the entropy zero line crosses the almeida thouless at line of the rs solution for a small number of examples we find the optimal 1rsb solution which has the maximum free energy non negative entropy and satisfies the stability condition the at criterion for the 1rsb solution the transition from rs to 1rsb is continuous or discontinuous depending on whether the rs at line is above or below the zero entropy line however for a relatively large number of examples the 1rsb solution which maximizes the free energy becomes unstable and should be replaced by higher step rsb solutions we also obtain the at line for the 1rsb solution

temporal evolution of generalization during learning in linear networks we study generalization in a simple framework of feedforward linear networks with n inputs and n outputs trained from examples by gradient descent on the usual quadratic error function we derive analytical results on the behavior of the validation function corresponding to the lms error function calculated on a set of validation patterns we show that the behavior of the validation function depends critically on the initial conditions and on the characteristics of the noise under certain simple assumptions if the initial weights are sufficiently small the validation function has a unique minimum corresponding to an optimal stopping time for training for which simple bounds can be calculated there exists also situations where the validation function can have more complicated and somewhat unexpected behavior such as multiple local minima at most n of variable depth and long but finite plateau effects additional results and possible extensions are briefly discussed

on line learning in a discrete state space on line learning of a rule given by an n dimensional ising perceptron is considered for the case when the student is constrained to take values in a discrete state space of size for l 2 no on line algorithm can achieve a finite overlap with the teacher in the thermodynamic limit however if l is on the order of hebbian learning does achieve a finite overlap

the canonical coset decomposition of unitary matrices through householder transformations this paper reveals the relation between the canonical coset decomposition of unitary matrices and the corresponding decomposition via householder reflections these results can be used to parametrize unitary matrices via householder reflections

numerical solutions of second order implicit non linear ordinary differential equations abstract the existing literature usually assumes that second order ordinary differential equations can be put in first order form and this assumption is the starting point of most treatments of ordinary differential equations this paper examines numerical schemes for solving second order implicit non linear differential equations based on the literature review three specific methods have been selected here two of them are based on series expansions picard iteration using chebyshev series and incremental harmonic balance ihb in which the non linear differential equation is transformed into a set of algebraic ones that are solved iteratively the third method is based on a fourth order houbolt s scheme and an eighth order backward finite difference method fdm each method is presented first and then applied to specific examples it is shown 1 that the picard method is not valid for solving implicit equations containing large non linear inertial terms 2 that the ihb method yields accurate periodic solutions together with the frequency of oscillation and the dynamical stability of the system may be assessed very easily and 3 that both houbolt s and the eighth order scheme can be used to compute time histories of initial value problems if the time step is properly chosen finally it is also illustrated how the combination of ihb and fdm can be a powerful tool for the analysis of non linear vibration problems defined by implicit differential equations including also explicit ones since bifurcation diagrams of stable and unstable periodic solutions can be computed easily with ihb while periodic and non periodic stable oscillations may be obtained with fdm

generalization in a linear perceptron in the presence of noise the authors study the evolution of the generalization ability of a simple linear perceptron with n inputs which learns to imitate a teacher perceptron the system is trained on p alpha n example inputs drawn from some distribution and the generalization ability is measured by the average agreement with the teacher on test examples drawn from the same distribution the dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at alpha 1 when there are no errors static noise in the training examples if the examples are produced by an erroneous teacher overfitting is observed i e the generalization error starts to increase after a finite time of training it is shown that a weight decay of the same size as the variance of the noise errors on the teacher improves on the generalization and suppresses the overfitting the generalization error as a function of time is calculated numerically for various values of the parameters finally dynamic noise in the training is considered white noise on the input corresponds on average to a weight decay and can thus improve generalization whereas white noise on the weights or the output degrades generalization generalization is particularly sensitive to noise on the weights for alpha 1 where it makes the error constantly increase with time but this effect is also shown to be damped by a weight decay weight noise and output noise acts similarly above the transition at alpha 1

storage capacities of committee machines with overlapping and non overlapping receptive fields we present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number m of hidden units and spherical weights difficulties arise in the solution of this problem in the limit of large m in the case of overlapping receptive fields as the number of patterns increases both permutation symmetry and replica symmetry are broken which leads to the appearance of many order parameters and causes additional difficulty we observe that the relations among these order parameters yield a set of quantities which are small in the limit of large m making the asymptotic calculation tractable using the one step replica symmetry breaking scheme we compute the asymptotic value of the storage capacity per input unit in the limit of large m we find that the shift to the case of non overlapping receptive fields can be made easily we then find both values satisfy the bound of mitchison and durbin

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

vc dimension of univariate decision trees in this paper we give and prove the lower bounds of the vapnik chervonenkis vc dimension of the univariate decision tree hypothesis class the vc dimension of the univariate decision tree depends on the vc dimension values of its subtrees and the number of inputs via a search algorithm that calculates the vc dimension of univariate decision trees exhaustively we show that our vc dimension bounds are tight for simple trees to verify that the vc dimension bounds are useful we also use them to get vc generalization bounds for complexity control using structural risk minimization in decision trees i e pruning our simulation results show that structural risk minimization pruning using the vc dimension bounds finds trees that are more accurate as those pruned using cross validation

how to use expert advice we analyze algorithms that predict a binary value by combining the predictions of several prediction strategies called experts our analysis is for worst case situations i e we make no assumptions about the way the sequence of bits to be predicted is generated we measure the performance of the algorithm by the difference between the expected number of mistakes it makes on the bit sequence and the expected number of mistakes made by the best expert on this sequence where the expectation is taken with respect to the randomization in the predictins we show that the minimum achievable difference is on the order of the square root of the number of mistakes of the best expert and we give efficient algorithms that achieve this our upper and lower bounds have matching leading constants in most cases we then show how this leads to certain kinds of pattern recognition learning algorithms with performance bounds that improve on the best results currently know in this context we also compare our analysis to the case in which log loss is used instead of the expected number of mistakes

a deep connection between the vapnik chervonenkis entropy and the rademacher complexity in this paper we derive a deep connection between the vapnik chervonenkis vc entropy and the rademacher complexity for this purpose we first refine some previously known relationships between the two notions of complexity and then derive new results which allow computing an admissible range for the rademacher complexity given a value of the vc entropy and vice versa the approach adopted in this paper is new and relies on the careful analysis of the combinatorial nature of the problem the obtained results improve the state of the art on this research topic

stochastic dynamics of supervised learning the stochastic evolution of adiabatic slow backpropagation training of a neural network is discussed and a fokker planck equation for the post training distribution function in the network space is derived the distribution obtained differs from the one given by radons et al 1990 studying the character of the post training distribution the authors find that except under very special circumstances the distribution will be non gibbsian the validity of the present approach is tested on a simple backpropagation learning system in one dimension which can be solved analytically as well implications of the fokker planck approach for general situations are examined in the local linear approximation surprisingly they find that the post training distribution is isotropic close to its peak hence simpler than the corresponding gibbs distribution

neural network training without spurious minima physics department national university la plata casilla de correro 727 1900 la plata argentina received 2 january 1996 in training student perceptions recourse to information theory concepts allows one to select the best workinghypothesis and obtain an exact solution for the associated probability distribution we apply this trainingscheme to perceptions with binary weights and show that no phase transition ensues by recourse to ourapproach fast learning is guaranteed and trapping by spurious local minima is avoided s1063 651x 96 07505 8 pacs number s 87 10 1e 05 20 2y 02 70 2ci introduction

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

average case learning curves for radial basis function networks the application of statistical physics to the study of the learning curves of feedforward connectionist networks has to date been concerned mostly with perceptron like networks recent work has extended the theory to networks such as committee machines and parity machines and an important direction for current and future research is the extension of this body of theory to further connectionist networks in this article we use this formalism to investigate the learning curves of gaussian radial basis function networks rbfns having fixed basis functions these networks have also been called generalized linear regression models we address the problem of learning linear and nonlinear realizable and unrealizable target rules from noise free training examples using a stochastic training algorithm expressions for the generalization error defined as the expected error for a network with a given set of parameters are derived for general gaussian rbfns for which all parameters including centers and sp