dual processes in neural network models i neural dynamics versus dynamics of learning the dynamics of learning in an unsupervised formulation of the kohonen model is shown to be equivalent to the dynamics of local order parameters in an attractor neural network with short range hebbian interactions and long range anti hebbian interactions this duality can be used to study analytically the creation of topology conserving maps

asymptotics in directed exponential random graph models with an increasing bi degree sequence although asymptotic analyses of undirected network models based on degree sequences have started to appear in recent literature it remains an open problem to study the statistical properties of directed network models in this paper we provide for the first time a rigorous analysis of directed exponential random graph models using the in degrees and out degrees as sufficient statistics with binary and non binary weighted edges we establish the uniform consistency and the asymptotic normality of the maximum likelihood estimator when the number of parameters grows and only one realized observation of the graph is available one key technique in the proofs is to approximate the inverse of the fisher information matrix using a simple matrix with high accuracy along the way we also establish a geometrically fast rate of convergence for the newton iterative algorithm which is used to obtain the maximum likelihood estimate numerical studies confirm our theoretical findings

phase transitions in simple learning the authors investigate learning in the simplest type of a layered neural network the one layer perceptron the learning process is treated as a statistical dynamical problem quantities one is interested in include the relaxation time the learning time and the capacity and how they depend on noise and constraints on the weights the relaxation time is calculated as a function of the noise level and the number p of associations to be learned they consider three different cases for input patterns that are random and uncorrelated in the first where the connection weights are constrained to satisfy n 1 sigma i omega i2 s2 there is a critical value of p n separating regimes of perfect and imperfect learning at zero noise in contrast the second model unconstrained learning exhibits a different kind of transition at p n and noise plays no role in the third model where the constraint is imposed only on the thermal fluctuations there is a line of phase transitions terminating at p n and zero noise they have also considered learning with correlated input patterns the most important difference is the emergence of a second relaxation time which the authors interpret as the time it takes to learn a prototype of the patterns

homogeneous linear predictor models for contingency tables maximum likelihood ml inference for the class of homogeneous linear predictor hlp models for contingency tables is described hlp models constrain the expected table counts m through l m x where the link l is allowed to be a many to one nonlinear function generalized log linear association trend marginal cumulative probit and conditional marginal homogeneity models are given as specific examples ml fit results which include point estimates goodness of fit statistics and asymptotic based approximate distributions are described and compared for equivalent hlp models the results are valid for a wide variety of sampling plans including combinations of product multinomial and poisson sampling an important practical implication of this article is that the implementation of ml fitting and theory is straightforward and an attractive alternative to weighted least squares estimation for hlp models

supervised learning with restricted training sets a generating functional analysis we study the dynamics of supervised on line learning of realizable tasks in feed forward neural networks we focus on the regime where the number of examples used for training is proportional to the number of input channels n using generating functional techniques from spin glass theory we are able to average over the composition of the training set and transform the problem for n to an effective single pattern system described completely by the student autocovariance the student teacher overlap and the student response function with exact closed equations our method applies to arbitrary learning rules i e not necessarily of a gradient descent type the resulting exact macroscopic dynamical equations can be integrated without finite size effects up to any degree of accuracy but their main value is in providing an exact and simple starting point for analytical approximation schemes finally we show how in the region of absent anomalous response and using the hypothesis that as in detailed balance systems the short time part of the various operators can be transformed away one can describe the stationary state of the network succesfully by a set of coupled equations involving only four scalar order parameters

a statistical mechanical analysis of a bayesian inference scheme for an unrealizable rule within a bayesian framework we consider a system that learns from examples in particular using a statistical mechanical formalism we calculate the evidence and two performance measures namely the generalization error and the consistency measure for a linear perceptron trained and tested on a set of examples generated by a nonlinear teacher the teacher is said to be unrealizable because the student can never model it without error in fact our model allows us to interpolate between the known linear case and an unrealizable nonlinear case a comparison of the hyperparameters which maximize the evidence with those that optimize the performance measures reveals that when the student and teacher are fundamentally mismatched the evidence procedure is a misleading guide to optimizing the performance measures considered

langevin equations and surface growth departamento de fi sica universidad nacional de san luis chacabuco 917 5700 san luis argentina received 9 january 1997 the three approaches usually used to study surface growth are i master equation ii stochastic langevinequation and iii microscopic models all of them give the same scaling exponents recently vvedenskyet al phys rev e 48 852 1993 derived a stochastic langevin equation from a master equation of the birthand death type for the epitaxial growth demonstrating the equivalence of both approaches in this paper astochastic langevin equation is derived from a discrete model the results are the same as those obtained byvvedensky et al demonstrating that the three approaches are equivalent as a nontrivial example our proce dure is used to derive the kardar parisi zhang kpz equation from the ballistic deposition process thismodel with vacancies and overhangs is very dif cult to handle due to the algebraic complications that arisewhen the master equation approach is used s1063 651x 97 11905 5 pacs number s 05 40 1ji introduction

learning dynamics a replica approach a continuous time approximation for the learning process in single layer perceptrons is presented both statics and dynamics are treated by the replica method within the framework of the replica symmetric theory the adaline rule extended by a decay factor is investigated in detail especially for linearly separable problems it is shown that the poor generalization ability of this rule near alpha 1 the so called overfitting can be cured by an appropriate decay factor or by an appropriate training time

elementary local configurations microscopic rules and langevin equations for two models of molecular beam epitaxy stochastic langevin equations for the wolf villain wv and das sarma tamborenea dt models are derived using the alternative method recently developed by costanza 1997 phys rev e 55 6501 which avoids the complications arising in the calculation of the first moment of the transition rate required in the master equation approach the calculations are compared with those recently published by zhi feng huang et al 1996 phys rev e 54 5935 obtaining the same results as is expected the microscopic rules are derived from the set of 243 elementary local configurations needed for the description of these two models and after using simple summation rules these were reduced to 16 for the dt model and to 25 for the wv model the number of microscopic rules needed for the description of the present models are considerably larger than the previously solved models

finite size effects in bayesian model selection and generalization we show that in supervised learning from a supplied data set bayesian model selection based on the evidence does not optimize generalization performance even for a learnable linear problem this is demonstrated by examining the finite size effects in hyperparameter assignment from the evidence procedure and the resultant generalization performance our approach demonstrates the weakness of average case and asymptotic analyses using simulations we corroborate our analytic results and examine an alternative model selection criterion namely cross validation this numerical study shows that the cross validation hyperparameter estimates correlate more strongly than those of the evidence with optimal performance however we show that for a sufficiently large input dimension the evidence procedure could provide a reliable alternative to the more computationally expensive cross validation

structure of metastable states in the hopfield model an upper bound for the number of metastable states in the hopfield model is calculated as a function of the hamming fraction from an input pattern for all finite values of alpha the ratio of number of patterns to nodes the hamming fraction from the input pattern to the nearest metastable state is infinite when alpha 0 113 the bound also implies that there is a gap between a set of states close to the input pattern and another set centred around the hamming fraction 0 5 from it

finite size effects in learning and generalization in linear perceptrons most properties of learning and generalization in linear perceptrons can be derived from the average response function g we present a method for calculating g using only simple matrix identities and partial differential equations using this method we first rederive the known result for g in the thermodynamic limit of perceptrons of infinite size n which has previously been calculated using replica and diagrammatic methods we also show explicitly that the response function is self averaging in the thermodynamic limit extensions of our method to more general learning scenarios with anisotropic teacher space priors input distributions and weight decay terms are discussed finally finite size effects are considered by calculating the o 1 n correction to g we verify the result by computer simulations and discuss the consequences for generalization and learning dynamics in linear perceptrons of finite size

onset of super retrieval phase and enhancement of the storage capacity in neural networks of nonmonotonic neurons analogue neural networks of associative memory with continuous time dynamics are studied for nonmonotonic transfer functions using the method of self consistent signal to noise analysis the hebb learning rule with unbiased random patterns is assumed for the synaptic couplings a novel phenomenon is found to occur as a result of a phase transition concerning the property of the local field distribution in retrieval states of the newly found phase which the authors refer to as the super retrieval phase noise in the local field vanishes and the memory retrieval without errors ensures even for an extensive number of memory patterns stored under the local learning rule the storage capacity is obtained as a function of the parameter representing the degree of nonmonotonicity of the transfer functions with the result that an enhancement of the storage capacity can also occur

learning and retrieval in attractor neural networks above saturation the authors investigate neural networks in the range of parameters when the ground state energy is positive namely when a synaptic matrix which satisfies all the desired constraints cannot be found by the learning algorithm in particular they calculate the typical distribution functions of local stabilities obtained for a number of algorithms in this region these functions are used to investigate the retrieval properties as reflected by the size of the basins of attraction this is done analytically in sparsely connected networks and numerically in fully connected networks the main conclusion is that the retrieval behaviour of attractor neural networks can be improved by learning above saturation

inferring hidden states in langevin dynamics on large networks average case performance we present average performance results for dynamical inference problems in large networks where a set of nodes is hidden while the time trajectories of the others are observed examples of this scenario can occur in signal transduction and gene regulation networks we focus on the linear stochastic dynamics of continuous variables interacting via random gaussian couplings of generic symmetry we analyze the inference error given by the variance of the posterior distribution over hidden paths in the thermodynamic limit and as a function of the system parameters and the ratio ensuremath alpha between the number of hidden and observed nodes by applying kalman filter recursions we find that the posterior dynamics is governed by an effective drift that incorporates the effect of the observations we present two approaches for characterizing the posterior variance that allow us to tackle respectively equilibrium and nonequilibrium dynamics the first appeals to random matrix theory and reveals average spectral properties of the inference error and typical posterior relaxation times the second is based on dynamical functionals and yields the inference error as the solution of an algebraic equation

discrete stochastic evolution rules and continuum evolution equations abstract the continuum evolution equations are derived from updating rules for three classes of stochastic models the first class corresponds to models whose stochastic continuum equations are of the langevin type obtained after carrying out a local average known as coarse graining the second class consists of a hierarchy of continuum equations for the correlations of the dynamical variables obtained after making an average over realizations this average generates a hierarchy of deterministic partial differential equations except when the dynamical variables do not depend on the values of the neighboring dynamical variables in which case a hierarchy of ordinary differential equations is obtained the third class of evolution equations for the correlations of the dynamical variable constitutes another hierarchy after calculating an average over both realizations and all the sites of the lattice this double average generates a hierarchy of deterministic ordinary differential equations the second and third classes of equations are truncated using a mean field m n closure approximation in order to obtain a finite set of equations illustrative examples of every class are given

learning unrealizable tasks from minimum entropy queries in supervised learning learning from queries rather than from random examples can improve generalization performance significantly we study the performance of query learning for unrealizable tasks where the student cannot learn from the perfectly as a simple model scenario of this kind we consider a linear perceptron student learning a general nonlinear perceptron teacher two kinds of queries for maximum information gain i e minimum entropy are investigated minimum student space entropy msse queries which are appropriate if the teacher space is unknown and minimum teacher space entropy mtse queries which can be used if the teacher space is assumed to be known but a student of a simpler form has deliberately been chosen we find that for msse queries the structure of the student space determines the efficacy of query learning mtse queries on the other hand which we investigate for the extreme case of a binary perceptron teacher lead to a higher generalization error than random examples due to a lack of feedback about the progress of the student in the way queries are selected

asymptotic statistical theory of overtraining and cross validation a statistical theory for overtraining is proposed the analysis treats general realizable stochastic neural networks trained with kullback leibler divergence in the asymptotic case of a large number of training examples it is shown that the asymptotic gain in the generalization error is small if we perform early stopping even if we have access to the optimal stopping time based on the cross validation stopping we consider the ratio the examples should be divided into training and cross validation sets in order to obtain the optimum performance although cross validated early stopping is useless in the asymptotic region it surely decreases the generalization error in the nonasymptotic region our large scale simulations done on a cm5 are in good agreement with our analytical findings

optimised network for sparsely coded patterns the performance of attractor neural networks storing sparsely coded patterns has been shown to be greatly improved on shifting from the 1 1 representation of neural states to the 0 1 representation here the authors show that when this shift is considered as a special case of the transformation of the dynamical variables which depends on a continuous parameter the value of the parameter can be chosen to improve the performance of the network even further for every value of the bias in the patterns

goodness of fit tests for log linear models in sparse contingency tables abstract the asymptotic normality of the likelihood ratio goodness of fit statistic is demonstrated for testing the fit of log linear models with closed form maximum likelihood estimates in sparse contingency tables unlike the traditional chi squared theory the number of categories in the table increases as the sample size increases but not all of the expected frequencies are required to become large some results of a small monte carlo study are presented the traditional chi squared approximation is reasonably accurate for the pearson statistic for many sparse tables but cases are presented for which it fails the normal approximation can be much more accurate than the chi squared approximation for the likelihood ratio statistic but the bias of estimated moments is a potential problem for very sparse tables

content addressability and learning in neural networks the content addressability of patterns stored in ising spin neural network models with symmetric interactions is studied numerical results from simulations on the icl distributed array processor dap involving systems with up to 2048 neurons are presented behaviour consistent with finite size scaling characteristic of a first order phase transition is shown to be exhibited by the basins of attraction of the stored patterns both in the case of the hopfield model and for systems using a local iterative learning algorithm designed to optimise the basins of attraction estimates are obtained for the critical minimum overlaps which an input pattern must have with a stored pattern in order to successfully retrieve it

the effects of adding noise during backpropagation training on a generalization performance we study the effects of adding noise to the inputs outputs weight connections and weight changes of multilayer feedforward neural networks during backpropagation training we rigorously derive and analyze the objective functions that are minimized by the noise affected training processes we show that input noise and weight noise encourage the neural network output to be a smooth function of the input or its weights respectively in the weak noise limit noise added to the output of the neural networks only changes the objective function by a constant hence it cannot improve generalization input noise introduces penalty terms in the objective function that are related to but distinct from those found in the regularization approaches simulations have been performed on a regression and a classification problem to further substantiate our analysis input noise is found to be effective in improving the generalization performance for both problems however weight noise is found to be effective in improving the generalization performance only for the classification problem other forms of noise have practically no effect on generalization

categorization and effective perceptron learning in feed forward neural networks in the initial stages of back propagation learning weights in the first synaptic layer are small and hidden neurons operate in the linear regime so the input output map is then the same as that of a perceptron as the weights increase the hidden neurons begin to saturate each saturation pattern represents a particular category of input vectors for which the neural network nn locally behaves as a perceptron the effective perceptron weights for each category are functions of the synaptic weights of the nn we define an internal temperature as the inverse of the norm of the first synaptic matrix and an entropy function that measures the disorder of the hidden neuron layer the learning curves are recast in the language of a condensation process the entropy drops abruptly when the hidden neurons begin to saturate marking a clear transition from the global perceptron state to an ordered state characterized by different local effective perceptrons in different categories the problem of optimizing the effective perceptrons in different categories is frustrated

accurate directional inference for vector parameters we consider a vector valued parameter of interest in the presence of a nitedimensional nuisance parameter based on higher order asymptotic theory for likelihood inference we propose a directional test for the vector parameter of interest that is computed using one dimensional integration for discrete responses this extends the development of davison et al 2006 and several examples below concern testing hypotheses in contingency tables for continuous responses the work extends the directional test of cheah et al 1994 exponential family examples and simulations illustrate the high accuracy of the method which we compare with an adjusted likelihood ratio test of skovgaard 2001 in a high dimensional covariance selection example the approach works essentially

sequential importance sampling for multiway tables we describe an algorithm for the sequential sampling of entries in multiway contingency tables with given constraints the algorithm can be used for computations in exact conditional inference to justify the algorithm a theory relates sampling values at each step to properties of the associated toric ideal using computational commutative algebra in particular the property of interval cell counts at each step is related to exponents on lead indeterminates of a lexicographic grobner basis also the approximation of integer programming by linear programming for sampling is related to initial terms of a toric ideal we apply the algorithm to examples of contingency tables which appear in the social and medical sciences the numerical results demonstrate that the theory is applicable and that the algorithm performs well

maximum lilkelihood estimation in the beta model we study maximum likelihood estimation for the statistical model for undirected random graphs known as the beta model in which the degree sequences are minimal sufficient statistics we derive necessary and sufficient conditions based on the polytope of degree sequences for the existence of the maximum likelihood estimator mle of the model parameters we characterize in a combinatorial fashion sample points leading to a nonexistent mle and nonestimability of the probability parameters under a nonexistent mle we formulate conditions that guarantee that the mle exists with probability tending to one as the number of nodes increases

learning times of a perceptron that learns from examples we calculate the distribution of learning times of the optimal stability perceptron algorithm of krauth and mezard 1987 for the learning from noisy examples problem in particular we find that in the case of noiseless examples the average total number of learning steps scales with alpha 2 where alpha is the training set size although the number of examples that must effectively be learned tends to zero as alpha 1

finite size effects and optimal test set size in linear perceptrons fluctuations in the test error are important in the learning theory of finite dimensional systems as they represent how well the test error matches the average test error by explicitly finding the variance of the test error due to randomness present in both the data set and algorithm for a linear perceptron of dimension n we are able to address such questions as the optimal test set size where exact results were not tractable a good approximation is given to the variance we find that the optimal test set size possesses a phase transition between linear and 2 3 power law scaling in the system size n

algebraic algorithms for sampling from conditional distributions we construct markov chain algorithms for sampling from discrete exponential families conditional on a sufficient statistic examples include contingency tables logistic regression and spectral analysis of permutation data the algorithms involve computations in polynomial rings using grobner bases

from statistical mechanics to neural networks and back abstract the neural network models which have attracted so much attention in the physics community consist of a large assembly of two state elements highly interconnected by ferromagnetic and antiferromagnetic interactions methods of statistical mechanics have been applied to these models bringing new discoveries and surprises to neural network science at the same time neural network models have posed new problems and incentives to statistical mechanics one of the basic problems is that of learning which in the context of these models means a choice of the interactions for which the network dynamics leads to specific attractors the main emphasis in this paper is on the essential developments and results on this problem derived from methods of statistical mechanics

on the toric algebra of graphical models we formulate necessary and sufficient conditions for an arbitrary discrete probability distribution to factor according to an undirected graphical model or a log linear model or other more general exponential models for decomposable graphical models these conditions are equivalent to a set of conditional independence statements similar to the hammersley clifford theorem however we show that for nondecomposable graphical models they are not we also show that nondecomposable models can have nonrational maximum likelihood estimates these results are used to give several novel characterizations of decomposable graphical models

large systems of random linear equations with nonnegative solutions characterizing the solvable and the unsolvable phase large systems of linear equations are ubiquitous in science quite often e g when considering population dynamics or chemical networks the solutions must be nonnegative recently it has been shown that large systems of random linear equations exhibit a sharp transition from a phase where a nonnegative solution exists with probability one to one where typically no such solution may be found the critical line separating the two phases was determined by combining farkas lemma with the replica method here we show that the same methods remain viable to characterize the two phases away from criticality to this end we analytically determine the residual norm of the system in the unsolvable phase and a suitable measure of robustness of solutions in the solvable one our results are in very good agreement with numerical simulations

the failure distribution in percolation models of breakdown the probability of finding a largest defect cluster of size n in a percolation network is calculated analytically using a new distribution function scaling equation from this result and the stress voltage enhancement at the tip of the most critical defect in the network the probability of failure in percolation models of breakdown is calculated for defect fractions less than the percolation point this distribution is found to be of the form exponential of an exponential numerical simulations on the two dimensional random fuse network confirm the new distribution function and convincingly distinguish between it and the weibull 1951 form most often used in the fitting of breakdown data

learning with noise in a linear perceptron the learning of a set of p random patterns in a linear perceptron is studied in the limit of a large n of input units with noise on the weights inputs and output the problem is formulated in continuous time as a langevin equation and the first task is to evaluate the response or green function for the system white noise on the output is shown to correspond to spatially correlated weight noise acting only in a subspace of the weight space it is shown that the input noise acts as a simple weight decay with a size proportional to the load parameter alpha p n with no weight decay the relaxation time diverges at alpha 1 with a weight decay it becomes shorter and finite for alpha 1 but at the cost of a larger asymptotic learning error that is found analytically it is shown that a small weight decay decreases the effect of noise on the weights or outputs

multinomial poisson homogeneous models for contingency tables a unified approach to maximum likelihood inference for a broad new class of contingency table models is presented the model class comprises multinomial poisson homogeneous mph models which can be characterized by an independent sampling plan and a system of homogeneous constraints h m 0 where m is the vector of expected table counts maximum likelihood ml fitting and large sample inference for mph models are described the mph models are partitioned into well defined equivalence classes and explicit comparisons of the large sample behaviors of ml estimators of equivalent models are given the equivalence theory not only unifies a large collection of previously known results it also leads to useful generalizations and many new results the practical computational implication is that ml fit results for any particular mph model can be obtained directly from the ml fit results for any conveniently chosen equivalent model issues of hypothesis testability and parameter estimability are also addressed to illustrate an example based on statistics journal citation patterns is given for which the data can be used to test the hypothesis that a certain model holds but they cannot be used to estimate any of that model s parameters

complex dynamics in learning systems abstract to describe the dynamics of learning an input output relation from a set of examples the evolution of an appropriate choice of macroscopic dynamical variables have to be found recent progress in on line learning only addresses the often unrealistic case of an infinite training set for restricted training sets previous studies have so far been limited to asymptotic dynamics or simple learning rules using the cavity method and diagrammatic techniques we introduce a new framework to model batch learning of restricted sets of examples widely applicable to any learning cost function and fully taking into account the temporal correlations introduced by the recycling of the examples

accuracy of pseudo inverse covariance learning a random matrix theory analysis for many learning problems estimates of the inverse population covariance are required and often obtained by inverting the sample covariance matrix increasingly for modern scientific data sets the number of sample points is less than the number of features and so the sample covariance is not invertible in such circumstances the moore penrose pseudo inverse sample covariance matrix constructed from the eigenvectors corresponding to nonzero sample covariance eigenvalues is often used as an approximation to the inverse population covariance matrix the reconstruction error of the pseudo inverse sample covariance matrix in estimating the true inverse covariance can be quantified via the frobenius norm of the difference between the two the reconstruction error is dominated by the smallest nonzero sample covariance eigenvalues and diverges as the sample size becomes comparable to the number of features for high dimensional data we use random matrix theory techniques and results to study the reconstruction error for a wide class of population covariance matrices we also show how bagging and random subspace methods can result in a reduction in the reconstruction error and can be combined to improve the accuracy of classifiers that utilize the pseudo inverse sample covariance matrix we test our analysis on both simulated and benchmark data sets

generalization ability of perceptrons with continuous outputs in this paper we examine the influence of different input output relations on the generalization ability of a single layer perceptron the input output relations can be linear binary or sigmoid with this choice we take into account most of the cases which are of present interest the generalization problem will be realizable or unrealizable if the input output relations for teacher and student are identical or not we show that sometimes it can have a positive effect on the generalization ability if one learns with errors

prior induction in log linear models for general contingency table analysis log linear modelling plays an important role in many statistical applications particularly in the analysis of contingency table data with the advent of powerful new computational techniques such as reversible jump mcmc bayesian analyses of these models and in particular model selection and averaging have become feasible coupled with this is the desire to construct and use suitably flexible prior structures which allow efficient computation while facilitating prior elicitation the latter is greatly improved in the case where priors can be specified on interpretable parameters about which relevant experts can express their beliefs in this paper we show how the specification of a general multivariate normal prior on the log linear parameters induces a multivariate lognormal prior on the corresponding cell counts of a contingency table we derive the parameters of this distribution in an explicit practical form and state the corresponding mean and covariances of the cell counts we discuss the importance of these results in terms of applying both uninformative and informative priors to the model parameters and provide an illustration in the context of the analysis of a 2 3 contingency table 1 introduction the analysis of general k way contingency table data is of interest in a wide variety of areas of statistical application model selection is notoriously difficult in suchsituations since th e number of models rises doubly exponentially withdimension obviously exhaustive comparisons are impossible but various computational techniques have been proposed for obtaining posterior model probabilities for problems of this sort depending upon the parameters upon which the analyst wishes to express prior opinions with the introduction of powerful new computational techniques these forms of analysis have been greatly simplified and are becoming increasingly common in the applied literature in undertaking a bayesian analysis of contingency table data it is necessary to specify priors either for the cell counts which could alternatively be expressed in terms of the cell probabilities and total cell count or equivalently the log linear parameters madigan and york 1997 choose to choose to place hyper dirichlet priors dawid and lauritzen 1993 on the cell probabilities which has the advantage that priors of this form allow a factorization of the likelihood through the identification of cliques within the corresponding model graph giudici green and tarantola 1999 illustrate how this decomposition

on the comparison of multinomial and poisson log linear models summary we introduce a method for comparing multinomial and poisson log linear models which affords an explicit description of their equivalences and differences the method involves specifying the model in terms of constraint equations rather than the more common freedom equations the poisson and multinomial large sample distributions of log linear model parameter estimators are derived and compared within this constraint equation context reparameterizations are thereby avoided as a by product the method provides the practitioner with the adjustment that is necessary to make valid inferences about all multinomial log linear parameters when as a matter of convenience the poisson log linear model is fitted this implies that valid large sample inferences about the multinomial cell probabilities can be made directly by using the poisson log linear model to illustrate the utility of this approach several examples are considered

equilibrium properties of the linear perceptron we study the equilibrium properties of the real weights linear perceptron within the replica formalism framework focusing on the effects of the normalization of the weights on the learning and generalization capabilities of the network we also investigate the effects of static noise corrupting the training data and of dynamical noise acting on the weights during the training stage

using additive noise in back propagation training the possibility of improving the generalization capability of a neural network by introducing additive noise to the training samples is discussed the network considered is a feedforward layered neural network trained with the back propagation algorithm back propagation training is viewed as nonlinear least squares regression and the additive noise is interpreted as generating a kernel estimate of the probability density that describes the training vector distribution two specific application types are considered pattern classifier networks and estimation of a nonstochastic mapping from data corrupted by measurement errors it is not proved that the introduction of additive noise to the training vectors always improves network generalization however the analysis suggests mathematically justified rules for choosing the characteristics of noise if additive noise is used in training results of mathematical statistics are used to establish various asymptotic consistency results for the proposed method numerical simulations support the applicability of the training method

slowly evolving connectivity in recurrent neural networks i the extreme dilution regime we study extremely diluted spin models of neural networks in which the connectivity evolves in time although adiabatically slowly compared to the neurons according to stochastic equations which on average aim to reduce frustration the fast neurons and slow connectivity variables equilibrate separately but at different temperatures our model is exactly solvable in equilibrium we obtain phase diagrams upon making the condensed ansatz i e recall of one pattern these show that as the connectivity temperature is lowered the volume of the retrieval phase diverges and the fraction of mis aligned spins is reduced still one always retains a region in the retrieval phase where recall states other than the one corresponding to the condensed pattern are locally stable so the associative memory character of our model is preserved

modified pseudo inverse neural networks storing correlated patterns neural networks with symmetric couplings which have an intermediate form between the hebb learning rule and the pseudo inverse one storing strongly correlated patterns are studied signal to noise analysis is made and replica symmetric thermodynamic calculations are performed both approaches show that both in the hopfield model limit and in the pseudo inverse model limit the maximal capacity of the order of 2p ln 1 p 1 where p 1 is the average neural activity can be achieved by appropriate adjustment of the threshold term of the hamiltonian

game theory maximum entropy minimum discrepancy and robust bayesian decision theory we describe and develop a close relationship between two problems that have customarily been regarded as distinct that of maximizing entropy and that of minimizing worst case expected loss using a formulation grounded in the equilibrium theory of zero sum games between decision maker and nature these two problems are shown to be dual to each other the solution to each providing that to the other although topsoe described this connection for the shannon entropy over 20 years ago it does not appear to be widely known even in that important special case we here generalize this theory to apply to arbitrary decision problems and loss functions we indicate how an appropriate generalized definition of entropy can be associated with such a problem and we show that subject to certain regularity conditions the above mentioned duality continues to apply in this extended context this simultaneously provides a possible rationale for maximizing entropy and a tool for finding robust bayes acts we also describe the essential identity between the problem of maximizing entropy and that of minimizing a related discrepancy or divergence between distributions this leads to an extension to arbitrary discrepancies of a well known minimax theorem for the case of kullback leibler divergence the redundancy capacity theorem of information theory for the important case of families of distributions having certain mean values specified we develop simple sufficient conditions and methods for identifying the desired solutions we use this theory to introduce a new concept of generalized exponential family linked to the specific decision problem under consideration and we demonstrate that this shares many of the properties of standard exponential families finally we show that the existence of an equilibrium in our game can be rephrased in terms of a pythagorean property of the related divergence thus generalizing previously announced results for kullback leibler and bregman divergences

generalized maximum likelihood estimates for exponential families for a standard full exponential family on mathbb r d or its canonically convex subfamily the generalized maximum likelihood estimator is an extension of the mapping that assigns to the mean a in mathbb r d of a sample for which a maximizer vartheta of a corresponding likelihood function exists the member of the family parameterized by vartheta this extension assigns to each a in mathbb r d with the likelihood function bounded above a member of the closure of the family in variation distance its detailed description complete characterization of domain and range and additional results are presented not imposing any regularity assumptions in addition to basic convex analysis tools the authors prior results on convex cores of measures and closures of exponential families are used

domains of attraction of neural networks at finite temperature the authors consider neural networks trained by the symmetric edinburgh algorithm at finite temperature they show that while the introduction of the thermal noise affects the dynamics in a computable way it does not change the domains of attraction of stored patterns

maximum likelihood estimation in log linear models we study maximum likelihood estimation in log linear models under conditional poisson sampling schemes we derive necessary and sufficient conditions for existence of the maximum likelihood estimator mle of the model parameters and investigate estimability of the natural and mean value parameters under a nonexistent mle our conditions focus on the role of sampling zeros in the observed table we situate our results within the framework of extended exponential families and we exploit the geometric properties of log linear models we propose algorithms for extended maximum likelihood estimation that improve and correct the existing algorithms for log linear model analysis

quenched noise in surface growth abstract the analytical continuous equations for the tang and leschhorn phys rev a 45 1992 r8309 and the buldyrev et al phys rev a 45 1992 r8313 models are derived from the microscopic rules using a regularization procedure as was shown in a previous paper phys rev e 62 2000 6970 the continuous equation for the tang and leschhorn model is formally different to the kardar parisi zhang equation phys rev lett 56 1986 889 with quenched noise qkpz nevertheless after expanding the multiplicative noise it is shown that we recover the usual qkpz demonstrating analytically that both equations belong to the same universality class

on line learning with restricted training sets an exactly solvable case we solve the dynamics of on line hebbian learning in large perceptrons exactly for the regime where the size of the training set scales linearly with the number of inputs we consider both noiseless and noisy teachers our calculation cannot be extended to non hebbian rules but the solution provides a convenient and welcome benchmark with which to test more general and advanced theories for solving the dynamics of learning with restricted training sets

order parameter evolution in a feedforward neural network we consider layered neural networks in which the weights are trained with the pseudoinverse rule to store a set of random patterns using many body diagrammatic techniques the evolution in the network can be described by the overlap order parameter m and the noise parameter delta looping effects are shown to be significant in contrast to a previous conjecture order parameter pairs corresponding to various input conditions are found to collapse on a universal curve

learning and generalization in multineuron interacting feed forward neural networks we consider learning and generalization in the multi interacting feed forward network model recently proposed by h o carmesin 1994 with an a priori definition of the net architecture based on symmetries presented by the function to be learnt we define a generalized hebb rule extend the maximum stability learning algorithm to multi interactions and obtain training and generalization curves for rules where different orders of synapses are not correlated the results obtained for the simple perceptron concerning the hebb rule and through replica calculations in the space of couplings may be straightforwardly adapted to multi interactions through a simple renormalization of the total number of independent couplings analytical and numerical simulation results are compared and show excellent agreement

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

a system of coupled oscillators can have arbitrary prescribed attractors a system of coupled nonlinear oscillators is considered if the size n of this oscillator network is large enough then the model can have arbitrary prescribed attractors which can be defined by some finite dimensional flows one can effectively find a simple interaction between the oscillators which gives these prescribed attractors

on the ability of the optimal perceptron to generalise a linearly separable boolean function is derived from a set of examples by a perceptron with optimal stability the probability to reconstruct a pattern which is not learnt is calculated analytically using the replica method

generalization in a linear perceptron in the presence of noise the authors study the evolution of the generalization ability of a simple linear perceptron with n inputs which learns to imitate a teacher perceptron the system is trained on p alpha n example inputs drawn from some distribution and the generalization ability is measured by the average agreement with the teacher on test examples drawn from the same distribution the dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at alpha 1 when there are no errors static noise in the training examples if the examples are produced by an erroneous teacher overfitting is observed i e the generalization error starts to increase after a finite time of training it is shown that a weight decay of the same size as the variance of the noise errors on the teacher improves on the generalization and suppresses the overfitting the generalization error as a function of time is calculated numerically for various values of the parameters finally dynamic noise in the training is considered white noise on the input corresponds on average to a weight decay and can thus improve generalization whereas white noise on the weights or the output degrades generalization generalization is particularly sensitive to noise on the weights for alpha 1 where it makes the error constantly increase with time but this effect is also shown to be damped by a weight decay weight noise and output noise acts similarly above the transition at alpha 1

dynamics of batch training in a perceptron early stopping and weight decay are studied in a linear perceptron using a new simplified approach to the dynamics in the thermodynamical limit the approach is directly deduced from the gradient descent weight update it allows an exact description of the dynamics of the batch training process the results are compared with a recent study of early stopping and weight decay based on the equilibrium statistical mechanics approach it is shown that the equilibrium results for early stopping are good approximations and are exact for weight decay furthermore in the dynamical approach it is possible to determine the necessary number of training steps to fulfil certain termination conditions it can be shown that asymptotically i e if the number of examples is large only two batch steps are required to reach the optimal convergence if the learning rate is optimally chosen

weight decay induced phase transitions in multilayer neural networks we investigate layered neural networks with differentiable activation function and student vectors without normalization constraint by means of equilibrium statistical physics we consider the learning of perfectly realizable rules and find that the length of student vectors becomes infinite unless a proper weight decay term is added to the energy then the system undergoes a first order phase transition between states with very long student vectors and states where the lengths are comparable to those of the teacher vectors additionally in both configurations there is a phase transition between a specialized and an unspecialized phase an anti specialized phase with long student vectors exists in networks with a small number of hidden units

generalization performance of bayes optimal classification algorithm for learning a perceptron the generalization error of the bayes optimal classification algorithm when learning a perceptron from noise free random training examples is calculated exactly using methods of statistical mechanics it is shown that if an assumption of replica symmetry is made then in the thermodynamic limit the error of the bayes optimal algorithm is less than the error of a canonical stochastic learning algorithm by a factor approaching ensuremath surd 2 as the ratio of the number of training examples to perceptron weights grows in addition it is shown that approximations to the generalization error of the bayes optimal algorithm can be achieved by learning algorithms that use a two layer neutral net to learn a perceptron

learning and generalization in a linear perceptron stochastically trained with noisy data a linear perceptron is stochastically trained on a corrupted training data set this enables the effect of noise on the data to be studied the average properties of the network are calculated using the gardner method following seung et al a weight decay term is added to the training energy and the effect on generalization studied and compared with previously known results a prescription for setting the optimal weight decay parameter at finite temperature is presented the results also suggest an initial temperature for an annealing schedule

theoretical continuous equation derived from the microscopic dynamics for growing interfaces in quenched media we present an analytical continuous equation for the tang and leschhorn model phys rev a 45 r8309 1992 derived from their microscopic rules using a regularization procedure as well in this approach the nonlinear term nablah 2 arises naturally from the microscopic dynamics even if the continuous equation is not the kardar parisi zhang equation phys rev lett 56 889 1986 with quenched noise qkpz our equation is similar to a qkpz equation but with multiplicative quenched and thermal noise the numerical integration of our equation reproduces all the scaling exponents of the directed percolation depinning model

generating functional analysis of minority games with inner product strategy definitions we use generating functional methods to solve the so called inner product versions of the minority game mg with fake and or real market histories by generalizing the theory developed recently for look up table mgs with real histories the phase diagrams of the look up table and inner product mg versions are generally found to be identical with the exception of inner product mgs where histories are sampled linearly which are found to be structurally critical however we encounter interesting differences both in the theory where the role of the history frequency distribution in look up table mgs is taken over by the eigenvalue spectrum of a history covariance matrix in inner product mgs and in the static and dynamic phenomenology of the models our theoretical predictions are supported by numerical simulations

the langevin method in the statistical dynamics of learning the statistical dynamics of learning in the presence of noise has been treated recently by hertz krogh and thorbergson 1989 using a langevin model fluctuations introduced by this model into the space orthogonal to the space of patterns are shown to influence the dynamics of learning in a spurious way this is avoided in the new langevin model in which the noise is introduced directly into the cost function by using randomly modulated training patterns correspondingly a new kind of response function is defined unconstrained learning and learning under constrained thermal fluctuations are discussed as examples explicit expressions e g for the learning time are found to differ substantially from those given by hertz et al

on langevin updating in multilayer perceptrons the langevin updating rule in which noise is added to the weights during learning is presented and shown to improve learning on problems with initially ill conditioned hessians this is particularly important for multilayer perceptrons with many hidden layers that often have ill conditioned hessians in addition manhattan updating is shown to have a similar effect

temporal sequence storage capacity of time summating neural networks the maximum capacity for storage of temporal sequences in a time summating neural network is analysed within a statistical mechanical framework each neuron of such a network maintains an activity trace consisting of a decaying sum of all previous inputs to that neuron the maximum storage capacity is expressed as a function of the temporal correlations of the local fields of the neurons and evaluated using a perturbation expansion in powers of the decay rate

stochastic dynamics of supervised learning the stochastic evolution of adiabatic slow backpropagation training of a neural network is discussed and a fokker planck equation for the post training distribution function in the network space is derived the distribution obtained differs from the one given by radons et al 1990 studying the character of the post training distribution the authors find that except under very special circumstances the distribution will be non gibbsian the validity of the present approach is tested on a simple backpropagation learning system in one dimension which can be solved analytically as well implications of the fokker planck approach for general situations are examined in the local linear approximation surprisingly they find that the post training distribution is isotropic close to its peak hence simpler than the corresponding gibbs distribution

a theorem allowing to derive deterministic evolution equations from stochastic evolution equations abstract the deterministic evolution equations of classical as well as quantum mechanical models are derived from a set of stochastic evolution equations after taking an average over realizations using a theorem examples are given that show that deterministic quantum mechanical evolution equations obtained initially by r p feynman and subsequently studied by boghosian and taylor iv b m boghosian w taylor iv phys rev e 57 1998 54 see also arxiv quant ph 9904035 and meyer d a meyer phys rev e 55 1997 5261 among others are derived from a set of stochastic evolution equations in addition a deterministic classical evolution equation for the diffusion of monomers similar to the second fick law is also obtained

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state

accurate directional inference for vector parameters in linear exponential families we consider inference on a vector valued parameter of interest in a linear exponential family in the presence of a finite dimensional nuisance parameter based on higher order asymptotic theory for likelihood we propose a directional test whose p value is computed using one dimensional integration the work simplifies and develops earlier research on directional tests for continuous models and on higher order inference for discrete models and the examples include contingency tables and logistic regression examples and simulations illustrate the high accuracy of the method which we compare with the usual likelihood ratio test and with an adjusted version due to skovgaard in high dimensional settings such as covariance selection the approach works essentially perfectly whereas its competitors can fail catastrophically

three unfinished works on the optimal storage capacity of networks the optimal storage properties of three different neural network models are studied for two of these models the architecture of the network is a perceptron with or j interactions whereas for the third model the output can be an arbitrary function of the inputs analytic bounds and numerical estimates of the optimal capacities and of the minimal fraction of errors are obtained for the first two models the third model can be solved exactly and the exact solution is compared to the bounds and to the results of numerical simulations used for the two other models