measuring the strangeness of strange attractors we study the correlation exponent v introduced recently as a characteristic measure of strange attractors which allows one to distinguish between deterministic chaos and random noise the exponent v is closely related to the fractal dimension and the information dimension but its computation is considerably easier its usefulness in characterizing experimental data which stem from very high dimensional systems is stressed algorithms for extracting v from the time series of a single variable are proposed the relations between the various measures of strange attractors and between them and the lyapunov exponents are discussed it is shown that the conjecture of kaplan and yorke for the dimension gives an upper bound for v various examples of finite and infinite dimensional systems are treated both numerically and analytically

using sampling and simplex derivatives in pattern search methods in this paper we introduce ways of making a pattern search more efficient by reusing previous evaluations of the objective function based on the computation of simplex derivatives e g simplex gradients at each iteration one can attempt to compute an accurate simplex gradient by identifying a sampling set of previously evaluated points with good geometrical properties this can be done using only past successful iterates or by considering all past function evaluations the simplex gradient can then be used to reorder the evaluations of the objective function associated with the directions used in the poll step or to update the mesh size parameter according to a sufficient decrease criterion neither of which requires new function evaluations we present these procedures in detail and apply them to a set of problems from the cuter collection numerical results show that these procedures can enhance significantly the practical performance of pattern search methods

bifurcation diagrams in estimated parameter space using a pruned extreme learning machine we propose an algorithm to estimate parameter spaces by using a pruned extreme learning machine but without using principal component analysis and we plot bifurcation diagrams in the estimated parameter spaces to visualize changes in system patterns the estimation of parameter spaces can predict changes in the behavior of a system when its parameters are changed it can be very helpful to adjust the optimal parameters of unknown systems in this paper we estimate the dimension of parameter spaces using the singular values of trained synaptic weights and the parameter spaces based on the method proposed by bagarinao et al using a pruned extreme learning machine we motivate this use of a pruned extreme learning machine through numerical experiments with the estimation of the dimension of parameter spaces for various systems and we show that the proposed method can successfully plot bifurcation diagrams in the estimated parameter spaces in addition we show the results of a bifurcation diagram in the estimated parameter space for maps that are nonlinear in parameters since bagarinao et al limited their method to linear in parameters maps in particular we estimate the parameter spaces for a mathematical model of induction motor drives and for a model of vegetation biomass in ecosystems which are both nonlinear in parameter maps corresponding to real world systems

recent progress in unconstrained nonlinear optimization without derivatives we present an introduction to a new class of derivative free methods for unconstrained optimization we start by discussing the motivation for such methods and why they are in high demand by practitioners we then review the past developments in this field before introducing the features that characterize the newer algorithms in the context of a trust region framework we focus on techniques that ensure a suitable geometric quality of the considered models we then outline the class of algorithms based on these techniques as well as their respective merits we finally conclude the paper with a discussion of open questions and perspectives

reducing the number of function evaluations in mesh adaptive direct search algorithms the mesh adaptive direct search mads class of algorithms is designed for nonsmooth optimization where the objective function and constraints are typically computed by launching a time consuming computer simulation each iteration of a mads algorithm attempts to improve the current best known solution by launching the simulation at a finite number of trial points common implementations of mads generate 2n trial points at each iteration where n is the number of variables in the optimization problem the objective of the present work is to dynamically reduce that number we present an algorithmic framework that reduces the number of simulations to exactly n 1 without impacting the theoretical guarantees from the convergence analysis numerical experiments are conducted for several different contexts the results suggest that these strategies allow the new algorithms to reach a better solution with fewer function evaluations

orthomads a deterministic mads instance with orthogonal directions the purpose of this paper is to introduce a new way of choosing directions for the mesh adaptive direct search mads class of algorithms the advantages of this new orthomads instantiation of mads are that the polling directions are chosen deterministically ensuring that the results of a given run are repeatable and that they are orthogonal to each other which yields convex cones of missed directions at each iteration that are minimal in a reasonable measure convergence results for orthomads follow directly from those already published for mads and they hold deterministically rather than with probability one as is the case for ltmads the first mads instance the initial numerical results are quite good for both smooth and nonsmooth and constrained and unconstrained problems considered here

on the convergence of the multidirectional search algorithm this paper presents the convergence analysis for the multidirectional search algorithm a direct search method for unconstrained minimization the analysis follows the classic lines of proofs of convergence for gradient related methods the novelty of the argument lies in the fact that explicit calculation of the gradient is unnecessary although it is assumed that the function is continuously differentiable over some subset of the domain the proof can be extended to treat most nonsmooth cases of interest the argument breaks down only at points where the derivative exists but is not continuous finally it is shown how a general convergence theory can be developed for an entire class of direct search methods which includes such methods as the factorial design algorithm and the pattern search algorithm that share a key feature of the multidirectional search algorithm

generalization properties of multilayered neural networks generalization properties of multilayered neural networks with binary couplings are studied in the high temperature limit the transition to the perfect generalization phase is evaluated for systems with an arbitrary number of layers it is found that the thermodynamic transition occurs for a number of examples lower than for the perceptron but the opposite occurs for the transition in which the poor generalization solution disappears the generalization error is also decomposed according to the contributions coming from different numbers of hidden neurons that have a wrong sign in the internal field this allows the authors to describe the generalization behaviour of the hidden neurons

time series based bifurcation diagram reconstruction abstract we consider the problem of reconstructing bifurcation diagrams bds of maps using time series this study goes along the same line of ideas presented by tokunaga et al physica d 79 1994 348 and tokuda et al physica d 95 1996 380 the aim is to reconstruct the bd of a dynamical system without the knowledge of its functional form and its dependence on the parameters instead time series at different parameter values assumed to be available are used a three layer fully connected neural network is employed in the approximation of the map the task of the network is to learn the dynamics of the system as function of the parameters from the available time series we determine a class of maps for which one can always find a linear subspace in the weight space of the network where the network s bifurcation structure is qualitatively the same as the bifurcation structure of the map we discuss a scheme in locating this subspace using the time series we further discuss how to recognize time series generated by this class of maps finally we propose an algorithm in reconstructing the bds of this class of maps using predictor functions obtained by neural network this algorithm is flexible so that other classes of predictors apart from neural networks can be used in the reconstruction

inferring hidden states in langevin dynamics on large networks average case performance we present average performance results for dynamical inference problems in large networks where a set of nodes is hidden while the time trajectories of the others are observed examples of this scenario can occur in signal transduction and gene regulation networks we focus on the linear stochastic dynamics of continuous variables interacting via random gaussian couplings of generic symmetry we analyze the inference error given by the variance of the posterior distribution over hidden paths in the thermodynamic limit and as a function of the system parameters and the ratio ensuremath alpha between the number of hidden and observed nodes by applying kalman filter recursions we find that the posterior dynamics is governed by an effective drift that incorporates the effect of the observations we present two approaches for characterizing the posterior variance that allow us to tackle respectively equilibrium and nonequilibrium dynamics the first appeals to random matrix theory and reveals average spectral properties of the inference error and typical posterior relaxation times the second is based on dynamical functionals and yields the inference error as the solution of an algebraic equation

bifurcation phenomena near homoclinic systems a two parameter analysis the bifurcations of periodic orbits in a class of autonomous three variable nonlinear differential equation systems possessing a homoclinic orbit associated with a saddle focus with eigenvalues i where 1 sil nikov s condition are studied in a two parameter space the perturbed homoclinic systems undergo a countable set of tangent bifurcations followed by period doubling bifurcations leading to periodic orbits which may be attractors if 1 2 the accumulation rate of the critical parameter values at the homoclinic system is exp 2 a global mechanism for the onset of homoclinicity in strongly contractive flows is analyzed cusp bifurcations with bistability and hysteresis phenomena exist locally near the onset of homoclinicity a countable set of these cusp bifurcations with scaling properties related to the eigenvalues i of the stationary state are shown to occur in infinitely contractive flows in the two parameter space the periodic orbit attractor domain exhibits a spiral structure globally around the set of homoclinic systems in which all the different periodic orbits are continuously connected

efficient surrogate modelling of nonlinear aerodynamics in aerostructural coupling schemes this paper considers the substitution of the computational fluid dynamics solver within an aerostructural coupling scheme by an efficient reduced order surrogate model to realize high fidelity nonlinear aeroelastic analyses within a fraction of the time compared to the corresponding coupled computational fluid dynamics analysis the presented surrogate model approach is a combination of parameter reduction via proper orthogonal decomposition and system identification methods designed to cover nonlinear aerodynamic effects such as viscosity as well as transient effects the investigated test case is the thick nlr7301 airfoil that shows significant nonlinear aerodynamic behavior along with nonnegligible viscous effects the model is identified from a set of transient forced motion computational fluid dynamics analyses after identification the model is used to predict the discrete surface force distribution of the nlr7301 airfoil in static as well as transient coupled analyses it is shown that the static

perturbative treatments and learning techniques the replica trick rt of statistical mechanics has become a very useful tool for the investigation of complex systems in general and in particular for studying learning and generalization processes in neural networks nn 1 2 the trick overcomes the difficulty of performing an ensemble average of the logarithm of the partition function z by that of averaging z over n replicas of the original network with n a very large integer for feedforward single layered nn 3 4 the rt ansatz has been proved to be a valuable tool for the learning of a rule on the basis of a suitable set of examples in particular the full quenched theory has been carefully studied within the framework of a replica symmetry approximation 5 however as the temperature drops symmetry breakings may invalidate this approximation so that an approach to the full quenched theory that incorporates disorder effects due to improper examples may be of some interest in the present effort we wish to study in the spirit of a second order approximation noise effects in the training set unavoidable in any realistic setting the noise will here be the result of letting just part of the examples to be produced by the perceptron teacher pt the rest are to be randomly selected bad examples two types of situations are to be confronted learnable and unlearnable rules 6 for the former there is at least a vector in the concomitant weight space that can learn the rule in an exact fashion the latter arises mostly in cases of architectural mismatch in such a situation the training error can never vanish the question to be answered is can a perceptron trained under these circumstances correctly respond to queries posed by a pt in other words is the rule underlying the good examples a learnable one we will show here that these questions can be adequately dealt with the paper is organized as follows section ii is devoted to a brief recapitulation of basic concepts concerning the rt while sec iii deals with the thermodynamics of the situation that interests us here a second order high temperature approximation is derived boolean perceptrons with ising weights on the learning curves are the subject of sec iv finally some conclusions are drawn in sec v ii the replica method

information and entropy in strange attractors a technique for analyzing time series data from experiments is presented that provides estimates of four basic characteristics of a system 1 the measure theoretic entropy 2 the accuracy of the measurements 3 the number of measurements necessary to specify a system state and 4 the best delay time t to use in order to construct phase space portraits by the method of delays these characteristics are obtained by separating the entropy of measurements into a part due to noise and parts due to deterministic effects for the technique to work the noise associated with each measurement must be independent of the noise associated with all other measurements an algorithm for implementing the analysis is presented with three examples

on the convergence of pattern search algorithms we introduce an abstract definition of pattern search methods for solving nonlinear unconstrained optimization problems our definition unifies an important collection of optimization methods that neither compute nor explicitly approximate derivatives we exploit our characterization of pattern search methods to establish a global convergence theory that does not enforce a notion of sufficient decrease our analysis is possible because the iterates of a pattern search method lie on a scaled translated integer lattice this allows us to relax the classical requirements on the acceptance of the step at the expense of stronger conditions on the form of the step and still guarantee global convergence

parallel space decomposition of the mesh adaptive direct search algorithm this paper describes a parallel space decomposition psd technique for the mesh adaptive direct search mads algorithm mads extends a generalized pattern search for constrained nonsmooth optimization problems the objective of the present work is to obtain good solutions to larger problems than the ones typically solved by mads the new method psd mads is an asynchronous parallel algorithm in which the processes solve problems over subsets of variables the convergence analysis based on the clarke calculus is essentially the same as for the mads algorithm a practical implementation is described and some numerical results on problems with up to 500 variables illustrate the advantages and limitations of psd mads

vapnik chervonenkis bounds for generalization the authors review the vapnik and chervonenkis theorem as applied to the problem of generalization by combining some of the technical modifications proposed in the literature they derive tighter bounds and a new version of the theorem bounding the accuracy in the estimation of generalization probabilities from finite samples a critical discussion and comparison with the results from statistical mechanics is given

four types of learning curves if machines are learning to make decisions given a number of examples the generalization error e t is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples the generalization error decreases as t increases and the curve e t is called a learning curve the present paper uses the bayesian approach to show that given the annealed approximation learning curves can be classified into four asymptotic types if the machine is deterministic with noiseless teacher signals then 1 e at 1 when the correct machine parameter is unique and 2 e at 2 when the set of the correct parameters has a finite measure if the teacher signals are noisy then 3 e at 1 2 for a deterministic machine and 4 e c at 1 for a stochastic machine

a numerical study on learning curves in stochastic multilayer feedforward networks the universal asymptotic scaling laws proposed by amari et al are studied in large scale simulations using a cm5 small stochastic multilayer feedforward networks trained with backpropagation are investigated in the range of a large number of training patterns t the asymptotic generalization error scales as 1 t as predicted for a medium range t a faster 1 t2 scaling is observed this effect is explained by using higher order corrections of the likelihood expansion it is shown for small t that the scaling law changes drastically when the network undergoes a transition from strong overfitting to effective learning

generalized one parameter bifurcation diagram reconstruction using time series abstract in this paper we look into the use of time series in reconstructing bifurcation diagrams of systems that cannot be modeled directly from first principles in this reconstruction problem time series at different unknown parameter values are available these time series are then used to obtain a suitable family of nonlinear predictor functions which exhibits qualitatively similar bifurcations as the original system to solve the reconstruction problem we propose a generalized one parameter reconstruction algorithm based on principal curves we illustrate its feasibility numerically by reconstructing the bifurcation diagrams of the fitzhugh nagumo equations and the lorenz equations with parameter values mostly in the fixed point regime using the proposed algorithm we obtain a reconstruction which preserved the important features of the bifurcation diagrams of the original systems such as the hopf bifurcations of the two systems the limit cycle of the fitzhugh nagumo equations and the symmetry of the fixed points of the lorenz equations among others moreover the reconstructed systems also exhibit the same sequence of behavior as the original systems with respect to the changes in the bifurcation parameters

linear readout of object manifolds objects are represented in sensory systems by continuous manifolds due to sensitivity of neuronal responses to changes in physical features such as location orientation and intensity what makes certain sensory representations better suited for invariant decoding of objects by downstream networks we present a theory that characterizes the ability of a linear readout network the perceptron to classify objects from variable neural responses we show how the readout perceptron capacity depends on the dimensionality size and shape of the object manifolds in its input neural representation

learning of higher order perceptrons with tunable complexities we study learning from examples by higher order perceptrons which realize polynomially separable rules the model complexities of the networks are made tunable by varying the relative orders of different monomial terms we analyse the learning curves of higher order perceptrons when the gibbs algorithm is used for training it is found that learning occurs in a stepwise manner this is because the number of examples needed to constrain the corresponding phase space component scales differently

application of multi input volterra theory to nonlinear multi degree of freedom aerodynamic systems this paper presents a reduced order modeling approach for nonlinear multi degree of freedom aerodynamic systems using multi input volterra theory the method is applied to a two dimensional 2 degree of freedom transonic airfoil undergoing simultaneous forced pitch and heave harmonic oscillations the so called volterra cross kernels are identified and shown to successfully model the aerodynamic nonlinearities associated with the simultaneous pitch and heave motions the improvements in accuracy over previous approaches that effectively ignored the cross kernels by using superposition are demonstrated

detecting switch dynamics in chaotic time waveform using a parametrized family of nonlinear predictors abstract an algorithm is presented for detecting switch dynamics in chaotic time waveform by the switch dynamics we mean that the chaotic time waveform is measured from a dynamical system whose bifurcation parameters are occasionally switched among a set of slightly different parameter values first the switched chaotic time waveform is divided into windows of short term time waveforms from the set of windowed time waveforms qualitatively similar parametrized family of nonlinear predictors is constructed qualitatively similar parametrized family means that the family of nonlinear predictors exhibits qualitatively similar bifurcation phenomena as the original by characterizing the windows of short term chaotic time waveforms in terms of the qualitative parameters of nonlinear predictors switch dynamics of their associated bifurcation parameters are detected for the lorenz equations the rossler equations and the mackey glass equations efficiency of the algorithm is demonstrated in the experiment chaotic time waveforms contaminated with observational noise is considered

finite size effects and bounds for perceptron models ablncl in lhis paper we consider two main aspsln of the binary perccptron problem the maximal capacity when random paltem are stored model a and its generalhation ability model b we have extended previous numerical estimates of critical capacilies and studied thermal properties of systems of small sizes to lest recent replica predictions we have also considered some simpler versions of thest models the discrete spherical vemions can be salved exactly using gardner s replica calculation for the spherical model and are shown to give a rigorous upper bound and lower bound on the capacities of models a and b respectively iby vemions of models a and b are soivcd in detail and provide information which is useful for interpreting the bnile size eeeciects present in the numerical studies of models a and b

on the generalization ability of diluted perceptrons a linearly separable boolean function is teamed by a diluted perceptron with optimal stability a different level of dilution is allowed for teacher and student perceptron the learning algorithms used were the optimal annealed dilution and hebbian dilution the generalization ability i e the probability to recognize a pattern which has not been learned before is calculated in replica symmetry

recognizing chaotic time waveforms in terms of a parametrized family of nonlinear predictors abstract consider a chaotic dynamical system which exhibits a variety of chaotic time waveforms with a change in the bifurcation parameters this paper presents an algorithm for estimating the underlying bifurcation parameters of the chaotic time waveforms in experimental situation in which no a priori analytical knowledge of the dynamical system is available first we construct qualitatively similar parametrized family of nonlinear predictors only from several sets of chaotic time waveforms qualitatively similar parametrized family means that the family of nonlinear predictors exhibits qualitatively similar bifurcation phenomena as the original chaotic time waveforms are then characterized in terms of the qualitatively similar bifurcation parameters of the nonlinear predictors we call the characterization of chaotic time waveforms in terms of the underlying bifurcation parameters chaotic time waveform recognition several numerical experiments using the rossler equations show the efficiency of the algorithm the effect of observational noise included in chaotic time waveforms is also considered

geometry of interpolation sets in derivative free optimization we consider derivative free methods based on sampling approaches for nonlinear optimization problems where derivatives of the objective function are not available and cannot be directly approximated we show how the bounds on the error between an interpolating polynomial and the true function can be used in the convergence theory of derivative free sampling methods these bounds involve a constant that reflects the quality of the interpolation set the main task of such a derivative free algorithm is to maintain an interpolation sampling set so that this constant remains small and at least uniformly bounded this constant is often described through the basis of lagrange polynomials associated with the interpolation set we provide an alternative more intuitive definition for this concept and show how this constant is related to the condition number of a certain matrix this relation enables us to provide a range of algorithms whilst maintaining the interpolation set so that this condition number or the geometry constant remain uniformly bounded we also derive bounds on the error between the model and the function and between their derivatives directly in terms of this condition number and of this geometry constant

control of nonlinear dynamical systems using neural networks ii observability identification and control for pt i see ibid vol 4 1993 this paper considers the problems of regulation and tracking of a dynamical system when the state variables of the dynamical system are not accessible the existence of the nonlinear maps describing the identifier and controller are first established and the implications for neural network realizations are described simulation results are included to complement the theoretical discussions

anti pairing in learning of a neural network with redundant hidden units we study the statistical mechanics of learning from examples between the two layered committee machines with different numbers of hidden units using the replica theory the number m of hidden units of the student network is larger than the number mt of those of the target network called the teacher we choose the networks to have binary synaptic weights 1 which makes it possible to compare the calculation with the monte carlo simulation we propose an effective teacher as a virtual target network which has the same m hidden units as the student and gives identical outputs with those of the original teacher this is a way of making a conjecture for a ground state of a thermodynamic system given by the weights of the effective teacher in our study we suppose that the weights on mt hidden units of the effective teacher are the same as those of the original teacher while those on m mt redundant hidden units are composed of anti pairs 1 1 with probability 1 p in the limit p 0 for p 0 exact there are no terms related to the effective teacher in the calculation for the contributions of anti pairs to outputs are exactly cancelled in the limit p 0 however we find that the learnt weights of the student are actually equivalent to those of the suggested effective teacher which is not possible from the calculation for p 0 p plays the role of a symmetry breaking parameter for anti pairing ordering which is analogous to the magnetic field for the ising model a first order phase transition is found to be signalled by breaking of symmetry in permuting hidden units above a critical number of examples the student is shown to learn perfectly the effective teacher anti pairing can be measured by a set of order parameters zero in the permutation symmetric phase and nonzero in the permutation symmetry breaking phase results from the monte carlo simulation are shown to be in good agreement with those from the replica calculation

inexact restoration method for nonlinear optimization without derivatives a derivative free optimization method is proposed for solving a general nonlinear programming problem it is assumed that the derivatives of the objective function and the constraints are not available the new method is based on the inexact restoration scheme where each iteration is decomposed in two phases in the first one the violation of the feasibility is reduced in the second one the objective function is minimized onto a linearization of the nonlinear constraints at both phases polynomial interpolation models are used in order to approximate the objective function and the constraints at the first phase a derivative free solver for box constrained optimization can be used for the second phase we propose a new method ad hoc based on trust region strategy that uses the projection of the simplex gradient on the tangent space under suitable assumptions the algorithm is well defined and convergence results are proved a numerical implementation is described and numerical experiments are presented to validate the theoretical results

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state

measurement of the lyapunov spectrum from a chaotic time series the exponential divergence or convergence of nearby trajectories lyapunov exponents is conceptually the most basic indicator of deterministic chaos we propose a new method to determine the spectrum of several lyapunov exponents including positive zero and even negative ones from the observed time series of a single variable we have applied the method to various known model systems and also to the rayleigh b enard experiment and have elucidated the dependence of the lyapunov exponents on the rayleigh number

tight bounds on transition to perfect generalization in perceptrons sudden transition to perfect generalization in binary perceptrons is investigated building on recent theoretical works of gardner and derrida 1989 and baum and lyuu 1991 we show the followi

optimization by direct search new perspectives on some classical and modern methods direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis nonetheless users remained loyal to these methods most of which were easy to program some of which were reliable in the past fifteen years these methods have seen a revival due in part to the appearance of mathematical analysis as well as to interest in parallel and distributed com puting this review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results the underlying principles allow general ization to handle bound constraints and linear constraints we also discuss extensions to problems with nonlinear constraints

direct search methods on parallel machines direct search methods are methods designed to solve unconstrained minimization problems of the form mathop min limits x in mathbb r n f x where f n these methods are distinguished by the fact that they neither use nor require explicit derivative information the search for a local minimizer is driven solely by function information popular methods in this class include the factorial design algorithm of box 1 the pattern search algorithm of hooke and jeeves 4 and the simplex method of neider and mead 5

how tight are the vapnik chervonenkis bounds we describe a series of numerical experiments that measure the average generalization capability of neural networks trained on a variety of simple functions these experiments are designed to test the relationship between average generalization performance and the worst case bounds obtained from formal learning theory using the vapnik chervonenkis vc dimension blumer et al 1989 haussler et al 1990 recent statistical learning theories tishby et al 1989 schwartz et al 1990 suggest that surpassing these bounds might be possible if the spectrum of possible generalizations has a gap near perfect performance we indeed find that in some cases the average generalization is significantly better than the vc bound the approach to perfect performance is exponential in the number of examples m rather than the 1 m result of the bound however in these cases we have not found evidence of the gap predicted by the above statistical theories in other cases we do find the 1 m behavior of the vc bound

generalization from educated teachers the authors study the generalization ability of a perceptron as a function of the structural similarity between teacher and student to this end they consider a teacher perceptron b designed to implement a certain number of random input output mappings and a student perceptron bold j trained with input output examples generated by the teacher if the pattern sets of teacher and student overlap different combinations of teacher and student learning rules result in different generalization probabilities generalization can be facilitated or can be hindered depending on the combination of learning rules and the storage levels of teacher and student analytical calculations of the generalization ability for several combinations of teacher and student learning rules are in very good agreement with numerical simulations

finding optimal algorithmic parameters using derivative free optimization the objectives of this paper are twofold we devise a general framework for identifying locally optimal algorithmic parameters algorithmic parameters are treated as decision variables in a problem for which no derivative knowledge or existence is assumed a derivative free method for optimization seeks to minimize some measure of performance of the algorithm being fine tuned this measure is treated as a black box and may be chosen by the user examples are given in the text the second objective is to illustrate this framework by specializing it to the identification of locally optimal trust region parameters in unconstrained optimization the derivative free method chosen to guide the process is the mesh adaptive direct search a generalization of pattern search methods we illustrate the flexibility of the latter and in particular make provision for surrogate objectives locally optimal parameters with respect to overall computational time on a set of test problems are identified each function call may take several hours and may not always return a predictable result a tailored surrogate function is used to guide the search towards a local solution the parameters thus identified differ from traditionally used values and allow one to solve a problem that remained otherwise unsolved in a reasonable time using traditional values

structural optimization using computational aerodynamics a recently developed methodology for aircraft structural design based on nonlinear airloads is extended to include a modal based optimization option and is employed with a new computational aerodynamics code for loads analysis nonlinear maneuver loads are evaluated by a computational scheme that efficiently combines fluid dynamics iterations with iterations for elastic shape deformations and trim corrections an efficient design process is obtained by performing several structural optimization runs during one maneuver load analysis where each optimization is based on the interim nonconverged airloads to allow for the efficient application of the method with large finite element structural models and many constraints the discrete coordinate optimization scheme is replaced by a modal based optimization where a set of low frequency vibration modes of the baseline structure is used to represent the structure throughout the optimization both for response analysis and for sensitivity analysis comparative modal based and discrete coordinate design cases are shown to converge to the same optimal design variable values even though they do not follow the same path two flow solvers are used one of which is a newly developed euler navier stokes computational aerodynamics code that is capable of handling complex geometries by using the chimera overset grid method the method avoids the problem of mesh discontinuities due to elastic shape deformations and control surface deflections because the displacements of each component affect only the component s mesh the method is demonstrated with a wing fuselage elevator transport aircraft model performing symmetric and antisymmetric maneuvers at mach 0 85

on line learning in the ising perceptron on line learning of both binary and continuous rules in an ising space is studied learning is achieved by using an artificial parameter a weight vector which is constrained to the surface of a hypersphere spherical constraint in the case of a binary rule the generalization error decays to zero super exponentially as exp c 2 where is the number of examples divided by n the size of the input vector and c 0 much faster learning is obtained in the case of continuous activation functions where the generalization error decays as exp e the number of steps required for perfect learning is estimated for both scenarios and compared with simulations

evaluation of dynamical models dissipative synchronization and other techniques some recent developments for the validation of nonlinear models built from data are reviewed besides giving an overall view of the field a procedure is proposed and investigated based on the concept of dissipative synchronization between the data and the model which is very useful in validating models that should reproduce dominant dynamical features like bifurcations of the original system in order to assess the discriminating power of the procedure four well known benchmarks have been used namely duffing ueda duffingholmes and van der pol oscillators plus the henon map the procedure developed for discrete time systems is focused on the dynamical properties of the model rather than on statistical issues for all the systems investigated it is shown that the discriminating power of the procedure is similar to that of bifurcation diagrams which in turn is much greater than say that of correlation dimension but at a much lower computational cost

perfect learning and power law in learning from stochastic examples by ising perceptrons analysis under one step replica symmetry breaking ansatz we study learning from stochastic examples by perceptrons with ising weights in the framework of statistical mechanics under the one step replica symmetry breaking ansatz the behaviour of learning curves are classified according to some local property of the rules by which examples are drawn the conditions for the existence of perfect learning together with other behaviours of the learning curves are given precisely the results agree with those obtained by seung 1995 using a refined annealled approximation

on learning perceptrons with binary weights we present an algorithm that pac learns any perceptron with binary weights and arbitrary threshold under the family of product distributions the sample complexity of this algorithm is of o n e 4 ln n and its running time increases only linearly with the number of training examples the algorithm does not try to find an hypothesis that agrees with all of the training examples rather it constructs a binary perceptron based on various probabilistic estimates obtained from the training examples we show that under the restricted case of the uniform distribution and zero threshold the algorithm reduces to the well known clipped hebb rule we calculate exactly the average generalization rate i e the learning curve of the algorithm under the uniform distribution in the limit of an infinite number of dimensions we find that the error rate decreases exponentially as a function of the number of training examples hence the average case analysis gives a sample complexity of o n ln 1 e a large improvement over the pac learning analysis the analytical expression of the learning curve is in excellent agreement with the extensive numerical simulations in addition the algorithm is very robust with respect to classification noise

learning from examples in fully connected committee machines the problem of learning from examples in two layer neural networks is studied within the framework of statistical mechanics a fully connected committee machine is trained to implement a task which is not linearly separable the generalization error as a function of the number of training examples per adjustable weight is calculated in the annealed approximation for both binary and continuous weights we find a first order transition with a discontinuous drop in the generalization error the transitions occur due to a specialization of the hidden units from a symmetric state to one with each hidden unit in the student network specialized on a corresponding unit in the teacher network the symmetric states of poor generalization remain metastable even for large training sets

extracting qualitative dynamics from experimental data abstract we consider the notion of qualitative information and the practicalities of extracting it from experimental data our approach based on a theorem of takens draws on ideas from the generalized theory of information known as singular system analysis due to bertero pike and co workers we illustrate our technique with numerical data from the chaotic regime of the lorenz model

towards more practical average bounds on supervised learning in this paper we describe a method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities the resulting theory provides a unified viewpoint of average case learning curves of concept learning and regression in realistic learning problems not necessarily within the bayesian framework the advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the vapnik chervonenkis vc theory and its alike and provides general insights into generalization besides the bounds on learning curves are directly related to the number of adjustable system weights although the theory is based on an approximation assumption and cannot apply to the worst case learning setting the precondition of the assumption is mild and the approximation itself is only a sufficient condition for the validity of the theory we illustrate the results with numerical simulations and apply the theory to examining the generalization ability of combination of neural networks

a derivative free nonmonotone line search technique for unconstrained optimization a tolerant derivative free nonmonotone line search technique is proposed and analyzed several consecutive increases in the objective function and also nondescent directions are admitted for unconstrained minimization to exemplify the power of this new line search we describe a direct search algorithm in which the directions are chosen randomly the convergence properties of this random method rely exclusively on the line search technique we present numerical experiments to illustrate the advantages of using a derivative free nonmonotone globalization strategy with approximated gradient type methods and also with the inverse sr1 update that could produce nondescent directions in all cases we use a local variation finite differences approximation to the gradient

dynamics of learning and generalization in perceptrons with constraints abstract depending on the kind of constraints imposed on the weights of a perceptron learning can be a combinatorially hard problem as an example of this type i discuss a perception with binary weights comparing results obtained from replica theory dynamic mean field theory and simulated annealing contrary to the replica calculation dynamics yields information about the performance of a polynomial algorithm in a situation where the best solution cannot be found in polynomial time i also discuss improved learning algorithms and results for finite size perceptrons

one step replica symmetry breaking solution for a perceptron learning with weight mismatch we investigate the properties of the one step replica symmetry breaking 1rsb solution for a perceptron learning from examples with weight mismatch where the entropy zero line crosses the almeida thouless at line of the rs solution for a small number of examples we find the optimal 1rsb solution which has the maximum free energy non negative entropy and satisfies the stability condition the at criterion for the 1rsb solution the transition from rs to 1rsb is continuous or discontinuous depending on whether the rs at line is above or below the zero entropy line however for a relatively large number of examples the 1rsb solution which maximizes the free energy becomes unstable and should be replaced by higher step rsb solutions we also obtain the at line for the 1rsb solution

on line learning in a discrete state space on line learning of a rule given by an n dimensional ising perceptron is considered for the case when the student is constrained to take values in a discrete state space of size for l 2 no on line algorithm can achieve a finite overlap with the teacher in the thermodynamic limit however if l is on the order of hebbian learning does achieve a finite overlap

pattern search algorithms for mixed variable programming many engineering optimization problems involve a special kind of discrete variable that can be represented by a number but this representation has no significance such variables arise when a decision involves some situation like a choice from an unordered list of categories this has two implications the standard approach of solving problems with continuous relaxations of discrete variables is not available and the notion of local optimality must be defined through a user specified set of neighboring points we present a class of direct search algorithms to provide limit points that satisfy some appropriate necessary conditions for local optimality for such problems we give a more expensive version of the algorithm that guarantees additional necessary optimality conditions a small example illustrates the differences between the two versions a real thermal insulation system design problem illustrates the efficacy of the user controls for this class of algorithms

generalization in a linear perceptron in the presence of noise the authors study the evolution of the generalization ability of a simple linear perceptron with n inputs which learns to imitate a teacher perceptron the system is trained on p alpha n example inputs drawn from some distribution and the generalization ability is measured by the average agreement with the teacher on test examples drawn from the same distribution the dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at alpha 1 when there are no errors static noise in the training examples if the examples are produced by an erroneous teacher overfitting is observed i e the generalization error starts to increase after a finite time of training it is shown that a weight decay of the same size as the variance of the noise errors on the teacher improves on the generalization and suppresses the overfitting the generalization error as a function of time is calculated numerically for various values of the parameters finally dynamic noise in the training is considered white noise on the input corresponds on average to a weight decay and can thus improve generalization whereas white noise on the weights or the output degrades generalization generalization is particularly sensitive to noise on the weights for alpha 1 where it makes the error constantly increase with time but this effect is also shown to be damped by a weight decay weight noise and output noise acts similarly above the transition at alpha 1

storage capacities of committee machines with overlapping and non overlapping receptive fields we present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number m of hidden units and spherical weights difficulties arise in the solution of this problem in the limit of large m in the case of overlapping receptive fields as the number of patterns increases both permutation symmetry and replica symmetry are broken which leads to the appearance of many order parameters and causes additional difficulty we observe that the relations among these order parameters yield a set of quantities which are small in the limit of large m making the asymptotic calculation tractable using the one step replica symmetry breaking scheme we compute the asymptotic value of the storage capacity per input unit in the limit of large m we find that the shift to the case of non overlapping receptive fields can be made easily we then find both values satisfy the bound of mitchison and durbin

reduced order models for computational fluid dynamics based nonlinear aeroelastic problems in this work a nonlinear state space based identification method is proposed to describe compactly unsteady aerodynamic responses such a reduced order model is trained on a series of signals that implicitly represent the relationship between the structural motion and the aerodynamic loads the determination of the model parameters is obtained through a two level training procedure where in the first stage the matrices associated to the linear part of the model are computed by a robust subspace projection technique whereas the remaining nonlinear terms are determined by an output error minimization procedure in the second stage the present approach is tested on two different problems proving the convergence toward the reference results obtained by a computational fluid dynamics solver in linear and nonlinear aerodynamic and aeroelastic applications whereas the aerodynamic reduced order models are coupled with the related structural mechanical systems demonstrating the ability of capturing the ma

how to use expert advice we analyze algorithms that predict a binary value by combining the predictions of several prediction strategies called experts our analysis is for worst case situations i e we make no assumptions about the way the sequence of bits to be predicted is generated we measure the performance of the algorithm by the difference between the expected number of mistakes it makes on the bit sequence and the expected number of mistakes made by the best expert on this sequence where the expectation is taken with respect to the randomization in the predictins we show that the minimum achievable difference is on the order of the square root of the number of mistakes of the best expert and we give efficient algorithms that achieve this our upper and lower bounds have matching leading constants in most cases we then show how this leads to certain kinds of pattern recognition learning algorithms with performance bounds that improve on the best results currently know in this context we also compare our analysis to the case in which log loss is used instead of the expected number of mistakes

reconstructing bifurcation diagrams only from time waveforms abstract a simple algorithm is proposed for reconstructing bifurcation diagrams only from time waveforms of a dynamical system without knowing an explicit form of the dynamics and its dependence on the parameters reconstruction means that the obtained bifurcation diagram is qualitatively similar to that of the original system the algorithm consists of two steps first a nonlinear predictor is sought using a particular class of functions with many parameters second the karhunen loeve transform is used to find only significant parameters contributing the bifurcations the algorithm is tested against two families of dynamical system the henon family and the coupled logistic delayed logistic family

neural networks for feedback feedforward nonlinear control systems this paper deals with the problem of designing feedback feedforward control strategies to drive the state of a dynamic system in general nonlinear so as to track any desired trajectory joining the points of given compact sets while minimizing a certain cost function in general nonquadratic due to the generality of the problem conventional methods are difficult to apply thus an approximate solution is sought by constraining control strategies to take on the structure of multilayer feedforward neural networks after discussing the approximation properties of neural control strategies a particular neural architecture is presented which is based on what has been called the linear structure preserving principle the original functional problem is then reduced to a nonlinear programming one and backpropagation is applied to derive the optimal values of the synaptic weights recursive equations to compute the gradient components are presented which generalize the classical adjoint system equations of n stage optimal control theory simulation results related to nonlinear nonquadratic problems show the effectiveness of the proposed method

reduced order models for nonlinear unsteady aerodynamics two reduced order modeling approaches for the evaluation of nonlinear aerodynamic forces based on cfd computations are presented these reducedorder models roms provide a means for rapid calculation of frequency domain generalized aerodynamic forces which can be used in traditional flutter analysis scheme to calculate flutter characteristics about nonlinear steady flows two roms are presented one that is based on the volterra theory for nonlinear systems and a new rom that is based on step indicial responses rom kernels are identified directly from input output relations and the study focuses on issues of kernel identification and their effect on the quality of the rom first and second order roms are generated for the response of the agard 445 6 wing to forced harmon ic excitation of its elastic modes responses computed from the roms are compared to responses obtained directly from a cfd analysis in which the boundary conditions are excited harmonically results show that the quality of the volterra rom is very sensitive to the amplitudes of the impulse inputs used for identification the step response rom is shown to be more accurate than the volterra rom and less sensitive to the amplitude used for its identification

determining lyapunov exponents from a time series we present the first algorithms that allow the estimation of non negative lyapunov exponents from an experimental time series lyapunov exponents which provide a qualitative and quantitative characterization of dynamical behavior are related to the exponentially fast divergence or convergence of nearby orbits in phase space a system with one or more positive lyapunov exponents is defined to be chaotic our method is rooted conceptually in a previously developed technique that could only be applied to analytically defined model systems we monitor the long term growth rate of small volume elements in an attractor the method is tested on model systems with known lyapunov spectra and applied to data for the belousov zhabotinskii reaction and couette taylor flow

neural network training without spurious minima physics department national university la plata casilla de correro 727 1900 la plata argentina received 2 january 1996 in training student perceptions recourse to information theory concepts allows one to select the best workinghypothesis and obtain an exact solution for the associated probability distribution we apply this trainingscheme to perceptions with binary weights and show that no phase transition ensues by recourse to ourapproach fast learning is guaranteed and trapping by spurious local minima is avoided s1063 651x 96 07505 8 pacs number s 87 10 1e 05 20 2y 02 70 2ci introduction

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

average case learning curves for radial basis function networks the application of statistical physics to the study of the learning curves of feedforward connectionist networks has to date been concerned mostly with perceptron like networks recent work has extended the theory to networks such as committee machines and parity machines and an important direction for current and future research is the extension of this body of theory to further connectionist networks in this article we use this formalism to investigate the learning curves of gaussian radial basis function networks rbfns having fixed basis functions these networks have also been called generalized linear regression models we address the problem of learning linear and nonlinear realizable and unrealizable target rules from noise free training examples using a stochastic training algorithm expressions for the generalization error defined as the expected error for a network with a given set of parameters are derived for general gaussian rbfns for which all parameters including centers and sp

pattern search methods for user provided points application to molecular geometry problems this paper deals with the application of pattern search methods to the numerical solution of a class of molecular geometry problems with important applications in molecular physics and chemistry the goal is to find a configuration of a cluster or a molecule with minimum total energy the minimization problems in this class of molecular geometry problems have no constraints and the objective function is smooth the difficulties arise from the existence of several local minima and especially from the expensive function evaluation total energy and the possible nonavailability of first order derivatives we introduce a pattern search approach that attempts to exploit the physical nature of the problem by using energy lowering geometrical transformations and to take advantage of parallelism without the use of derivatives numerical results for a particular instance of this new class of pattern search methods are presented showing the promise of our approach the new pattern search methods can be used in any other context where there is a user provided scheme to generate points leading to a potential objective function decrease