high dimensional inference with the generalized hopfield model principal component analysis and corrections we consider the problem of inferring the interactions between a set of n binary variables from the knowledge of their frequencies and pairwise correlations the inference framework is based on the hopfield model a special case of the ising model where the interaction matrix is defined through a set of patterns in the variable space and is of rank much smaller than n we show that maximum likelihood inference is deeply related to principal component analysis when the amplitude of the pattern components ensuremath xi is negligible compared to sqrt n using techniques from statistical mechanics we calculate the corrections to the patterns to the first order in ensuremath xi sqrt n we stress the need to generalize the hopfield model and include both attractive and repulsive patterns in order to correctly infer networks with sparse and strong interactions we present a simple geometrical criterion to decide how many attractive and repulsive patterns should be considered as a function of the sampling noise we moreover discuss how many sampled configurations are required for a good inference as a function of the system size n and of the amplitude ensuremath xi the inference approach is illustrated on synthetic and biological data

statistical mechanics and phase transitions in clustering a new approach to clustering based on statistical physics is presented the problem is formulated as fuzzy clustering and the association probability distribution is obtained by maximizing the entropy at a given average variance the corresponding lagrange multiplier is related to the temperature and motivates a deterministic annealing process where the free energy is minimized at each temperature critical temperatures are derived for phase transitions when existing clusters split it is a hierarchical clustering estimating the most probable cluster parameters at various average variances

phase transitions in simple learning the authors investigate learning in the simplest type of a layered neural network the one layer perceptron the learning process is treated as a statistical dynamical problem quantities one is interested in include the relaxation time the learning time and the capacity and how they depend on noise and constraints on the weights the relaxation time is calculated as a function of the noise level and the number p of associations to be learned they consider three different cases for input patterns that are random and uncorrelated in the first where the connection weights are constrained to satisfy n 1 sigma i omega i2 s2 there is a critical value of p n separating regimes of perfect and imperfect learning at zero noise in contrast the second model unconstrained learning exhibits a different kind of transition at p n and noise plays no role in the third model where the constraint is imposed only on the thermal fluctuations there is a line of phase transitions terminating at p n and zero noise they have also considered learning with correlated input patterns the most important difference is the emergence of a second relaxation time which the authors interpret as the time it takes to learn a prototype of the patterns

on consistency and sparsity for principal components analysis in high dimensions principal components analysis pca is a classic method for the reduction of dimensionality of data in the form of n observations or cases of a vector with p variables contemporary datasets often have p comparable with or even much larger than n our main assertions in such settings are a that some initial reduction in dimensionality is desirable before applying any pca type search for principal modes and b the initial reduction in dimensionality is best achieved by working in a basis in which the signals have a sparse representation we describe a simple asymptotic model in which the estimate of the leading principal component vector via standard pca is consistent if and only if p n n 0 we provide a simple algorithm for selecting a subset of coordinates with largest sample variances and show that if pca is done on the selected subset then consistency is recovered even if p n n

structure of metastable states in the hopfield model an upper bound for the number of metastable states in the hopfield model is calculated as a function of the hamming fraction from an input pattern for all finite values of alpha the ratio of number of patterns to nodes the hamming fraction from the input pattern to the nearest metastable state is infinite when alpha 0 113 the bound also implies that there is a gap between a set of states close to the input pattern and another set centred around the hamming fraction 0 5 from it

learning and retrieval in attractor neural networks above saturation the authors investigate neural networks in the range of parameters when the ground state energy is positive namely when a synaptic matrix which satisfies all the desired constraints cannot be found by the learning algorithm in particular they calculate the typical distribution functions of local stabilities obtained for a number of algorithms in this region these functions are used to investigate the retrieval properties as reflected by the size of the basins of attraction this is done analytically in sparsely connected networks and numerically in fully connected networks the main conclusion is that the retrieval behaviour of attractor neural networks can be improved by learning above saturation

empirical risk minimization versus maximum likelihood estimation a case study we study the interaction between input distributions learning algorithms and finite sample sizes in the case of learning classification tasks focusing on the case of normal input distributions we use statistical mechanics techniques to calculate the empirical and expected or generalization errors for several well known algorithms learning the weights of a single layer perceptron in the case of spherically symmetric distributions within each class we find that the simple hebb rule corresponding to maximum likelihood parameter estimation outperforms the other more complex algorithms based on error minimization moreover we show that in the regime where the overlap between the classes is large algorithms with low empirical error do worse in terms of generalization a phenomenon known as overtraining

is there something out there inferring space from sensorimotor dependencies this letter suggests that in biological organisms the perceived structure of reality in particular the notions of body environment space object and attribute could be a consequence of an effort on the part of brains to account for the dependency between their inputs and their outputs in terms of a small number of parameters to validate this idea a procedure is demonstrated whereby the brain of a simulated organism with arbitrary input and output connectivity can deduce the dimensionality of the rigid group of the space underlying its input output relationship that is the dimension of what the organism will call physical space

learning and retrieval in attractor neural networks with noise abstract a recent study on noiseless learning and retrieval in attractor neural networks above saturation by griniasty and gutfreund is extended to take account for imperfect learning by means of a temperature 1 t violations of the constraint imposed on the local stabilities are taken into account by various cost functions the distribution of local stabilities and the fraction of errors during the learning stage are analysed the retrieval dynamics for the sparsely connected network is studied showing high retrieval overlap in reduced retrieval regions for finite t

finite element theory for curved and twisted beams based on exact solutions for three dimensional solids part 2 anisotropic and advanced beam models consistent finite element formulations for beams made of anisotropic materials and taking into account non classic inhomogeneous torsion have been developed the formulations are based on a kinematical hypothesis that includes exact solutions for three dimensional solids under terminal loading they describe warping of the cross sections in and out of their planes as well as their rigid displacements and rotations their large deformation and geometrically exact description by finite rotations are considered for the cases of monoclinic orthotropic and transversely isotropic materials exact solutions for the solid made from a monoclinic material have been deduced c 1998 elsevier science s a all rights reserved

large displacement analysis of thin walled beams with generic open section a finite element formulation for geometric non linear analysis of elastic straight thin walled beams is presented the updated lagrangian procedure is used for a stiffness matrix derivation to analyse large displacements or buckling behaviour of space frames the stiffness matrix is developed for a generic cross section and is equally efficient for symmetric monosymmetric or asymmetric open sections some non linear strain terms usually neglected in the literature have been considered resulting in a formulation which also includes the initial bimoment and torsion effect for all cross sections analysed the proposed method has been implemented and the results of sample problems are presented

retrieval phase diagrams for attractor neural networks with optimal interactions the authors consider the retrieval properties of attractor neural networks whose synaptic matrices have been constructed to maximise the number of patterns which can be stored in a perceptron satisfying certain constraints retrieval is studied in the absence as well as in the presence of fast noise temperature the discussion is restricted to dilute networks for which dynamical equations for the overlaps are available when the patterns are stored with a prescribed lower limit on the stability parameter on every site the full stability of the perceptron ensures the existence of an attractor with perfect retrieval it is found that the curve of critical storage capacity alpha against temperature t as a line of first order transitions for high values of alpha and becomes second order for low alpha at a point of a tricritical nature the phase diagram is compared with the dilute hopfield model it is found that at high synaptic noise levels the diluted hopfield net stores more effectively than the network trained for optimal perceptron storage when a given fraction of sites is allowed to violate the stability bound the solution of the perceptron learning problem does not ensure the existence of an attractor of finite overlap even in the absence of noise this case is studied separately for t 0 and for finite t

a corotational based geometrically nonlinear generalized beam theory buckling fe analysis abstract a geometrically nonlinear generalized beam theory is formulated and the results in the framework of buckling analyses are discussed the geometrically nonlinear model is recovered reusing the model available in the linear context this generalization to the nonlinear context is obtained exploiting the corotational based method called implicit corotational method starting from a mixed energy description of the continuum in terms of nonlinear biot stress strain tensors and using convenient assumptions for the linear stress strain tensors once obtained the nonlinear model has been implemented using a flexibility based finite element the results of buckling analyses for different and complex beam cross section geometries emphasizing distortional and local buckling behaviors are presented and comparisons with finite element shell models made

learning from correlated examples in a perceptron using the replica method the authors investigate learning from correlated data in a perceptron both types of association spatial within each example and semantic among the different ones are considered for both boolean and continuous weights and output functions spatial correlations of data may significantly improve the learning properties whereas the semantic ones do not practically influence them

playing billiards in version space a ray tracing method inspired by ergodic billiards is used to estimate the theoretically best decision rule for a given set of linear separable examples for randomly distributed examples the billiard estimate of the single perceptron with best average generalization probability agrees with known analytic results while for real life classification problems the generalization probability is consistently enhanced when compared to the maximal stability perceptron

the roles of stability and symmetry in the dynamics of neural networks the authors study the retrieval phase of spin glass like neural networks considering that the dynamics should depend only on gauge invariant quantities they propose that two such parameters characterising the symmetry of the neural net s connections and the stabilities of the patterns are responsible for most of the dynamical effects this is supported by a numerical study of the shape of the basins of attraction for a one pattern neural network model the effects of stability and symmetry on the short time dynamics of this model are studied analytically and the full dynamics for vanishing symmetry is shown to be exactly solvable

finite size effects and bounds for perceptron models ablncl in lhis paper we consider two main aspsln of the binary perccptron problem the maximal capacity when random paltem are stored model a and its generalhation ability model b we have extended previous numerical estimates of critical capacilies and studied thermal properties of systems of small sizes to lest recent replica predictions we have also considered some simpler versions of thest models the discrete spherical vemions can be salved exactly using gardner s replica calculation for the spherical model and are shown to give a rigorous upper bound and lower bound on the capacities of models a and b respectively iby vemions of models a and b are soivcd in detail and provide information which is useful for interpreting the bnile size eeeciects present in the numerical studies of models a and b

static and dynamic analysis of space frames using simple timoshenko type elements in this paper a finite element method for geometrically and materially non linear analyses of space frames is described beams with both solid and thin walled open cross sections are considered the equations of equilibrium are formulated using an updated incremental lagrangian description the elements developed can undergo large displacements and rotations but the incremental rotations are assumed to be small the material behaviour is described by elastoplastic temperature dependent elastoplastic and viscoplastic models with special reference to metals computationally more economical formulations based on the relationship between stress resultants and generalized strain quantities are also presented in the case of thin walled beams the torsional behaviour is modelled using a two parameter warping model where the angle of twist and the axial variation of warping have independent approximations this approach yields average warping shear strains directly from the displacement assumptions and no discrepancy between stress and strain fields exists

large systems of random linear equations with nonnegative solutions characterizing the solvable and the unsolvable phase large systems of linear equations are ubiquitous in science quite often e g when considering population dynamics or chemical networks the solutions must be nonnegative recently it has been shown that large systems of random linear equations exhibit a sharp transition from a phase where a nonnegative solution exists with probability one to one where typically no such solution may be found the critical line separating the two phases was determined by combining farkas lemma with the replica method here we show that the same methods remain viable to characterize the two phases away from criticality to this end we analytically determine the residual norm of the system in the unsolvable phase and a suitable measure of robustness of solutions in the solvable one our results are in very good agreement with numerical simulations

optimal pruning in neural networks we study pruning strategies in simple perceptrons subjected to supervised learning our analytical results obtained through the statistical mechanics approach to learning theory are independent of the learning algorithm used in the training process we calculate the post training distribution mathcal p j of synaptic weights which depends only on the overlap ensuremath rho 0 achieved by the learning algorithm before pruning and the fraction ensuremath kappa of relevant weights in the teacher network from this distribution we calculate the optimal pruning strategy for deleting small weights the optimal pruning threshold grows from zero as ensuremath theta mathrm opt ensuremath rho 0 ensuremath kappa ensuremath propto ensuremath rho 0 ensuremath ensuremath rho c ensuremath kappa 1 2 above some critical value ensuremath rho c ensuremath kappa thus the elimination of weak synapses enhances the network performance only after a critical learning period possible implications for biological pruning phenomena are discussed

beyond storage capacity in a single model neuron continuous replica symmetry breaking a single mcculloch pitts neuron that is the simple perceptron is studied with focus on the region beyond storage capacity it is shown that parisi s hierarchical ansatz for the overlap matrix of the synaptic couplings with so called continuous replica symmetry breaking is a solution and as we propose it is the exact one to the equilibrium problem we describe some of the most salient features of the theory and give results about the low temperature region in particular the basics of the parisi technique and the way to calculate thermodynamical expectation values is explained we have numerically extremized the replica free energy functional for some parameter settings and thus obtained the order parameter function i e the probability distribution of overlaps that enabled us to evaluate the probability density of the local stability parameter we also performed a simulation and found a local stability density closer to the theoretical curve than previous numerical results were

statistical mechanics of learning multiple orthogonal signals asymptotic theory and fluctuation effects the learning of signal directions in high dimensional data through orthogonal decomposition or principal component analysis pca has many important applications in physics and engineering disciplines e g wireless communication information theory and econophysics the accuracy of the orthogonal decomposition can be studied using mean field theory previous analysis of data produced from a model with a single signal direction has predicted a retarded learning phase transition below which learning is not possible i e if the signal is too weak or the data set is too small then it is impossible to learn anything about the signal direction or magnitude in this contribution we show that the result can be generalized to the case where there are multiple signal directions each nondegenerate signal is associated with a retarded learning transition however fluctuations around the mean field solution lead to large finite size effects unless the signal strengths are very well separated we evaluate the one loop contribution to the mean field theory which shows that signal directions are indistinguishable from one another if their corresponding population eigenvalues are separated by o n tau with exponent tau 1 3 where n is the data dimension numerical simulations are consistent with the analysis and show that finite size effects can persist even for very large data sets

the failure distribution in percolation models of breakdown the probability of finding a largest defect cluster of size n in a percolation network is calculated analytically using a new distribution function scaling equation from this result and the stress voltage enhancement at the tip of the most critical defect in the network the probability of failure in percolation models of breakdown is calculated for defect fractions less than the percolation point this distribution is found to be of the form exponential of an exponential numerical simulations on the two dimensional random fuse network confirm the new distribution function and convincingly distinguish between it and the weibull 1951 form most often used in the fitting of breakdown data

optimal storage of a neural network model a replica symmetry breaking solution a break in replica symmetry is found above the critical line alpha c kappa for storage with a finite minimal fraction of errors within the first stage of the parisi scheme for noiseless networks with continuous synapses the replica symmetry breaking solution yields the highest most stable minimal fraction of errors everywhere above alpha c kappa

statistical mechanics of unsupervised hebbian learning a model describing the dynamics of the synaptic weights of a single neuron performing hebbian learning is described the neutron is repeatedly excited by a set of input patterns its response is modelled as a continuous nonlinear function of its excitation the authors study how the model forms a self organized representation of the set of input patterns the dynamical equations ae solved directly in a few simple cases the model is studied for random patterns by a signal to noise analysis and by introducing a partition function and applying the replica approach as the number of patterns is increased a first order phase transition occurs where the neuron becomes unable to remember one pattern but learns instead a mixture of very many patterns the critical number of patterns for this transition scales as nb where n is the number of synapses and b is the degree of nonlinearity the leading order finite size corrections are calculated and compared with numerical simulations it is shown how the representation of the input patterns learned by the neutron depends upon the nonlinearity in the neuron s response two types of behaviour can be identified depending on the degree of nonlinearity either the neuron learns to discriminate one pattern from all the others or it will learn to discriminate a complex mixture of many of the patterns

perceptrons above saturation we study the storage of random patterns by a perceptron above its storage capacity alpha c i e in the region where perfect storage becomes impossible we determine the minimal fraction of learning errors and the distribution of stabilities for different learning rules in one step replica symmetry breaking thereby we not only extend the known replica symmetric results to values of the storage capacity beyond the at line but also show that depending on the learning rule replica symmetry may be globally unstable already well below the at line as an example for possible implications we compare the results for the typical basins of attraction of an extremely diluted attractor neural network as given by replica symmetry and one step replica symmetry breaking

estimates of optimal storage conditions in neural network memories based on random matrix theory the authors formulate a method for estimating the critical conditions for storage of sets of data in neural network memory this variational method is based on random matrix theory and calculating the average spectrum of a matrix whose elements are given by overlaps of the stored patterns several generic cases of random overlap matrices are considered they investigate the cases of simply uncorrelated random patterns and spatially and semantically correlated ones they obtain bounds of the critical curve in the control parameters space which determine the stability of the stored data sets

weight space structure and generalization in the reversed wedge perceptron the generalization ability of the reversed wedge perceptron serving as a toy model for multilayer neural networks is investigated we analyse the decomposition of the version space into disjoint cells belonging to different internal representations defined by the signs of the aligning fields the version space is characterized by the number and size of these cells and their typical overlap with the teacher network for a small training set the system is unable to detect the structure of the patterns induced by the teacher accordingly it performs as if storing random input output patterns with very low generalization ability and a large misfit in the internal representation with increasing training set size cells with large misfit are eliminated at a much higher rate than those with internal representation similar to that of the teacher this results eventually in the discontinuous phase transition to good generalization typical for multilayer neural networks

semi analytical finite elements in the higher order theory of beams abstract the paper presents semi analytical finite elements leading to the development of a higher order theory for beams an analysis of existing beam theories reveals their semi analytical character the semi analytical finite element sfe method proposed assumes the cross section of a solid beam to be an assemblage of certain semi analytical elements the sfe method is based on independent approximations of three dimensional fields of displacements and stresses the method reflects the multi level nature of beams where conventional finite elements are transferred to the level of beam equations for heuristic considerations a mathematical formulation of the linear theory of a solid beam involves the equations of compatibility equilibrium and constitutive relationships i e the elastic law as well as static and kinematic boundary conditions compatibility conditions are derived directly by using finite element approximations while equilibrium equations constitutive equations and boundary conditions may be derived by applying stationarity of a modified hellinger reissner functional this derivation may be also defined by straightforward application of a generalised virtual work formulation the equations of a beam are formulated in terms of generalised variables integral resultants of stresses and strains as well as of nodal displacements the definition of generalised variables is assigned to the nodes of the finite element mesh bending warping and other higher order deformation modes are described automatically finally the set of governing equations includes mixed algebraic differential operators their explicit expressions depend on one of the element types isoparametric subparametric or superparametric the versatility of sfe is demonstrated in practical applications involving well known solutions the classical euler bernoulli and timoshenko theories are derived using a single one dimensional element the beams with a thin walled cross section are described as an assemblage of semi analytical elements on the basis of a higher order theory the conventional semi analytically based finite elements sabfe are derived their basic relations are also presented some numerical examples of the applications to thin walled beams illustrate the basic features possible extensions of sfe to dynamics geometric non linearity and shells are briefly discussed

maximal stability in unsupervised learning a perceptron can classify a set of random patterns into two groups the maximal gap between the two groups is calculated using replica theory the replica symmetric solution is shown to be unstable and gives results which are qualitatively different from the one step replica symmetry breaking solution rsb1 the results of a calculation without the replica method are given yielding an exact upper bound for the gap which almost coincides with the rsb1 solution finding the classification of a set of patterns with a maximal gap is a difficult combinatorial optimization problem an algorithm is developed which gives large gaps

optimal capacity of graded response perceptrons a replica symmetry breaking solution optimal capacities of perceptrons with graded input output relations are studied within the first step replica symmetry breaking gardner approach input data errors and a limited output precision are allowed in particular the role of non monotonicity in the input output relations on the breaking and on the overall performance is determined

the computational complexity of propositional strips planning abstract i present several computational complexity results for propositional strips planning i e strips planning restricted to ground formulas different planning problems can be defined by restricting the type of formulas placing limits on the number of pre and postconditions by restricting negation in pre and postconditions and by requiring optimal plans for these types of restrictions i show when planning is tractable polynomial and intractable np hard in general it is pspace complete to determine if a given planning instance has any solutions extremely severe restrictions on both the operators and the formulas are required to guarantee polynomial time or even np completeness for example when only ground literals are permitted determining plan existence is pspace complete even if operators are limited to two preconditions and two postconditions when definite horn ground formulas are permitted determining plan existence is pspace complete even if operators are limited to zero preconditions and one postcondition one of the interesting tractable problems is if each operator is restricted to positive preconditions and one postcondition only ground literals the blocks world problem slightly modified is a subproblem of this restricted planning problem these results in combination with previous analyses are not encouraging for domain independent planning

planning for conjunctive goals abstract the problem of achieving conjunctive goals has been central to domain independent planning research the nonlinear constraint posting approach has been most successful previous planners of this type have been complicated heuristic and ill defined i have combined and distilled the state of the art into a simple precise implemented algorithm tweak which i have proved correct and complete i analyze previous work on domain independent conjunctive planning in retrospect it becomes clear that all conjunctive planners linear and nonlinear work the same way the efficiency and correctness of these planners depends on the traditional add delete list representation for actions which drastically limits their usefulness i present theorems that suggest that efficient general purpose planning with more expressive action representations is impossible and suggest ways to avoid this problem

minimizing conflicts a heuristic repair method for constraint satisfaction and scheduling problems abstract the paper describes a simple heuristic approach to solving large scale constraint satisfaction and scheduling problems in this approach one starts with an inconsistent assignment for a set of variables and searches through the space of possible repairs the search can be guided by a value ordering heuristic the min conflicts heuristic that attempts to minimize the number of constraint violations after each step the heuristic can be used with a variety of different search strategies we demonstrate empirically that on the n queens problem a technique based on this approach performs orders of magnitude better than traditional backtracking techniques we also describe a scheduling application where the approach has been used successfully a theoretical analysis is presented both to explain why this method works well on certain types of problems and to predict when it is likely to be most effective

optimal basins of attraction in randomly sparse neural network models the size of the basin of attraction for randomly sparse neural networks with optimal interactions is calculated for all values of the storage ratio alpha p c 2 where p is the number of random uncorrelated patterns and c is the connectivity the basin of attraction is finite while for alpha 0 42 the basin of attraction is almost 100

learning and generalization with minimerror a temperature dependent learning algorithm we study the numerical performances of minimerror a recently introduced learning algorithm for the perceptron that has analytically been shown to be optimal both on learning linearly and nonlinearly separable functions we present its implementation on learning linearly separable boolean functions numerical results are in excellent agreement with the theoretical predictions

duality between learning machines a bridge between supervised and unsupervised learning we exhibit a duality between two perceptrons that allows us to compare the theoretical analysis of supervised and unsupervised learning tasks the first perceptron has one output and is asked to learn a classification of p patterns the second dual perceptron has p outputs and is asked to transmit as much information as possible on a distribution of inputs we show in particular that the maximum information that can be stored in the couplings for the supervised learning task is equal to the maximum information that can be transmitted by the dual perceptron

learning strategy for the binary perceptron pursuing the work of penney and sherrington we determine the optimal continuous weight perceptron which on clipping correctly predicts the largest number of weights for the binary perceptron with maximum stability we calculate the fraction of correctly predicted binary weights when only the continuous weights stronger than a certain threshold are clipped we finally carry out simulations for a perceptron with 50 weights to test the practicability of different learning strategies

effective modelling of spatial buckling of beam assemblages accounting for warping constraints and rotation dependency of moments a simple two noded finite element model for the three dimensional buckling analysis of beam assemblages is developed the underlying generalized beam theory employed accounts for all coupled significant modes of deformations including stretching bending shear and torsion as well as warping its mixed formulation is of the two field type utilizing linear and constant interpolants for displacements and strain fields due considerations are given to the effect off inite rotations in space on the non linear kinematic descriptions as well as the configuration dependent behaviour of externally applied moment vectors of the conservative and non conservative types the performance of the model and particularly the accuracy of its geometric and load correction stiffnesses is assessed in a fairly complete set of numerical simulations

a gaussian scenario for unsupervised learning we consider random patterns on the n sphere which are uniformly distributed with the exception of a single symmetry breaking orientation along which they are gaussian distributed the unsupervised recognition of this orientation by different learning rules is studied in the large n limit using the replica method the model is simple enough to be analytically tractable and rich enough to exhibit most of the phenomena observed with other pattern distributions a learning algorithm based on the minimization of a cost function is identified which reaches the upper theoretical limit imposed by the optimal bayes learning scenario an implementation of this algorithm is proposed and tested numerically

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

the dynamics of on line principal component analysis the learning dynamics of an on line algorithm for principal component analysis is described exactly in the thermodynamic limit by means of coupled ordinary differential equations for a set of order parameters it is demonstrated that learning is delayed significantly because existing symmetries among student vectors have to be broken a closely related effect is the perfect or partial loss of initial knowledge in the course of learning the analysis shows that different learning rates for the student vectors improve the performance of the algorithm drastically

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

bayes optimal performance in a discrete space we study a simple model of unsupervised learning where the single symmetry breaking vector has binary components 1 we calculate exactly the bayes optimal performance of an estimator which is required to lie in the same discrete space we also show that except for very special cases such an estimator cannot be obtained by minimization of a class of variationally optimal potentials

dimensionality estimation for optimal detection of functional networks in bold fmri data estimation of the intrinsic dimensionality of fmri data is an important part of data analysis that helps to separate the signal of interest from noise we have studied multiple methods of dimensionality estimation proposed in the literature and used these estimates to select a subset of principal components that was subsequently processed by linear discriminant analysis lda using simulated multivariate gaussian data we show that the dimensionality that optimizes signal detection in terms of the receiver operating characteristic roc metric goes through a transition from many dimensions to a single dimension as a function of the signal to noise ratio this transition happens when the loci of activation are organized into a spatial network and the variance of the networked task related signals is high enough for the signal to be easily detected in the data we show that reproducibility of activation maps is a metric that captures this switch in intrinsic dimensionality except for reproducibility all of the methods of dimensionality estimation we considered failed to capture this transition optimization of bayesian evidence minimum description length supervised and unsupervised lda prediction and stein s unbiased risk estimator this failure results in sub optimal roc performance of lda in the presence of a spatially distributed network and may have caused lda to underperform in many of the reported comparisons in the literature using real fmri data sets including multi subject group and within subject longitudinal analysis we demonstrate the existence of these dimensionality transitions in real data

on the ability of the optimal perceptron to generalise a linearly separable boolean function is derived from a set of examples by a perceptron with optimal stability the probability to reconstruct a pattern which is not learnt is calculated analytically using the replica method

geometrically exact thin walled beam including warping formulated on the special euclidean group se 3 abstract based on a formulation on the special euclidean group s e 3 a geometrically exact thin walled beam with an arbitrary open cross section is proposed to deal with the finite deformation and rotation issues the beam strains are based on a kinematic assumption where warping deformation and wagner effects are included such that the nonlinear behavior of a thin walled beam is predicted accurately particular under large torsion to reduce the nonlinearity of rigid motion static and dynamic equations are derived in the s e 3 framework based on the local frame approach as the value of the iteration matrix including the jacobian matrix of inertial and internal forces is invariable under arbitrary rigid motion the number of updates required during the computation process decreases sharply which drastically improves the computational efficiency furthermore the isogeometric analysis iga based on the non uniform rational b splines nurbs basis functions which promotes the integration of computer aided design cad and computer aided engineering cae is adopted to interpolate the displacement rotation and warping fields separately the interpolated strain measures satisfy the objectivity by removing the rigid motion of the reference point to obtain the symmetric jacobian matrix of internal forces the linearization operation is conducted based on the previously converged configuration a lie group s e 3 extension of the generalized time integration method is utilized to solve the equations of motion for thin walled beams finally the proposed formulation is successfully tested and validated in several static and dynamic numerical examples

biased learning in boolean perceptrons abstract the generalization ability of hebbian boolean perceptrons can be improved by a kind of feedback mechanism in which the student net judges the difficulty of a new example from its previous experience it is shown that by giving a higher weight to the hard examples both generalization and learning abilities can be increased analytical as well as numerical results are presented for both cases where the examples are drawn at random or selected in an intelligent form

automatic pca dimension selection for high dimensional data and small sample sizes bayesian inference from high dimensional data involves the integration over a large number of model parameters accurate evaluation of such high dimensional integrals raises a unique set of issues these issues are illustrated using the exemplar of model selection for principal component analysis pca a bayesian model selection criterion based on a laplace approximation to the model evidence for determining the number of signal principal components present in a data set has previously been show to perform well on various test data sets using simulated data we show that for d dimensional data and small sample sizes n the accuracy of this model selection method is strongly affected by increasing values of d by taking proper account of the contribution to the evidence from the large number of model parameters we show that model selection accuracy is substantially improved the accuracy of the improved model evidence is studied in the asymptotic limit d at fixed ratio n d with 1 in this limit model selection based upon the improved model evidence agrees with a frequentist hypothesis testing approach

pattern selectivity in neural networks as a means of understanding basin structures the authors study the problem of learning and retrieving for a pair of correlated patterns within an extensive number of uncorrelated patterns for networks where learning may be treated as an optimization process with respect to an arbitrary cost function this formulation is then applied to several specific examples where they study the pattern selectivity of these systems i e their ability to differentiate correlated patterns and investigate the process of basin shrinking with increasing loading levels they define and discuss several different retrieval phases whose existence depends on the competitive interplay of the loading level and the pattern correlation discussion of asymptotic retrieval is restricted to dilute asymmetric networks

a general framework for consistency of principal component analysis a general asymptotic framework is developed for studying consistency properties of principal component analysis pca our framework includes several previously studied domains of asymptotics as special cases and allows one to investigate interesting connections and transitions among the various domains more importantly it enables us to investigate asymptotic scenarios that have not been considered before and gain new insights into the consistency subspace consistency and strong inconsistency regions of pca and the boundaries among them we also establish the corresponding convergence rate within each region under general spike covariance models the dimension or number of variables discourages the consistency of pca while the sample size and spike information the relative size of the population eigenvalues encourage pca consistency our framework nicely illustrates the relationship among these three types of information in terms of dimension sample size and spike size and rigorously characterizes how their relationships aect pca consistency

statistical mechanics of unsupervised structure recognition a model of unsupervised learning is studied where the environment provides n dimensional input examples that are drawn from two overlapping gaussian clouds we consider the optimization of two different objective functions the search for the direction of the largest variance in the data and the largest separating gap stability between clusters of examples respectively by means of a statistical mechanics analysis we investigate how well the underlying structure is inferred from a set of examples the performances of the learning algorithms depend crucially on the actual shape of the input distribution a generic result is the existence of a critical number of examples needed for successful learning the learning strategies are compared with methods different in spirit such as the estimation of parameters in a model distribution and an information theoretical approach

unsupervised learning of binary vectors a gaussian scenario we study a model of unsupervised learning where the real valued data vectors are isotropically distributed except for a single symmetry breaking binary direction bin 1 1 n onto which the projections have a gaussian distribution we show that a candidate vector j undergoing gibbs learning in this discrete space approaches the perfect match j b exponentially in addition to the second order retarded learning phase transition for unbiased distributions we show that first order transitions can also occur extending the known result that the center of mass of the gibbs ensemble has bayes optimal performance we show that taking the sign of the components of this vector clipping leads to the vector with optimal performance in the binary space these upper bounds are shown generally not to be saturated with the technique of transforming the components of a special continuous vector except in asymptotic limits and in a special linear case simulations are presented which are in excellent agreement with the theoretical results

replica analysis of bayesian data clustering we use statistical mechanics to study model based bayesian data clustering in this approach each partition of the data into clusters is regarded as a microscopic system state the negative data log likelihood gives the energy of each state and the data set realisation acts as disorder optimal clustering corresponds to the ground state of the system and is hence obtained from the free energy via a low temperature limit we assume that for large sample sizes the free energy density is self averaging and we use the replica method to compute the asymptotic free energy density the main order parameter in the resulting replica symmetric theory the distribution of the data over the clusters satisfies a self consistent equation which can be solved by a population dynamics algorithm from this order parameter one computes the average free energy and all relevant macroscopic characteristics of the problem the theory describes numerical experiments perfectly and gives a significant improvement over the mean field theory that was used to study this model in past

complexity optimized data clustering by competitive neural networks data clustering is a complex optimization problem with applications ranging from vision and speech processing to data transmission and data storage in technical as well as in biological systems we discuss a clustering strategy that explicitly reflects the tradeoff between simplicity and precision of a data representation the resulting clustering algorithm jointly optimizes distortion errors and complexity costs a maximum entropy estimation of the clustering cost function yields an optimal number of clusters their positions and their cluster probabilities our approach establishes a unifying framework for different clustering methods like k means clustering fuzzy clustering entropy constrained vector quantization or topological feature maps and competitive neural networks

threshold induced phase transitions in perceptrons error rates of a boolean perceptron with threshold and either spherical or ising constraint on the weight vector are calculated for storing patterns from biased input and output distributions derived within a one step replica symmetry breaking rsb treatment for unbiased output distribution and non zero stability of the patterns we find a critical load p above which two solutions to the saddlepoint equations appear one with higher free energy and zero threshold and a dominant solution with non zero threshold we examine this second order phase transition and the dependence of p on the required pattern stability for both one step rsb and replica symmetry rs in the spherical case and for one step rsb in the ising case

a probabilistic analysis of prepositional strips planning abstract i present a probabilistic analysis of prepositional strips planning the analysis considers two assumptions one is that each possible precondition likewise postcondition of an operator is selected independently of other pre and postconditions the other is that each operator has a fixed number of preconditions likewise postconditions under both assumptions i derive bounds for when it is highly likely that a planning instance can be efficiently solved either by finding a plan or proving that no plan exists roughly if planning instances under either assumption have n propositions ground atoms and g goals and the number of operators is less than an o n in g bound then a simple efficient algorithm can prove that no plan exists for most instances if the number of operators is greater than an n in g bound then a simple efficient algorithm can find a plan for most instances the two bounds differ by a factor that is exponential in the number of pre and postconditions a similar result holds for plan modification i e solving a planning instance that is close to another planning instance with a known plan thus it appears that prepositional strips planning a pspace complete problem exhibits a easy hard easy pattern as the number of available operators increases with a narrow range of hard problems an empirical study demonstrates this pattern for particular parameter values because prepositional strips planning is pspace complete this extends previous phase transition analyses which have focused on np complete problems also the analysis shows that surprisingly simple algorithms can solve a large subset of the planning problem

sparse principal component analysis and iterative thresholding principal component analysis pca is a classical dimension reduction method which projects data onto the principal subspace spanned by the leading eigenvectors of the covariance matrix however it behaves poorly when the number of features p is comparable to or even much larger than the sample size n in this paper we propose a new iterative thresholding approach for estimating principal subspaces in the setting where the leading eigenvectors are sparse under a spiked covariance model we find that the new approach recovers the principal subspace and leading eigenvectors consistently and even optimally in a range of high dimensional sparse settings simulated examples also demonstrate its competitive performance

kernel functions for case based planning case based planning can take advantage of former problem solving experiences by storing in a plan library previously generated plans that can be reused to solve similar planning problems in the future although comparative worst case complexity analyses of plan generation and reuse techniques reveal that it is not possible to achieve provable efficiency gain of reuse over generation we show that the case based planning approach can be an effective alternative to plan generation when similar reuse candidates can be chosen in this paper we describe an innovative case based planning system called oakplan which can efficiently retrieve planning cases from plan libraries containing more than ten thousand cases choose heuristically a suitable candidate and adapt it to provide a good quality solution plan which is similar to the one retrieved from the case library given a planning problem we encode it as a compact graph structure that we call planning encoding graph which gives us a detailed description of the topology of the planning problem by using this graph representation we examine an approximate retrieval procedure based on kernel functions that effectively match planning instances achieving extremely good performance in standard benchmark domains the experimental results point out the effect of the case base size and the importance of accurate matching functions for global system performance overall we show that oakplan is competitive with state of the art plan generation systems in terms of number of problems solved cpu time plan difference values and plan quality when cases similar to the current planning problem are available in the plan library

bayesian learning versus optimal learning abstract we consider the optimal performance that may be reached in the problem of learning the symmetry breaking direction of a cloud of p n points in a n dimensional space the performance is measured through the overlap ropt between the true symmetry breaking direction and the learnt one depending on the problem the learning curves ropt may present discontinuities we show that close to these bayesian learning is not optimal

a cure for variance inflation in high dimensional kernel principal component analysis small sample high dimensional principal component analysis pca suffers from variance inflation and lack of generalizability it has earlier been pointed out that a simple leave one out variance renormalization scheme can cure the problem in this paper we generalize the cure in two directions first we propose a computationally less intensive approximate leave one out estimator secondly we show that variance inflation is also present in kernel principal component analysis kpca and we provide a non parametric renormalization scheme which can quite efficiently restore generalizability in kpca as for pca our analysis also suggests a simplified approximate expression

optimal unsupervised learning we introduce an inferential approach to unsupervised learning which allows us to define an optimal learning strategy applying these ideas to a simple previously studied model we show that it is impossible to detect structure in data until a critical number of examples have been presented an effect which will be observed in all problems with certain underlying symmetries thereafter the advantage of optimal learning over previously studied learning algorithms depends critically upon the distribution of patterns optimal learning may be exponentially faster models with more subtle correlations are harder to analyse but in a simple limit of one such problem we calculate exactly the efficacy of an algorithm similar to some used in practice and compare it to that of the optimal prescription

finite sample approximation results for principal component analysis a matrix perturbation approach principal component analysis pca is a standard tool for dimensional reduction of a set of n observations samples each with p variables in this paper using a matrix perturbation approach we study the nonasymptotic relation between the eigenvalues and eigenvectors of pca computed on a finite sample of size n and those of the limiting population pca as n oo as in machine learning we present a finite sample theorem which holds with high probability for the closeness between the leading eigenvalue and eigenvector of sample pca and population pca under a spiked covariance model in addition we also consider the relation between finite sample pca and the asymptotic results in the joint limit p n with p n c we present a matrix perturbation view of the phase transition phenomenon and a simple linear algebra based derivation of the eigenvalue and eigenvector overlap in this asymptotic limit moreover our analysis also applies for finite p n where we show that although there is no sharp phase transition as in the infinite case either as a function of noise level or as a function of sample size n the eigenvector of sample pca may exhibit a sharp loss of tracking suddenly losing its relation to the true eigenvector of the population pca matrix this occurs due to a crossover between the eigenvalue due to the signal and the largest eigenvalue due to noise whose eigenvector points in a random direction

gradient descent learning in perceptrons a review of its possibilities we present a streamlined formalism which reduces the calculation of the generalization error for a perceptron trained on random examples generated by a teacher perceptron to a matter of simple algebra the method is valid whenever the student perceptron can be identified as the unique minimum of a specific cost function the asymptotic generalization error is calculated explicitly for a broad class of cost functions and a specific cost function is singled out that leads to a generalization error extremely close to the one of the bayes classifier

finite element theory for curved and twisted beams based on exact solutions for three dimensional solids part 1 beam concept and geometrically exact nonlinear formulation a geometrically exact and completely consistent finite element theory for curved and twisted beams is proposed it is based on the kinematical hypothesis generally formulated for large deformation and accounts for all kinds of deformation of a three dimensional solid translational and rotational displacements of the cross sections warping of their plane and distortion of their contours the principle of virtual work is applied in a straightforward manner to all non zero six components of the strain and stress tensors expressions are given for tangent matrices of elastic inertia and external forces and specific techniques for discretization and updating are developed for the analysis of beams in inertial and non inertial frames finally the numerical properties of the finite element models are demonstrated through examples c 1998 elsevier science s a all rights reserved

planning as satisfiability parallel plans and algorithms for plan search we address two aspects of constructing plans efficiently by means of satisfiability testing efficient encoding of the problem of existence of plans of a given number t of time points in the propositional logic and strategies for finding plans given these formulae for different values of t for the first problem we consider three semantics for plans with parallel operator application in order to make the search for plans more efficient the standard semantics requires that parallel operators are independent and can therefore be executed in any order we consider a more relaxed definition of parallel plans which was first proposed by dimopoulos et al as well as a normal form for parallel plans that requires every operator to be executed as early as possible we formalize the semantics of parallel plans emerging in this setting and present translations of these semantics into the propositional logic the sizes of the translations are asymptotically optimal each of the semantics is constructed in such a way that there is a plan following the semantics exactly when there is a sequential plan and moreover the existence of a parallel plan implies the existence of a sequential plan with as many operators as in the parallel one for the second problem we consider strategies based on testing the satisfiability of several formulae representing plans of n time steps for several values of n concurrently by several processes we show that big efficiency gains can be obtained in comparison to the standard strategy of sequentially testing the satisfiability of formulae for an increasing number of time steps

dilution in boolean perceptrons that learn from noisy examples we investigate the effect of dilution after learning on the generalization ability of single layer boolean perceptrons that learn from noisy examples we present a thorough comparison between the relative performances of several well known learning rules in particular we show that the effect of dilution is always deleterious and that the bayes algorithm always gives the best generalization performance