in search of an optimal dilution algorithm for feedforward networks in a recent paper the authors presented a dilution algorithm that yields a storage capacity per synapse alpha eff larger than 2 in this letter two new algorithms with even higher alpha eff values are introduced they study a one step dilution algorithm where the perceptron is used to select a fraction of the couplings to be removed for the remaining bonds the perceptron of optimal stability is relearned they further compare simulation data from an iterative version of the one step dilution algorithm with phase space volume calculations

on the generalization ability of diluted perceptrons a linearly separable boolean function is teamed by a diluted perceptron with optimal stability a different level of dilution is allowed for teacher and student perceptron the learning algorithms used were the optimal annealed dilution and hebbian dilution the generalization ability i e the probability to recognize a pattern which has not been learned before is calculated in replica symmetry

learning dynamics on different timescales the special character of certain degrees of freedom in two layered neural networks is investigated for on line learning of realizable rules our analysis shows that the dynamics of these degrees of freedom can be put on a faster timescale than those remaining with the profit of speeding up the overall adaptation process this is shown for two groups of degrees of freedom second layer weights and bias weights for the former case our analysis provides a theoretical explanation of phenomenological findings the resulting learning algorithm is compared with natural gradient descent in order to check whether the proposed scaling can be naturally derived from that type of learning rule

on line learning in the ising perceptron on line learning of both binary and continuous rules in an ising space is studied learning is achieved by using an artificial parameter a weight vector which is constrained to the surface of a hypersphere spherical constraint in the case of a binary rule the generalization error decays to zero super exponentially as exp c 2 where is the number of examples divided by n the size of the input vector and c 0 much faster learning is obtained in the case of continuous activation functions where the generalization error decays as exp e the number of steps required for perfect learning is estimated for both scenarios and compared with simulations

the topology of wireless communication this article studies the topological properties of wireless communication maps and their usability in algorithmic design we consider the sinr model which compares the received power of a signal at a receiver against the sum of strengths of other interfering signals plus background noise to describe the behavior of a multistation network we use the convenient representation of a reception map which partitions the plane into reception zones one per station and the complementary region of the plane where no station can be heard sinr diagrams have been studied in avin et al 2009 for the specific case where all stations use the same power it was shown that the reception zones are convex hence connected and fat and this was used to devise an efficient algorithm for the fundamental problem of point location here we consider the more general and common case where transmission energies are arbitrary or nonuniform under that setting the reception zones are not necessarily convex or even connected this poses the algorithmic challenge of designing efficient point location techniques for the nonuniform setting as well as the theoretical challenge of understanding the geometry of sinr diagrams e g the maximal number of connected components they might have our key result exhibits a striking contrast between d and d 1 dimensional maps for a network embedded in d dimensional space specifically it is shown that whereas the d dimensional map might be highly fractured drawing the map in one dimension higher heals the zones which become connected in fact hyperbolically connected we also provide bounds for the fatness of reception zones subsequently we consider algorithmic applications and propose a new variant of approximate point location

parallel strategy for optimal learning in perceptrons we developed a parallel strategy for learning optimally specific realizable rules by perceptrons in an online learning scenario our result is a generalization of the caticha kinouchi ck algorithm developed for learning a perceptron with a synaptic vector drawn from a uniform distribution over the n dimensional sphere so called the typical case our method outperforms the ck algorithm in almost all possible situations failing only in a denumerable set of cases the algorithm is optimal in the sense that it saturates bayesian bounds when it succeeds

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

dual diversified dynamical gaussian process latent variable model for video repairing in this paper we propose a dual diversified dynamical gaussian process latent variable model text d 3 gplvm to tackle the video repairing issue for preservation purposes videos have to be conserved on media however storing on media such as films and hard disks can suffer from unexpected data loss for instance physical damage so repairing of missing or damaged pixels is essential for better video maintenance most methods seek to fill in missing holes by synthesizing similar textures from local patches the neighboring pixels consecutive frames or the whole video however these can introduce incorrect contexts especially when the missing hole or number of damaged frames is large furthermore simple texture synthesis can introduce artifacts in undamaged and recovered areas to address aforementioned problems we introduce two diversity encouraging priors to both of inducing points and latent variables for considering the variety in existing videos in text d 3 gplvm the inducing points constitute a smaller subset of observed data while latent variables are a low dimensional representation of observed data since they have a strong correlation with the observed data it is essential that both of them can capture distinct aspects of and fully represent the observed data the dual diversity encouraging priors ensure that the trained inducing points and latent variables are more diverse and resistant for context aware and artifacts free based video repairing the defined objective function in our proposed model is initially not analytically tractable and must be solved by variational inference finally experimental testing results illustrate the robustness and effectiveness of our method for damaged video repairing

a polynomial regularity lemma for semialgebraic hypergraphs and its applications in geometry and property testing in this paper we prove several extremal results for geometrically defined hypergraphs in particular we establish an improved lower bound single exponentially decreasing in k on the best constant delta 0 such that the vertex classes p 1 ldots p k of every k partite k uniform semialgebraic hypergraph h p 1 cup cdots cup p k e with e ge varepsilon pi j 1 k p i have for 1 leq i leq k delta p i element subsets p i subseteq p i satisfying p 1 times cdots times p k subseteq e the best previously known lower bound on delta due to bukh and hubard decreased triple exponentially fast in k we give three geometric applications of our results in particular we establish the following strengthening of the so called same type lemma of barany and valtr any disjoint finite sets p 1 ldots p k subset mathbb r d k d have for 1 leq i leq k subsets p i of size at least 2 o d 3k log k p i with the property that every k tuple formed by taking on

the rosenblatt bayesian algorithm learning in a nonstationary environment in this letter we study online learning in neural networks nns obtained by approximating bayesian learning the approach is applied to gibbs learning with the rosenblatt potential in a nonstationary environment the online scheme is obtained by the minimization maximization of the kullback leibler divergence cross entropy between the true posterior distribution and the parameterized one the complexity of the learning algorithm is further decreased by projecting the posterior onto a gaussian distribution and imposing a spherical covariance matrix we study in detail the particular case of learning linearly separable rules in the case of a fixed rule we observe an asymptotic generalization error egpropalpha 1 for both the spherical and the full covariance matrix approximations however in the case of drifting rule only the full covariance matrix algorithm shows a good performance this good performance is indeed a surprise since the algorithm is obtained by projecting without the benefit of the extra information on drifting

learning by dilution in a neural network a perceptron with n random weights can store of the order of n patterns by removing a fraction of the weights without changing their strengths the critical storage capacity as a function of the concentration of the remaining bonds for random outputs and for outputs given by a teacher perceptron is calculated a simple hebb like dilution algorithm is presented which in the teacher case reaches the optimal generalization ability

a dilution algorithm for neural networks a dilution algorithm to enlarge the storage capacity per synapse alpha eff of neural networks is proposed the algorithm is a hybrid method where hebb s rule is used to select a fraction of couplings to be removed afterwards the perceptron of optimal stability for the remaining couplings is learned the authors present an analytical calculation and the results of the numerical simulations in comparison with the fully connected or the randomly diluted perceptron the effective storage capacity alpha eff is remarkably enlarged

expectation propagation with factorizing distributions a gaussian approximation and performance results for simple models we discuss the expectation propagation ep algorithm for approximate bayesian inference using a factorizing posterior approximation for neural network models we use a central limit theorem argument to make ep tractable when the number of parameters is large for two types of models we show that ep can achieve optimal generalization performance when data are drawn from a simple distribution

learning a multiple kernel similarity metric for kinship verification abstract kinship verification kv has recently caught much attention in the computer vision community due to its potential applications ranging from missing children search to social media analysis most of the related work focuses either on developing hand crafted feature representations to describe the faces or on learning the mahalanobis distance metric to measure the similarity between facial images instead in this paper we propose a novel multiple kernel similarity metric mksm in which different from the mahalanobis metric the similarity computation is essentially based on an implicit nonlinear feature transformation the overall mksm is a weighted combination of basic similarities and therefore possesses the capacity for feature fusion the basic similarities are derived from base kernels and local features and the weights are obtained by solving a constrained linear programming lp problem that originates from a large margin lm criterion particularly the lm criterion not only guarantees the generalization on unseen samples when the training set is small but also leads to sparsity in the weight vector which in turn boosts the efficiency at the prediction stage extensive experiments on four publicly available datasets demonstrate the effectiveness of the proposed method

on approximations and incidence in cylindrical algebraic decompositions let p subset mathbb z x 1 cdots x r be a finite set this paper describes and analyzes a variant of the algorithm of collins and others for decomposing mathbb r r into semi algebraic cells so that the value of each f in p has constant sign positive negative or zero on the points of each cell the version here has several advantages 1 the boundary of each cell is a disjoint union of lower dimensional cells for each bounded cell alpha the pair bar alpha alpha is homeomorphic to a closed ball and its interior 2 an algorithm is presented which for fixed r computes incidence of cells in polynomial time 3 a priori estimates of the accuracy of approximations of roots of polynomials required in order to determine the combinatorial structure of the cell complex are given this avoids computation in algebraic number fields

general tensor discriminant analysis and gabor features for gait recognition traditional image representations are not suited to conventional classification methods such as the linear discriminant analysis lda because of the undersample problem usp the dimensionality of the feature space is much higher than the number of training samples motivated by the successes of the two dimensional lda 2dlda for face recognition we develop a general tensor discriminant analysis gtda as a preprocessing step for lda the benefits of gtda compared with existing preprocessing methods such as the principal components analysis pca and 2dlda include the following 1 the usp is reduced in subsequent classification by for example lda 2 the discriminative information in the training tensors is preserved and 3 gtda provides stable recognition rates because the alternating projection optimization algorithm to obtain a solution of gtda converges whereas that of 2dlda does not we use human gait recognition to validate the proposed gtda the averaged gait images are utilized for gait representation given the popularity of gabor function based image decompositions for image understanding and object recognition we develop three different gabor function based image representations 1 gabord is the sum of gabor filter responses over directions 2 gabors is the sum of gabor filter responses over scales and 3 gaborsd is the sum of gabor filter responses over scales and directions the gabord gabors and gaborsd representations are applied to the problem of recognizing people from their averaged gait images a large number of experiments were carried out to evaluate the effectiveness recognition rate of gait recognition based on first obtaining a gabor gabord gabors or gaborsd image representation then using gdta to extract features and finally using lda for classification the proposed methods achieved good performance for gait recognition based on image sequences from the university of south florida usf humanid database experimental comparisons are made with nine state of the art classification methods in gait recognition

almost tight upper bounds for vertical decompositions in four dimensions we show that the complexity of the vertical decomposition of an arrangement of n fixed degree algebraic surfaces or surface patches in four dimensions is o n4 e for any e 0 this improves the best previously known upper bound for this problem by a near linear factor and settles a major problem in the theory of arrangements of surfaces open since 1989 the new bound can be extended to higher dimensions yielding the bound o n2d 4 e for any e 0 on the complexity of vertical decompositions in dimensions d 4 we also describe the immediate algorithmic applications of these results which include improved algorithms for point location range searching ray shooting robot motion planning and some geometric optimization problems

discriminative elastic net regularized linear regression in this paper we aim at learning compact and discriminative linear regression models linear regression has been widely used in different problems however most of the existing linear regression methods exploit the conventional zero one matrix as the regression targets which greatly narrows the flexibility of the regression model another major limitation of these methods is that the learned projection matrix fails to precisely project the image features to the target space due to their weak discriminative capability to this end we present an elastic net regularized linear regression enlr framework and develop two robust linear regression models which possess the following special characteristics first our methods exploit two particular strategies to enlarge the margins of different classes by relaxing the strict binary targets into a more feasible variable matrix second a robust elastic net regularization of singular values is introduced to enhance the compactness and effectiveness of the learned projection matrix third the resulting optimization problem of enlr has a closed form solution in each iteration which can be solved efficiently finally rather than directly exploiting the projection matrix for recognition our methods employ the transformed features as the new discriminate representations to make final image classification compared with the traditional linear regression model and some of its variants our method is much more accurate in image classification extensive experiments conducted on publicly available data sets well demonstrate that the proposed framework can outperform the state of the art methods the matlab codes of our methods can be available at http www yongxu org lunwen html

performance of the bayesian online algorithm for the perceptron in this letter we derive continuum equations for the generalization error of the bayesian online algorithm bona for the one layer perceptron with a spherical covariance matrix using the rosenblatt potential and show by numerical calculations that the asymptotic performance of the algorithm is the same as the one for the optimal algorithm found by means of variational methods with the added advantage that the bona does not use any inaccessible information during learning

optimal pruning in neural networks we study pruning strategies in simple perceptrons subjected to supervised learning our analytical results obtained through the statistical mechanics approach to learning theory are independent of the learning algorithm used in the training process we calculate the post training distribution mathcal p j of synaptic weights which depends only on the overlap ensuremath rho 0 achieved by the learning algorithm before pruning and the fraction ensuremath kappa of relevant weights in the teacher network from this distribution we calculate the optimal pruning strategy for deleting small weights the optimal pruning threshold grows from zero as ensuremath theta mathrm opt ensuremath rho 0 ensuremath kappa ensuremath propto ensuremath rho 0 ensuremath ensuremath rho c ensuremath kappa 1 2 above some critical value ensuremath rho c ensuremath kappa thus the elimination of weak synapses enhances the network performance only after a critical learning period possible implications for biological pruning phenomena are discussed

sinr diagrams convexity and its applications in wireless networks the rules governing the availability and quality of connections in a wireless network are described by physical models such as the signal to interference noise ratio sinr model for a collection of simultaneously transmitting stations in the plane it is possible to identify a reception zone for each station consisting of the points where its transmission is received correctly the resulting sinr diagram partitions the plane into a reception zone per station and the remaining plane where no station can be heard sinr diagrams appear to be fundamental to understanding the behavior of wireless networks and may play a key role in the development of suitable algorithms for such networks analogous perhaps to the role played by voronoi diagrams in the study of proximity queries and related issues in computational geometry so far however the properties of sinr diagrams have not been studied systematically and most algorithmic studies in wireless networking rely on simplified graph based models such as the unit disk graph udg model which conveniently abstract away interference related complications and make it easier to handle algorithmic issues but consequently fail to capture accurately some important aspects of wireless networks this article focuses on obtaining some basic understanding of sinr diagrams their properties and their usability in algorithmic applications specifically we have shown that assuming uniform power transmissions the reception zones are convex and relatively well rounded these results are then used to develop an efficient approximation algorithm for a fundamental point location problem in wireless networks

local feature discriminant projection in this paper we propose a novel subspace learning algorithm called local feature discriminant projection lfdp for supervised dimensionality reduction of local features lfdp is able to efficiently seek a subspace to improve the discriminability of local features for classification we make three novel contributions first the proposed lfdp is a general supervised subspace learning algorithm which provides an efficient way for dimensionality reduction of large scale local feature descriptors second we introduce the differential scatter discriminant criterion dsdc to the subspace learning of local feature descriptors which avoids the matrix singularity problem third we propose a generalized orthogonalization method to impose on projections leading to a more compact and less redundant subspace extensive experimental validation on three benchmark datasets including uiuc sports scene 15 and mit indoor demonstrates that the proposed lfdp outperforms other dimensionality reduction methods and achieves state of the art performance for image classification

learning in feedforward layered networks the tiling algorithm the authors propose a new algorithm which builds a feedforward layered network in order to learn any boolean function of n boolean units the number of layers and the number of hidden units in each layer are not prescribed in advance they are outputs of the algorithm it is an algorithm for growth of the network which adds layers and units inside a layer at will until convergence the convergence is guaranteed and numerical tests of this strategy look promising

hetero manifold regularisation for cross modal hashing recently cross modal search has attracted considerable attention but remains a very challenging task because of the integration complexity and heterogeneity of the multi modal data to address both challenges in this paper we propose a novel method termed hetero manifold regularisation hmr to supervise the learning of hash functions for efficient cross modal search a hetero manifold integrates multiple sub manifolds defined by homogeneous data with the help of cross modal supervision information taking advantages of the hetero manifold the similarity between each pair of heterogeneous data could be naturally measured by three order random walks on this hetero manifold furthermore a novel cumulative distance inequality defined on the hetero manifold is introduced to avoid the computational difficulty induced by the discreteness of hash codes by using the inequality cross modal hashing is transformed into a problem of hetero manifold regularised support vector learning therefore the performance of cross modal search can be significantly improved by seamlessly combining the integrated information of the hetero manifold and the strong generalisation of the support vector machine comprehensive experiments show that the proposed hmr achieve advantageous results over the state of the art methods in several challenging cross modal tasks

storage capacity of ultrametric committee machines the problem of computing the storage capacity of a feed forward network with l hidden layers n inputs and k units in the first hidden layer is analyzed using techniques from statistical mechanics we found that the storage capacity strongly depends on the network architecture c log k 1 1 2l and that the number of units k limits the number of possible hidden layers l through the relationship 2l 1 2log k

capacity of diluted multi state neural networks the optimal storage capacity is studied for diluted networks with multi state neurons and continuous respectively discrete couplings within the replica symmetric gardner approach the gardner derrida line the de almeida thouless line and the zero entropy line are compared and the validity of the replica symmetric approximation is discussed in detail the distribution of the synaptic couplings is determined the results are analysed in terms of the number of states of the neuron the distribution of the stored patterns the amount of dilution and the number of discrete values for the couplings

selecting examples for perceptrons a technique is introduced for analysing the learning process of perceptrons which continually select their own examples the most efficient training algorithm yet devised the authors predict a 42 reduction in the number of examples wasted in training an ising perceptron compared to the case in which examples are random optimally stable spherical perceptrons may also be taught significantly more efficiently and their results compare well with an existing numerical simulation

learning strategy for the binary perceptron pursuing the work of penney and sherrington we determine the optimal continuous weight perceptron which on clipping correctly predicts the largest number of weights for the binary perceptron with maximum stability we calculate the fraction of correctly predicted binary weights when only the continuous weights stronger than a certain threshold are clipped we finally carry out simulations for a perceptron with 50 weights to test the practicability of different learning strategies

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

on line learning in a discrete state space on line learning of a rule given by an n dimensional ising perceptron is considered for the case when the student is constrained to take values in a discrete state space of size for l 2 no on line algorithm can achieve a finite overlap with the teacher in the thermodynamic limit however if l is on the order of hebbian learning does achieve a finite overlap

cylindrical algebraic decomposition ii an adjacency algorithm for the plane in part i of the present paper we defined cylindrical algebraic decompositions cad s and described an algorithm for cad construction in part ii we give an algorithm that provides information about the topological structure of a cad of the plane informally two disjoint cells in e r r 1 are adjacent if they touch each other formally they are adjacent if their union is connected in a picture of a cad eg fig 2 of part i it is obvious to the eye which pairs of cells are adjacent however the cad algorithm of part i does not actually produce this information nor did the original version of that algorithm in collins 1975

zero shot visual recognition via bidirectional latent embedding zero shot learning for visual recognition e g object and action recognition has recently attracted a lot of attention however it still remains challenging in bridging the semantic gap between visual features and their underlying semantics and transferring knowledge to semantic categories unseen during learning unlike most of the existing zero shot visual recognition methods we propose a stagewise bidirectional latent embedding framework of two subsequent learning stages for zero shot visual recognition in the bottom up stage a latent embedding space is first created by exploring the topological and labeling information underlying training data of known classes via a proper supervised subspace learning algorithm and the latent embedding of training data are used to form landmarks that guide embedding semantics underlying unseen classes into this learned latent space in the top down stage semantic representations of unseen class labels in a given label vocabulary are then embedded to the same latent space to preserve the semantic relatedness between all different classes via our proposed semi supervised sammon mapping with the guidance of landmarks thus the resultant latent embedding space allows for predicting the label of a test instance with a simple nearest neighbor rule to evaluate the effectiveness of the proposed framework we have conducted extensive experiments on four benchmark datasets in object and action recognition i e awa cub 200 2011 ucf101 and hmdb51 the experimental results under comparative studies demonstrate that our proposed approach yields the state of the art performance under inductive and transductive settings

non deterministic learning dynamics in large neural networks due to structural data bias we study the dynamics of on line learning in large n perceptrons for the case of training sets with a structural n0 bias of the input vectors by deriving exact and closed macroscopic dynamical laws using non equilibrium statistical mechanical tools in sharp contrast to the more conventional theories developed for homogeneously distributed or only weakly biased data these laws are found to describe a non trivial and persistently non deterministic macroscopic evolution and a generalization error which retains both stochastic and sample to sample fluctuations even for infinitely large networks furthermore for the standard error correcting microscopic algorithms such as the perceptron learning rule one obtains learning curves with distinct bias induced phases our theoretical predictions find excellent confirmation in numerical simulations

dynamics of batch training in a perceptron early stopping and weight decay are studied in a linear perceptron using a new simplified approach to the dynamics in the thermodynamical limit the approach is directly deduced from the gradient descent weight update it allows an exact description of the dynamics of the batch training process the results are compared with a recent study of early stopping and weight decay based on the equilibrium statistical mechanics approach it is shown that the equilibrium results for early stopping are good approximations and are exact for weight decay furthermore in the dynamical approach it is possible to determine the necessary number of training steps to fulfil certain termination conditions it can be shown that asymptotically i e if the number of examples is large only two batch steps are required to reach the optimal convergence if the learning rate is optimally chosen

dilution in boolean perceptrons that learn from noisy examples we investigate the effect of dilution after learning on the generalization ability of single layer boolean perceptrons that learn from noisy examples we present a thorough comparison between the relative performances of several well known learning rules in particular we show that the effect of dilution is always deleterious and that the bayes algorithm always gives the best generalization performance

multiple kernel learning algorithms in recent years several methods have been proposed to combine multiple kernels instead of using a single one these different kernels may correspond to using different notions of similarity or may be using information coming from multiple sources different representations or different feature subsets in trying to organize and highlight the similarities and differences between them we give a taxonomy of and review several multiple kernel learning algorithms we perform experiments on real data sets for better illustration and comparison of existing algorithms we see that though there may not be large differences in terms of accuracy there is difference between them in complexity as given by the number of stored support vectors the sparsity of the solution as given by the number of used kernels and training time complexity we see that overall using multiple kernels instead of a single one is useful and believe that combining kernels in a nonlinear or data dependent way seems more promising than linear combination in fusing information provided by simple linear kernels whereas linear methods are more reasonable when combining complex gaussian kernels

statistical mechanics of the knapsack problem the knapsack problem is an np complete combinatorial optimization problem with inequality constraints using the replica method of statistical physics we study the space of its solutions for a large problem size it turns out that this problem is closely related to the theory of the binary perceptron

neural networks for optimization problems with inequality constraints the knapsack problem a strategy for finding approximate solutions to discrete optimization problems with inequality constraints using mean field neural networks is presented the constraints x d 0 are encoded by x x terms in the energy function a careful treatment of the mean field approximation for the self coupling parts of the energy is crucial and results in an essentially parameter free algorithm this methodology is extensively tested on the knapsack problem of size up to 103 items the algorithm scales like nm for problems with n items and m constraints comparisons are made with an exact branch and bound algorithm when this is computationally possible n d 30 the quality of the neural network solutions consistently lies above 95 of the optimal ones at a significantly lower cpu expense for the larger problem sizes the algorithm is compared with simulated annealing and a modified linear programming approach for nonhomogeneous problems these produce good solutions whereas for the more difficult homogeneous problems the neural approach is a winner with respect to solution quality and or cpu time consumption the approach is of course also applicable to other problems of similar structure like set covering

a randomized algorithm for closest point queries an algorithm for closest point queries is given the problem is this given a set s of n points in d dimensional space build a data structure so that given an arbitrary query point p a closest point in s to p can be found quickly the measure of distance is the euclidean norm this is sometimes called the post office problem the new data structure will be termed an rpo tree from randomized post office the expected time required to build an rpo tree is o n lceil d 2 rceil 1 epsilon for any fixed epsilon 0 and a query can be answered in o log n worst case time an rpo tree requires o n lceil d 2 rceil 1 epsilon space in the worst case the constant factors in these bounds depend on d and epsilon the bounds are average case due to the randomization employed by the algorithm and hold for any set of input points this result approaches the omega n lceil d 2 rceil worst case time required for any algorithm that constructs the voronoi

critical behaviour of a surface reaction model with infinitely many absorbing states in a recent letter yaldram et al 1993 studied the critical behaviour of a simple lattice gas model of the co no catalytic reaction the model exhibits a second order non equilibrium phase transition from an active state into one out of infinitely many absorbing states estimates for the critical exponent beta suggested that the model belongs to a new universality class the results reported in this article contradict this notion as estimates for various critical exponents show that the model belongs to the universality class of directed percolation

learning and retrieval in attractor neural networks above saturation the authors investigate neural networks in the range of parameters when the ground state energy is positive namely when a synaptic matrix which satisfies all the desired constraints cannot be found by the learning algorithm in particular they calculate the typical distribution functions of local stabilities obtained for a number of algorithms in this region these functions are used to investigate the retrieval properties as reflected by the size of the basins of attraction this is done analytically in sparsely connected networks and numerically in fully connected networks the main conclusion is that the retrieval behaviour of attractor neural networks can be improved by learning above saturation

transient dynamics of on line learning in two layered neural networks the dynamics of on line learning in neural networks with continuous units is dominated by plateaux in the time dependence of the generalization error using tools from statistical mechanics we show for a soft committee machine the existence of several fixed points of the dynamics of learning that give rise to complicated behaviour such as cascade like runs through different plateaux with a decreasing value of the corresponding generalization error we find learning rate dependent phenomena such as splitting and disappearing of fixed points of the equations of motion the dependence of plateau lengths on the initial conditions is described analytically and simulations confirm the results

content addressability and learning in neural networks the content addressability of patterns stored in ising spin neural network models with symmetric interactions is studied numerical results from simulations on the icl distributed array processor dap involving systems with up to 2048 neurons are presented behaviour consistent with finite size scaling characteristic of a first order phase transition is shown to be exhibited by the basins of attraction of the stored patterns both in the case of the hopfield model and for systems using a local iterative learning algorithm designed to optimise the basins of attraction estimates are obtained for the critical minimum overlaps which an input pattern must have with a stored pattern in order to successfully retrieve it

optimally adapted attractor neural networks in the presence of noise by adapting an attractor neural network to an appropriate training overlap the authors optimize its attractor overlap and subsequently the storage capacity when retrieval noise temperature is present in the system the training overlap is determined self consistently by the optimal attractor overlap the phase diagram of the optimal attractor overlap in the temperature storage space is found a novel co existence phase of strong and weak retrievers is present the maximum storage capacity deviates from the storage capacity of the maximally stable network on increasing temperature and in the high temperature regime t or 0 38 for gaussian noise the hopfield network yields the maximum storage capacity this analysis demonstrates the principles of specialization and adaptation in neural networks

the failure distribution in percolation models of breakdown the probability of finding a largest defect cluster of size n in a percolation network is calculated analytically using a new distribution function scaling equation from this result and the stress voltage enhancement at the tip of the most critical defect in the network the probability of failure in percolation models of breakdown is calculated for defect fractions less than the percolation point this distribution is found to be of the form exponential of an exponential numerical simulations on the two dimensional random fuse network confirm the new distribution function and convincingly distinguish between it and the weibull 1951 form most often used in the fitting of breakdown data

lower bounds on generalization errors for drifting rules the problem of generalization by single layer perceptrons is studied in the case of time dependent rules lower bounds for the generalization errors within the single presentation of examples case are obtained for randomly drifting rules these bounds suggest a learning algorithm which uses knowledge of the error itself since this error is not readily available it has to be estimated through a mechanism of self evaluation the capacity of incorporating recency information into the error estimate is highly desirable the mechanism proposed has the advantage beyond good performance of being self adaptive in the sense that it adjusts to changes in the unknown drift rate of the rule the performance of the rule is also studied for sudden changes in an attempt to mimic the so called wisconsin test

multi view intact space learning it is practical to assume that an individual view is unlikely to be sufficient for effective multi view learning therefore integration of multi view information is both valuable and necessary in this paper we propose the multi view intact space learning misl algorithm which integrates the encoded complementary information in multiple views to discover a latent intact representation of the data even though each view on its own is insufficient we show theoretically that by combing multiple views we can obtain abundant information for latent intact space learning employing the cauchy loss a technique used in statistical learning as the error measurement strengthens robustness to outliers we propose a new definition of multi view stability and then derive the generalization error bound based on multi view stability and rademacher complexity and show that the complementarity between multiple views is beneficial for the stability and generalization misl is efficiently optimized using a novel iteratively reweight residuals irr technique whose convergence is theoretically analyzed experiments on synthetic data and real world datasets demonstrate that misl is an effective and promising algorithm for practical applications

slowly evolving connectivity in recurrent neural networks i the extreme dilution regime we study extremely diluted spin models of neural networks in which the connectivity evolves in time although adiabatically slowly compared to the neurons according to stochastic equations which on average aim to reduce frustration the fast neurons and slow connectivity variables equilibrate separately but at different temperatures our model is exactly solvable in equilibrium we obtain phase diagrams upon making the condensed ansatz i e recall of one pattern these show that as the connectivity temperature is lowered the volume of the retrieval phase diverges and the fraction of mis aligned spins is reduced still one always retains a region in the retrieval phase where recall states other than the one corresponding to the condensed pattern are locally stable so the associative memory character of our model is preserved

convex partitions of polyhedra a lower bound and worst case optimal algorithm the problem of partitioning a polyhedron into a minimum number of convex pieces is known to be np hard we establish here a quadratic lower bound on the complexity of this problem and we describe an algorithm that produces a number of convex parts within a constant factor of optimal in the worst case the algorithm is linear in the size of the polyhedron and cubic in the number of reflex angles since in most applications areas the former quantity greatly exceeds the latter the algorithm is viable in practice

optimal basins of attraction in randomly sparse neural network models the size of the basin of attraction for randomly sparse neural networks with optimal interactions is calculated for all values of the storage ratio alpha p c 2 where p is the number of random uncorrelated patterns and c is the connectivity the basin of attraction is finite while for alpha 0 42 the basin of attraction is almost 100

on line learning in parity machines the optimal algorithm for on line learning in the tree k parity machine is studied we introduce a set of recursion relations for the relevant probability distributions which permit study of the general k case the generalization error curve is determined and shown to decay to zero for large as even in the presence of noise there is no critical noise level the dynamics of on line learning is studied analytically near the origin in the absence of previous knowledge the learning dynamics has a fixed point at previous knowledge is needed in at least k 1 branches for the learning to take place

a gaussian scenario for unsupervised learning we consider random patterns on the n sphere which are uniformly distributed with the exception of a single symmetry breaking orientation along which they are gaussian distributed the unsupervised recognition of this orientation by different learning rules is studied in the large n limit using the replica method the model is simple enough to be analytically tractable and rich enough to exhibit most of the phenomena observed with other pattern distributions a learning algorithm based on the minimization of a cost function is identified which reaches the upper theoretical limit imposed by the optimal bayes learning scenario an implementation of this algorithm is proposed and tested numerically

the dynamics of on line principal component analysis the learning dynamics of an on line algorithm for principal component analysis is described exactly in the thermodynamic limit by means of coupled ordinary differential equations for a set of order parameters it is demonstrated that learning is delayed significantly because existing symmetries among student vectors have to be broken a closely related effect is the perfect or partial loss of initial knowledge in the course of learning the analysis shows that different learning rates for the student vectors improve the performance of the algorithm drastically

learning drifting concepts with neural networks the learning of time dependent concepts with a neural network is studied analytically and numerically the linearly separable target rule is represented by an n vector whose time dependence is modelled by a random or deterministic drift process a single layer network is trained online using different hebb like algorithms training is based on examples which are chosen randomly and according to a query strategy the evolution of the generalization error can be calculated exactly in the thermodynamic limit n to infinity the rule is never learnt perfectly but can be tracked within a certain error margin the generalization performance of various learning rules is compared and simulations confirm the analytic results

large margin multi viewinformation bottleneck in this paper we extend the theory of the information bottleneck ib to learning from examples represented by multi view features we formulate the problem as one of encoding a communication system with multiple senders each of which represents one view of the data based on the precise components filtered out from multiple information sources through a bottleneck a margin maximization approach is then used to strengthen the discrimination of the encoder by improving the code distance within the frame of coding theory the resulting algorithm therefore inherits all the merits of the ib principle and coding theory it has two distinct advantages over existing algorithms namely that our method finds a tradeoff between the accuracy and complexity of the multi view model and that the encoded multi view data retains sufficient discrimination for classification we also derive the robustness and generalization error bound of the proposed algorithm and reveal the specific properties of multi view learning first the complementarity of multi view features guarantees the robustness of the algorithm second the consensus of multi view features reduces the empirical rademacher complexity of the objective function enhances the accuracy of the solution and improves the generalization error bound of the algorithm the resulting objective function is solved efficiently using the alternating direction method experimental results on annotation classification and recognition tasks demonstrate that the proposed algorithm is promising for practical applications

biased learning in boolean perceptrons abstract the generalization ability of hebbian boolean perceptrons can be improved by a kind of feedback mechanism in which the student net judges the difficulty of a new example from its previous experience it is shown that by giving a higher weight to the hard examples both generalization and learning abilities can be increased analytical as well as numerical results are presented for both cases where the examples are drawn at random or selected in an intelligent form

duplex metric learning for image set classification image set classification has attracted much attention because of its broad applications despite the success made so far the problems of intra class diversity and inter class similarity still remain two major challenges to explore a possible solution to these challenges this paper proposes a novel approach termed duplex metric learning dml for image set classification the proposed dml consists of two progressive metric learning stages with different objectives used for feature learning and image classification respectively the metric learning regularization is not only used to learn powerful feature representations but also well explored to train an effective classifier at the first stage we first train a discriminative stacked autoencoder dsae by layer wisely imposing a metric learning regularization term on the neurons in the hidden layers and meanwhile minimizing the reconstruction error to obtain new feature mappings in which similar samples are mapped closely to each other and dissimilar samples are mapped farther apart at the second stage we discriminatively train a classifier and simultaneously fine tune the dsae by optimizing a new objective function which consists of a classification error term and a metric learning regularization term finally two simple voting strategies are devised for image set classification based on the learnt classifier in the experiments we extensively evaluate the proposed framework for the tasks of face recognition object recognition and face verification on several commonly used data sets and state of the art results are achieved in comparison with existing methods

semialgebraic range reporting and emptiness searching with applications in a typical range emptiness searching resp reporting problem we are given a set p of n points in mathbb r d and we wish to preprocess it into a data structure that supports efficient range emptiness resp reporting queries in which we specify a range sigma which in general is a semialgebraic set in mathbb r d of constant description complexity and we wish to determine whether p cap sigma emptyset or to report all the points in p cap sigma range emptiness searching and reporting arise in many applications and have been treated by matousek comput geom theory appl 2 1992 pp 169 186 in the special case where the ranges are half spaces bounded by hyperplanes as shown in matousek s work the two problems are closely related and they have solutions for the case of half spaces with similar performance bounds in this paper we extend the analysis to general semialgebraic ranges and show how to adapt matousek s technique without the need to linearize the ranges into a higher dimensional space this yields more efficient solutions to several useful problems and we demonstrate the new technique in four applications with the following results i an algorithm for ray shooting amid balls in mathbb r 3 which uses o n storage and o n preprocessing we use the notation o n gamma to mean an upper bound of the form c varepsilon n gamma varepsilon which holds for any varepsilon 0 where c varepsilon is a constant that depends on varepsilon and answers a query in o n 2 3 time improving the previous bound of o n 3 4 ii an algorithm that preprocesses in o n time a set p of n points in mathbb r 3 into a data structure with o n storage so that for any query line ell or for that matter any simply shaped convex set the point of p farthest from ell can be computed in o n 1 2 time this in turn yields an algorithm that computes the largest area triangle spanned by p in time o n 26 11 as well as nontrivial algorithms for computing the largest perimeter or largest height triangle spanned by p iii an algorithm that preprocesses in o n time a set p of n points in mathbb r 2 into a data structure with o n storage so that for any query alpha fat triangle delta we can determine in o 1 time whether delta cap p is empty alternatively we can report in o 1 o k time the points of delta cap p where k delta cap p iv an algorithm that preprocesses in o n time a set p of n points in mathbb r 2 into a data structure with o n storage so that given any query semidisk c or a circular cap larger than a semidisk we can determine in o 1 time whether c cap p is empty or report the k points in c cap p in o 1 o k time adapting the recent techniques of b aronov and s har peled siam j comput 38 2008 pp 899 921 b aronov s har peled and m sharir on approximate halfspace range counting and relative epsilon approximations in proceedings of the 23rd acm symposium comput geom 2007 pp 327 336 b aronov and m sharir siam j comput 39 2010 pp 2704 2725 we can turn our solutions into efficient algorithms for approximate range counting with small relative error for the cases mentioned above our technique is closely related to the notions of nearest or farthest neighbor generalized voronoi diagrams and of the union or intersection of geometric objects where sharper bounds on the combinatorial complexity of decompositions of complements of these structures yield faster range emptiness searching or reporting algorithms

playing billiards in version space a ray tracing method inspired by ergodic billiards is used to estimate the theoretically best decision rule for a given set of linear separable examples for randomly distributed examples the billiard estimate of the single perceptron with best average generalization probability agrees with known analytic results while for real life classification problems the generalization probability is consistently enhanced when compared to the maximal stability perceptron

partially connected models of neural networks a partially connected hopfield neural network model is studied under the restriction that w the ratio of connections per site to the size of the system remains finite as the size n to infinity with the connection structure at each site being the same the replica symmetric mean field theory equations for the order parameters are derived the zero temperature forms of these equations are then solved numerically for a few different local connectivity architectures showing phase transitions at different critical storage ratios alpha c where the states which the authors are trying to store in the network become discontinuously unstable they show that the information capacity per connection improves for partially connected systems

dense trajectories and motion boundary descriptors for action recognition this paper introduces a video representation based on dense trajectories and motion boundary descriptors trajectories capture the local motion information of the video a dense representation guarantees a good coverage of foreground motion as well as of the surrounding context a state of the art optical flow algorithm enables a robust and efficient extraction of dense trajectories as descriptors we extract features aligned with the trajectories to characterize shape point coordinates appearance histograms of oriented gradients and motion histograms of optical flow additionally we introduce a descriptor based on motion boundary histograms mbh which rely on differential optical flow the mbh descriptor shows to consistently outperform other state of the art descriptors in particular on real world videos that contain a significant amount of camera motion we evaluate our video representation in the context of action classification on nine datasets namely kth youtube hollywood2 ucf sports ixmas uiuc olympic sports ucf50 and hmdb51 on all datasets our approach outperforms current state of the art results

optimal point location in a monotone subdivision point location often known in graphics as hit detection is one of the fundamental problems of computational geometry in a point location query we want to identify which of a given collection of geometric objects contains a particular point let mathcal s denote a subdivision of the euclidean plane into monotone regions by a straight line graph of m edges in this paper we exhibit a substantial refinement of the technique of lee and preparata siam j comput 6 1977 pp 594 606 for locating a point in mathcal s based on separating chains the new data structure called a layered dag can be built in o m time uses o m storage and makes possible point location in o log m time unlike previous structures that attain these optimal bounds the layered dag can be implemented in a simple and practical way and is extensible to subdivisions with edges more general than straight line segments

supervised learning with restricted training sets a generating functional analysis we study the dynamics of supervised on line learning of realizable tasks in feed forward neural networks we focus on the regime where the number of examples used for training is proportional to the number of input channels n using generating functional techniques from spin glass theory we are able to average over the composition of the training set and transform the problem for n to an effective single pattern system described completely by the student autocovariance the student teacher overlap and the student response function with exact closed equations our method applies to arbitrary learning rules i e not necessarily of a gradient descent type the resulting exact macroscopic dynamical equations can be integrated without finite size effects up to any degree of accuracy but their main value is in providing an exact and simple starting point for analytical approximation schemes finally we show how in the region of absent anomalous response and using the hypothesis that as in detailed balance systems the short time part of the various operators can be transformed away one can describe the stationary state of the network succesfully by a set of coupled equations involving only four scalar order parameters

kernelized multiview projection for robust action recognition conventional action recognition algorithms adopt a single type of feature or a simple concatenation of multiple features in this paper we propose to better fuse and embed different feature representations for action recognition using a novel spectral coding algorithm called kernelized multiview projection kmp computing the kernel matrices from different features views via time sequential distance learning kmp can encode different features with different weights to achieve a low dimensional and semantically meaningful subspace where the distribution of each view is sufficiently smooth and discriminative more crucially kmp is linear for the reproducing kernel hilbert space which allows it to be competent for various practical applications we demonstrate kmp s performance for action recognition on five popular action datasets and the results are consistently superior to state of the art techniques

generalization learning in a perceptron with binary synapses we consider the generalization problem for a perceptron with binary synapses implementing the stochastic belief propagation inspired sbpi learning algorithm which we proposed earlier and perform a mean field calculation to obtain a differential equation which describes the behaviour of the device in the limit of a large number of synapses n we show that the solving time of sbpi is of order n sqrt log n while the similar well known clipped perceptron cp algorithm does not converge to a solution at all in the time frame we considered the analysis gives some insight into the ongoing process and shows that in this context the sbpi algorithm is equivalent to a new simpler algorithm which only differs from the cp algorithm by the addition of a stochastic unsupervised meta plastic reinforcement process whose rate of application must be less than sqrt 2 pi n for the learning to be achieved effectively the analytical results are confirmed by simulations

learning in neural networks by eliminating frustrated bonds the authors study neural network models in which the initial interaction matrix elements are drawn from an arbitrary probability distribution and in which patterns are subsequently stored by eliminating the frustrated bonds generalizing a proposal by kinzel 1985 they show that the optimal choice for the a priori distribution corresponds to choosing uniform ferromagnetic initial interactions for the optimal model they study analytically the dynamical behaviour the equilibrium properties the sizes of the domains of attraction and a number of information theoretical performance measures

a singly exponential stratification scheme for real semi algebraic varieties and its applications abstract this paper describes an effective procedure for stratifying a real semi algebraic set into cells of constant description size the attractive feature of our method is that the number of cells produced is singly exponential in the number of input variables this compares favorably with the doubly exponential size of collins decomposition unlike collins construction however our scheme does not produce a cell complex but only a smooth stratification nevertheless we are able to apply our results in interesting ways to problems of point location and geometric optimization

on line learning in the committee machine the dynamics of learning from examples in the k 3 non overlapping committee machine with single presentation of examples is studied the optimal algorithm in the sense of mean generalization is obtained from a variational analysis of the differential equations which describe the dynamics the agreement of the theoretical predictions and the results of numerical simulations is excellent the optimized dynamics has the extra advantage with respect to the non optimized cases in that it uncouples the differential equations which describe the evolution of the relevant parameters i e the student teacher overlap and the norm of the student synaptic vector this in turn translates into the possibility of constructing useful practical optimized on line algorithms for the optimal algorithm the generalization error decays as approximately 0 88 alpha 1 the same nominal error as for the simple perceptron with optimized dynamics

supervised matrix factorization hashing for cross modal retrieval the target of cross modal hashing is to embed heterogeneous multimedia data into a common low dimensional hamming space which plays a pivotal part in multimedia retrieval due to the emergence of big multimodal data recently matrix factorization has achieved great success in cross modal hashing however how to effectively use label information and local geometric structure is still a challenging problem for these approaches to address this issue we propose a cross modal hashing method based on collective matrix factorization which considers both the label consistency across different modalities and the local geometric consistency in each modality these two elements are formulated as a graph laplacian term in the objective function leading to a substantial improvement on the discriminative power of latent semantic features obtained by collective matrix factorization moreover the proposed method learns unified hash codes for different modalities of an instance to facilitate cross modal search and the objective function is solved using an iterative strategy the experimental results on two benchmark data sets show the effectiveness of the proposed method and its superiority over state of the art cross modal hashing methods

on the ability of the optimal perceptron to generalise a linearly separable boolean function is derived from a set of examples by a perceptron with optimal stability the probability to reconstruct a pattern which is not learnt is calculated analytically using the replica method

optimal generalization in perceptions a new learning algorithm for the one layer perceptron is presented it aims to maximize the generalization gain per example analytical results are obtained for the case of single presentation of each example the weight attached to a hebbian term is a function of the expected stability of the example in the teacher perceptron this leads to the obtention of upper bounds for the generalization ability this scheme can be iterated and the results of numerical simulations show that it converges within errors to the theoretical optimal generalization ability of the bayes algorithm analytical and numerical results for an algorithm with maximized generalization in the learning strategy with selection of examples are obtained and it is proved that as expected orthogonal selection is optimal exponential decay of the generalization error is obtained for the single presentation of selected examples

capacity of neural networks with discrete synaptic couplings the authors study the optimal storage capacity of neural networks with discrete local constraints on the synaptic couplings jij models with such constraints include those with binary couplings jij or 1 or jij 0 1 quantised couplings with larger synaptic range e g jij or 1 l or 2 l or 1 and in the limit continuous couplings confined to the hypercube mod jij mod or 1 box confinement they find that the optimal storage capacity alpha kappa is best determined by the vanishing of a suitably defined entropy as calculated in the replica symmetric approximation they also extend their results to cases with biased memories and make contact with sparse coding models

generalization performance of bayes optimal classification algorithm for learning a perceptron the generalization error of the bayes optimal classification algorithm when learning a perceptron from noise free random training examples is calculated exactly using methods of statistical mechanics it is shown that if an assumption of replica symmetry is made then in the thermodynamic limit the error of the bayes optimal algorithm is less than the error of a canonical stochastic learning algorithm by a factor approaching ensuremath surd 2 as the ratio of the number of training examples to perceptron weights grows in addition it is shown that approximations to the generalization error of the bayes optimal algorithm can be achieved by learning algorithms that use a two layer neutral net to learn a perceptron

layered neural networks exact solutions for the dynamics of layered feedforward neural networks are presented these networks are expected to respond to an input by going through a sequence of preassigned states on the various layers the family of networks considered has a variety of interlayer couplings linear and non linear hebbian hebbian with gaussian synaptic noise and with various kinds of dilution in addition the authors also solve the problem of layered networks with the pseudoinverse projector matrix of couplings in all cases the solutions take the form of layer to layer recursions for the mean overlap with a random key pattern and for the width of the embedding field distribution the dynamics is governed by the fixed points of these recursions for all cases non trivial domains of attraction of the memory states are found and graphically displayed

gradient descent learning in perceptrons a review of its possibilities we present a streamlined formalism which reduces the calculation of the generalization error for a perceptron trained on random examples generated by a teacher perceptron to a matter of simple algebra the method is valid whenever the student perceptron can be identified as the unique minimum of a specific cost function the asymptotic generalization error is calculated explicitly for a broad class of cost functions and a specific cost function is singled out that leads to a generalization error extremely close to the one of the bayes classifier

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state