deterministic walks in random environments deterministic walks in random environments dwre occupy an intermediate position between purely random generated by random trials and purely deterministic generated by deterministic dynamical systems e g by maps models of diffusion these models combine deterministic and probabilistic features we review general properties of dwre and demonstrate that to a large extent their dynamics and their statistics can be analyzed consecutively and separately we also show that orbits of one dimensional walks in rigid environments with non constant rigidity almost surely visit each site infinitely many times

positive solutions for singular semi positone boundary value problems in this paper some new results about the existence of positive solutions for singular semi positone boundary value problems are obtained the results of this paper partially improve the former corresponding work

tensor representation learning based image patch analysis for text identification and recognition in this paper we introduce a novel framework for text identification and recognition called tensor representation learning based image patch analysis trl ipa unlike most of previous text identification approaches which can only be applied to binarized images trl ipa can be directly applied to gray level and color images trl ipa is built on a general formulation of the convergent tensor representation learning ctrl algorithms in the implementation of trl ipa image patches are represented in the form of tensors while low dimensional representations of these tensors are learned via a ctrl algorithm to identify text regions in new coming document images a random forest classifier is trained in the learned tensor subspace moreover the trl ipa framework can be straightforwardly applied to recognition problems such as handwritten digits recognition we conducted extensive experiments on ancient chinese arabic and cyrillic document images to evaluate trl ipa on text identification tasks experimental results demonstrate its effectiveness and robustness in addition recognition results on images of handwritten digits show its advantage over state of the art vector and tensor representation based approaches highlightsa novel model tensor representation learning based image patch analysis trl ipa is proposed for document understanding trl ipa is built on a general formulation of the convergent tensor representation learning ctrl algorithms the ctrl algorithms are theoretically guaranteed to converge to a local optimal solution of the learning problem extensive experiments demonstrate the superiority of trl ipa over related vector and tensor representation based approaches

dictionary pair learning on grassmann manifolds for image denoising image denoising is a fundamental problem in computer vision and image processing that holds considerable practical importance for real world applications the traditional patch based and sparse coding driven image denoising methods convert 2d image patches into 1d vectors for further processing thus these methods inevitably break down the inherent 2d geometric structure of natural images to overcome this limitation pertaining to the previous image denoising methods we propose a 2d image denoising model namely the dictionary pair learning dpl model and we design a corresponding algorithm called the dpl on the grassmann manifold dplg algorithm the dplg algorithm first learns an initial dictionary pair i e the left and right dictionaries by employing a subspace partition technique on the grassmann manifold wherein the refined dictionary pair is obtained through a sub dictionary pair merging the dplg obtains a sparse representation by encoding each image patch only with the selected sub dictionary pair the non zero elements of the sparse representation are further smoothed by the graph laplacian operator to remove the noise consequently the dplg algorithm not only preserves the inherent 2d geometric structure of natural images but also performs manifold smoothing in the 2d sparse coding space we demonstrate that the dplg algorithm also improves the structural similarity values of the perceptual visual quality for denoised images using the experimental evaluations on the benchmark images and berkeley segmentation data sets moreover the dplg also produces the competitive peak signal to noise ratio values from popular image denoising algorithms

stdp compatible approximation of backpropagation in an energy based model we show that langevin markov chain monte carlo inference in an energy based model with latent variables has the property that the early steps of inference starting from a stationary point correspond to propagating error gradients into internal layers similar to backpropagation the backpropagated error is with respect to output units that have received an outside driving force pushing them away from the stationary point backpropagated error gradients correspond to temporal derivatives with respect to the activation of hidden units these lead to a weight update proportional to the product of the presynaptic firing rate and the temporal rate of change of the postsynaptic firing rate simulations and a theoretical argument suggest that this rate based update rule is consistent with those associated with spike timing dependent plasticity the ideas presented in this article could be an element of a theory for explaining how brains perform credit assignment in deep hierarchies as efficiently as backpropagation does with neural computation corresponding to both approximate inference in continuous valued latent variables and error backpropagation at the same time

learning algorithm for boltzmann machines with additive weight and bias noise this brief presents analytical results on the effect of additive weight bias noise on a boltzmann machine bm in which the unit output is in 1 1 instead of 0 1 with such noise it is found that the state distribution is yet another boltzmann distribution but the temperature factor is elevated thus the desired gradient ascent learning algorithm is derived and the corresponding learning procedure is developed this learning procedure is compared with the learning procedure applied to train a bm with noise it is found that these two procedures are identical therefore the learning algorithm for noise free bms is suitable for implementing as an online learning algorithm for an analog circuit implemented bm even if the variances of the additive weight noise and bias noise are unknown

variable selection in kernel regression using measurement error selection likelihoods abstractthis article develops a nonparametric shrinkage and selection estimator via the measurement error selection likelihood approach recently proposed by stefanski wu and white the measurement error kernel regression operator mekro has the same form as the nadaraya watson kernel estimator but optimizes a measurement error model selection likelihood to estimate the kernel bandwidths much like lasso or cosso solution paths mekro results in solution paths depending on a tuning parameter that controls shrinkage and selection via a bound on the harmonic mean of the pseudo measurement error standard deviations we use small sample corrected aic to select the tuning parameter large sample properties of mekro are studied and small sample properties are explored via monte carlo experiments and applications to data supplementary materials for this article are available online

learning unrealizable tasks from minimum entropy queries in supervised learning learning from queries rather than from random examples can improve generalization performance significantly we study the performance of query learning for unrealizable tasks where the student cannot learn from the perfectly as a simple model scenario of this kind we consider a linear perceptron student learning a general nonlinear perceptron teacher two kinds of queries for maximum information gain i e minimum entropy are investigated minimum student space entropy msse queries which are appropriate if the teacher space is unknown and minimum teacher space entropy mtse queries which can be used if the teacher space is assumed to be known but a student of a simpler form has deliberately been chosen we find that for msse queries the structure of the student space determines the efficacy of query learning mtse queries on the other hand which we investigate for the extreme case of a binary perceptron teacher lead to a higher generalization error than random examples due to a lack of feedback about the progress of the student in the way queries are selected

multidimensional density shaping by sigmoids an estimate of the probability density function of a random vector is obtained by maximizing the output entropy of a feedforward network of sigmoidal units with respect to the input weights classification problems can be solved by selecting the class associated with the maximal estimated density newton s optimization method applied to the estimated density yields a recursive estimator for a random variable or a random sequence a constrained connectivity structure yields a linear estimator which is particularly suitable for real time prediction a gaussian nonlinearity yields a closed form solution for the network s parameters which may also be used for initializing the optimization algorithm when other nonlinearities are employed a triangular connectivity between the neurons and the input which is naturally suggested by the statistical setting reduces the number of parameters applications to classification and forecasting problems are demonstrated

large margin low rank tensor analysis we present a supervised model for tensor dimensionality reduction which is called large margin low rank tensor analysis lmlrta in contrast to traditional vector representation based dimensionality reduction methods lmlrta can take any order of tensors as input and unlike previous tensor dimensionality reduction methods which can learn only the low dimensional embeddings with a priori specified dimensionality lmlrta can automatically and jointly learn the dimensionality and the low dimensional representations from data moreover lmlrta delivers low rank projection matrices while it encourages data of the same class to be close and of different classes to be separated by a large margin of distance in the low dimensional tensor space lmlrta can be optimized using an iterative fixed point continuation algorithm which is guaranteed to converge to a local optimal solution of the optimization problem we evaluate lmlrta on an object recognition application where the data are represented as 2d tensors and a face recognition application where the data are represented as 3d tensors experimental results show the superiority of lmlrta over state of the art approaches

justifying and generalizing contrastive divergence we study an expansion of the log likelihood in undirected graphical models such as the restricted boltzmann machine rbm where each term in the expansion is associated with a sample in a gibbs chain alternating between two random variables the visible vector and the hidden vector in rbms we are particularly interested in estimators of the gradient of the log likelihood obtained through this expansion we show that its residual term converges to zero justifying the use of a truncation running only a short gibbs chain which is the main idea behind the contrastive divergence cd estimator of the log likelihood gradient by truncating even more we obtain a stochastic reconstruction error related through a mean field approximation to the reconstruction error often used to train autoassociators and stacked autoassociators the derivation is not specific to the particular parametric forms used in rbms and requires only convergence of the gibbs chain we present theoretical and empirical evidence linking the number of gibbs steps k and the magnitude of the rbm parameters to the bias in the cd estimator these experiments also suggest that the sign of the cd estimator is correct most of the time even when the bias is large so that cd k is a good descent direction even for small k

what is the goal of sensory coding a number of recent attempts have been made to describe early sensory coding in terms of a general information processing strategy in this paper two strategies are contrasted both strategies take advantage of the redundancy in the environment to produce more effective representations the first is described as a compact coding scheme a compact code performs a transform that allows the input to be represented with a reduced number of vectors cells with minimal rms error this approach has recently become popular in the neural network literature and is related to a process called principal components analysis pca a number of recent papers have suggested that the optimal compact code for representing natural scenes will have units with receptive field profiles much like those found in the retina and primary visual cortex however in this paper it is proposed that compact coding schemes are insufficient to account for the receptive field properties of cells in the mammalian visual pathway in contrast it is proposed that the visual system is near to optimal in representing natural scenes only if optimality is defined in terms of sparse distributed coding in a sparse distributed code all cells in the code have an equal response probability across the class of images but have a low response probability for any single image in such a code the dimensionality is not reduced rather the redundancy of the input is transformed into the redundancy of the firing pattern of cells it is proposed that the signature for a sparse code is found in the fourth moment of the response distribution i e the kurtosis in measurements with 55 calibrated natural scenes the kurtosis was found to peak when the bandwidths of the visual code matched those of cells in the mammalian visual cortex codes resembling wavelet transforms are proposed to be effective because the response histograms of such codes are sparse i e show high kurtosis when presented with natural scenes it is proposed that the structure of the image that allows sparse coding is found in the phase spectrum of the image it is suggested that natural scenes to a first approximation can be considered as a sum of self similar local functions the inverse of a wavelet possible reasons for why sensory systems would evolve toward sparse coding are presented

plurality and resemblance in fmri data analysis abstract we apply nine analytic methods employed currently in imaging neuroscience to simulated and actual bold fmri signals and compare their performances under each signal type starting with baseline time series generated by a resting subject during a null hypothesis study we compare method performance with embedded focal activity in these series of three different types whose magnitudes and time courses are simple convolved with spatially varying hemodynamic responses and highly spatially interactive we then apply these same nine methods to bold fmri time series from contralateral primary motor cortex and ipsilateral cerebellum collected during a sequential finger opposition study paired comparisons of results across methods include a voxel specific concordance correlation coefficient for reproducibility and a resemblance measure that accommodates spatial autocorrelation of differences in activity surfaces receiver operating characteristic curves show considerable model differences in ranges less than 10 significance level false positives and greater than 80 power true positives concordance and resemblance measures reveal significant differences between activity surfaces in both data sets these measures can assist researchers by identifying groups of models producing similar and dissimilar results and thereby help to validate consolidate and simplify reports of statistical findings a pluralistic strategy for fmri data analysis can uncover invariant and highly interactive relationships between local activity foci and serve as a basis for further discovery of organizational principles of the brain results also suggest that a pluralistic empirical strategy coupled formally with substantive prior knowledge can help to uncover new brain behavior relationships that may remain hidden if only a single method is employed

graph regularized restricted boltzmann machine the restricted boltzmann machine rbm has received an increasing amount of interest in recent years it determines good mapping weights that capture useful latent features in an unsupervised manner the rbm and its generalizations have been successfully applied to a variety of image classification and speech recognition tasks however most of the existing rbm based models disregard the preservation of the data manifold structure in many real applications the data generally reside on a low dimensional manifold embedded in high dimensional ambient space in this brief we propose a novel graph regularized rbm to capture features and learning representations explicitly considering the local manifold structure of the data by imposing manifold based locality that preserves constraints on the hidden layer of the rbm the model ultimately learns sparse and discriminative representations the representations can reflect data distributions while simultaneously preserving the local manifold structure of data we test our model using several benchmark image data sets for unsupervised clustering and supervised classification problem the results demonstrate that the performance of our method exceeds the state of the art alternatives

the effects of adding noise during backpropagation training on a generalization performance we study the effects of adding noise to the inputs outputs weight connections and weight changes of multilayer feedforward neural networks during backpropagation training we rigorously derive and analyze the objective functions that are minimized by the noise affected training processes we show that input noise and weight noise encourage the neural network output to be a smooth function of the input or its weights respectively in the weak noise limit noise added to the output of the neural networks only changes the objective function by a constant hence it cannot improve generalization input noise introduces penalty terms in the objective function that are related to but distinct from those found in the regularization approaches simulations have been performed on a regression and a classification problem to further substantiate our analysis input noise is found to be effective in improving the generalization performance for both problems however weight noise is found to be effective in improving the generalization performance only for the classification problem other forms of noise have practically no effect on generalization

similarities of error regularization sigmoid gain scaling target smoothing and training with jitter the generalization performance of feedforward layered perceptrons can in many cases be improved either by smoothing the target via convolution regularizing the training error with a smoothing constraint decreasing the gain i e slope of the sigmoid nonlinearities or adding noise i e jitter to the input training data in certain important cases the results of these procedures yield highly similar results although at different costs training with jitter for example requires significantly more computation than sigmoid scaling

dimension reduction with extreme learning machine data may often contain noise or irrelevant information which negatively affect the generalization capability of machine learning algorithms the objective of dimension reduction algorithms such as principal component analysis pca non negative matrix factorization nmf random projection rp and auto encoder ae is to reduce the noise or irrelevant information of the data the features of pca eigenvectors and linear ae are not able to represent data as parts e g nose in a face image on the other hand nmf and non linear ae are maimed by slow learning speed and rp only represents a subspace of original data this paper introduces a dimension reduction framework which to some extend represents data as parts has fast learning speed and learns the between class scatter subspace to this end this paper investigates a linear and non linear dimension reduction framework referred to as extreme learning machine ae elm ae and sparse elm ae selm ae in contrast to tied weight ae the hidden neurons in elm ae and selm ae need not be tuned and their parameters e g input weights in additive neurons are initialized using orthogonal and sparse random weights respectively experimental results on usps handwritten digit recognition data set cifar 10 object recognition and norb object recognition data set show the efficacy of linear and non linear elm ae and selm ae in terms of discriminative capability sparsity training time and normalized mean square error

learning consensus representation for weak style classification style classification e g baroque and gothic architecture style is grabbing increasing attention in many fields such as fashion architecture and manga most existing methods focus on extracting discriminative features from local patches or patterns however the spread out phenomenon in style classification has not been recognized yet it means that visually less representative images in a style class are usually very diverse and easily getting misclassified we name them weak style images another issue when employing multiple visual features towards effective weak style classification is lack of consensus among different features that is weights for different visual features in the local patch should have been allocated similar values to address these issues we propose a consensus style centralizing auto encoder cscae for learning robust style features representation especially for weak style classification first we propose a style centralizing auto encoder scae which centralizes weak style features in a progressive way then based on scae we propose both the non linear and linear version cscae which adaptively allocate weights for different features during the progressive centralization process consensus constraints are added based on the assumption that the weights of different features of the same patch should be similar specifically the proposed linear counterpart of cscae motivated by the shared weights idea as well as group sparsity improves both efficacy and efficiency for evaluations we experiment extensively on fashion manga and architecture style classification problems in addition we collect a new dataset online shopping for fashion style classification which will be publicly available for vision based fashion style research experiments demonstrate the effectiveness of the scae and cscae on both public and newly collected datasets when compared with the most recent state of the art works

dual autoencoders features for imbalance classification problem many classification problems encountered in real world applications exhibit a profile of imbalanced data current methods depend on data resampling in fact if the feature set provides a clear decision boundary resampling may not be needed to solve the imbalanced classification problem therefore this work proposes a feature learning method based on the autoencoder to learn a set of features with better classification capabilities of the minority and the majority classes to address the imbalanced classification problems two sets of features are learned by two stacked autoencoders with different activation functions to capture different characteristics of the data and they are combined to form the dual autoencoding features samples are then classified in the new feature space learned in this manner instead of the original input space experimental results show that the proposed method outperforms current resampling based methods with statistical significance for imbalanced pattern classification problems highlightsto our best knowledge this paper proposes the first feature learning based method for dealing with imbalance pattern classification using stacked autoencoders we provide a new viewpoint for solving imbalanced pattern classification problems by projecting the input space onto a learned feature space with better representation using stacked autoencoders the concatenation of learned features from two different stacked autoencoders further enhances the classification accuracy by providing a better representation with different characteristics of the data to the classifier to learn more specifically the sigmoid activation function is less sensitive to input changes and provides robust representation while the tanh activation function is more sensitive and provides detailed information of the data experimental results show that the daf yields statistical significant improvement in the auc the f1 score and the g mean in comparison to representative methods of resampling based and feature projection methods for dealing with the imbalanced pattern classification problems

the equivalence of weak strong and complete convergence in l_1 for kernel density estimates theorem 1 let k be a nonnegative borel measurable function on rd with f k x dx 1 then the following conditions are equivalent i j 0 in probability as n oo some f ii jn o 0 in probability as n s oo all f iii jn s 0 almost surely as n oo all f iv jn 0 exponentially as n oc i e for all e 0 there exist r no 0 such that p jne no all f v lim0h 0 and limnnhd oo also v implies iv when k is merely absolutely integrable and f k x dx 1 0

graph regularized auto encoders for image representation image representation has been intensively explored in the domain of computer vision for its significant influence on the relative tasks such as image clustering and classification it is valuable to learn a low dimensional representation of an image which preserves its inherent information from the original image space at the perspective of manifold learning this is implemented with the local invariant idea to capture the intrinsic low dimensional manifold embedded in the high dimensional input space inspired by the recent successes of deep architectures we propose a local invariant deep nonlinear mapping algorithm called graph regularized auto encoder gae with the graph regularization the proposed method preserves the local connectivity from the original image space to the representation space while the stacked auto encoders provide explicit encoding model for fast inference and powerful expressive capacity for complex modeling theoretical analysis shows that the graph regularizer penalizes the weighted frobenius norm of the jacobian matrix of the encoder mapping where the weight matrix captures the local property in the input space furthermore the underlying effects on the hidden representation space are revealed providing insightful explanation to the advantage of the proposed method finally the experimental results on both clustering and classification tasks demonstrate the effectiveness of our gae as well as the correctness of the proposed theoretical analysis and it also suggests that gae is a superior solution to the current deep representation learning techniques comparing with variant auto encoders and existing local invariant methods

information maximization and independent component analysis is there a difference this article provides a detailed and rigorous analysis of the two commonly used methods for redundancy reduction linear independent component analysis ica posed as a direct minimization of a suitably chosen redundancy measure and information maximization infomax of a continuous stochastic signal transmitted through an appropriate nonlinear network the article shows analytically that ica based on the kullbackleibler information as a redundancy measure and infomax lead to the same solution if the parameterization of the output nonlinear functions in the latter method is sufficiently rich furthermore this work discusses the alternative redundancy measures not based on the kullback leibler information distance the practical issues of applying ica and infomax are also discussed and illustrated on the problem of extracting statistically independent factors from a linear pixel by pixel mixture of images

locator checker scaler object tracking using spatially ordered and weighted patch descriptor in this paper we propose a simple yet effective object descriptor and a novel tracking algorithm to track a target object accurately for the object description we divide the bounding box of a target object into multiple patches and describe them with color and gradient histograms then we determine the foreground weight of each patch to alleviate the impacts of background information in the bounding box to this end we perform random walk with restart rwr simulation we then concatenate the weighted patch descriptors to yield the spatially ordered and weighted patch sowp descriptor for the object tracking we incorporate the proposed sowp descriptor into a novel tracking algorithm which has three components locator checker and scaler lcs the locator and the scaler estimate the center location and the size of a target respectively the checker determines whether it is safe to adjust the target scale in a current frame these three components cooperate with one another to achieve robust tracking experimental results demonstrate that the proposed lcs tracker achieves excellent performance on recent benchmarks

functional approximation by feed forward networks a least squares approach to generalization this paper considers a least squares approach to function approximation and generalization the particular problem addressed is one in which the training data are noiseless and the requirement is to define a mapping that approximates the data and that generalizes to situations in which data samples are corrupted by noise in the input variables the least squares approach produces a generalizer that has the form of a radial basis function network for a finite number of training samples the finite sample approximation is valid provided that the perturbations due to noise on the expected operating conditions are large compared to the sample spacing in the data space in the other extreme of small noise perturbations a particular parametric form must be assumed for the generalizer it is shown that better generalization will occur if the error criterion used in training the generalizer is modified by the addition of a specific regularization term this is illustrated by an approximator that has a feedforward architecture and is applied to the problem of point source location using the outputs of an array of receivers in the focal plane of a lens

a fuzzy restricted boltzmann machine novel learning algorithms based on the crisp possibilistic mean value of fuzzy numbers a fuzzy restricted boltzmann machine frbm is extended from a restricted boltzmann machine rbm by replacing all the real valued parameters with fuzzy numbers a new frbm that employs the crisp possibilistic mean value of a fuzzy number to defuzzify the fuzzy free energy function is presented this approach is much clearer and easier to obtain the expression of the defuzzified free energy function and its approximation than the centroid method several theorems that discuss the error bounds of the approximation to ensure the rationality and validity are also investigated learning algorithms are given for the designed frbm with symmetric triangular fuzzy numbers stfns asymmetric triangular fuzzy numbers and gaussian fuzzy numbers by appropriately choosing the parameters a theorem is concluded that all frbms with symmetric fuzzy numbers will have identical learning algorithm to that of frbms with stfns this is illustrated by a case of frbm with gaussian fuzzy numbers two experiments including the mnist handwriting recognition and the bars and stripes benchmark are carried out the results show that the proposed frbms significantly outperform rbms in learning accuracy and generalization ability especially when encountering unlearned samples and recovering incomplete images

probabilistic low rank multitask learning in this paper we consider the problem of learning multiple related tasks simultaneously with the goal of improving the generalization performance of individual tasks the key challenge is to effectively exploit the shared information across multiple tasks as well as preserve the discriminative information for each individual task to address this we propose a novel probabilistic model for multitask learning mtl that can automatically balance between low rank and sparsity constraints the former assumes a low rank structure of the underlying predictive hypothesis space to explicitly capture the relationship of different tasks and the latter learns the incoherent sparse patterns private to each task we derive and perform inference via variational bayesian methods experimental results on both regression and classification tasks on real world applications demonstrate the effectiveness of the proposed method in dealing with the mtl problems

singular dirichlet second order bvps with impulses abstract the paper deals with the existence of solutions to singular second order differential equations with impulse effects and with the dirichlet boundary conditions the right hand side of the differential equation can be singular in its phase variable

subspace clustering using a low rank constrained autoencoder the performance of subspace clustering is affected by data representation data representation for subspace clustering maps data from the original space into another space with the property of better separability many data representation methods have been developed in recent years typical among them are low rank representation lrr and an autoencoder lrr is a linear representation method that captures the global structure of data with low rank constraint alternatively an autoencoder nonlinearly maps data into a latent space using a neural network by minimizing the difference between the reconstruction and input to combine the advantages of an lrr globality and autoencoder self supervision based locality we propose a novel data representation method for subspace clustering the proposed method called low rank constrained autoencoder lrae forces the latent representation of the neural network to be of low rank and the low rank constraint is computed as a prior from the input space one major advantage of the lrae is that the learned data representation not only maintains the local features of the data but also preserves the underlying low rank global structure extensive experiments on several datasets for subspace clustering were conducted they demonstrated that the proposed lrae substantially outperformed state of the art subspace clustering methods

a latent manifold markovian dynamics gaussian process in this paper we propose a gaussian process gp model for analysis of nonlinear time series formulation of our model is based on the consideration that the observed data are functions of latent variables with the associated mapping between observations and latent representations modeled through gp priors in addition to capture the temporal dynamics in the modeled data we assume that subsequent latent representations depend on each other on the basis of a hidden markov prior imposed over them derivation of our model is performed by marginalizing out the model parameters in closed form using gp priors for observation mappings and appropriate stick breaking priors for the latent variable markovian dynamics this way we eventually obtain a nonparametric bayesian model for dynamical systems that accounts for uncertainty in the modeled data we provide efficient inference algorithms for our model on the basis of a truncated variational bayesian approximation we demonstrate the efficacy of our approach considering a number of applications dealing with real world data and compare it with the related state of the art approaches

on overfitting generalization and randomly expanded training sets an algorithmic procedure is developed for the random expansion of a given training set to combat overfitting and improve the generalization ability of backpropagation trained multilayer perceptrons mlps the training set is k means clustered and locally most entropic colored gaussian joint input output probability density function estimates are formed per cluster the number of clusters is chosen such that the resulting overall colored gaussian mixture exhibits minimum differential entropy upon global cross validated shaping numerical studies on real data and synthetic data examples drawn from the literature illustrate and support these theoretical developments

multiobjective optimization for model selection in kernel methods in regression regression plays a major role in many scientific and engineering problems the goal of regression is to learn the unknown underlying function from a set of sample vectors with known outcomes in recent years kernel methods in regression have facilitated the estimation of nonlinear functions however two major interconnected problems remain open the first problem is given by the bias versus variance tradeoff if the model used to estimate the underlying function is too flexible i e high model complexity the variance will be very large if the model is fixed i e low complexity the bias will be large the second problem is to define an approach for selecting the appropriate parameters of the kernel function to address these two problems this paper derives a new smoothing kernel criterion which measures the roughness of the estimated function as a measure of model complexity then we use multiobjective optimization to derive a criterion for selecting the parameters of that kernel the goal of this criterion is to find a tradeoff between the bias and the variance of the learned function that is the goal is to increase the model fit while keeping the model complexity in check we provide extensive experimental evaluations using a variety of problems in machine learning pattern recognition and computer vision the results demonstrate that the proposed approach yields smaller estimation errors as compared with methods in the state of the art

use of a tracer specific deep artificial neural net to denoise dynamic pet images application of kinetic modeling km on a voxel level in dynamic pet images frequently suffers from high levels of noise drastically reducing the precision of parametric image analysis in this paper we investigate the use of machine learning and artificial neural networks to denoise dynamic pet images we train a deep denoising autoencoder dae using noisy and noise free spatiotemporal image patches extracted from the simulated images of 11c raclopride a dopamine d2 receptor agonist the dae processed dynamic and corresponding parametric images simulated and acquired are compared with those obtained with conventional denoising techniques including temporal and spatial gaussian smoothing iterative spatiotemporal smoothing deconvolution and the highly constrained backprojection processing hypr the simulated acquired parametric image non uniformity was 7 75 19 49 with temporal and 5 90 14 50 with spatial smoothing 5 82 16 21 with smoothing deconvolution 5 49 13 38 with hypr and 3 52 11 41 with dae the dae also produced the best results in terms of the coefficient of variation of voxel values and structural similarity index denoising induced bias in the regional mean binding potential was 7 8 with temporal and 26 31 with spatial smoothing 28 61 with smoothing deconvolution 27 63 with hypr and 14 8 with dae when the test data did not match the training data erroneous outcomes were obtained our results demonstrate that a deep dae can provide a substantial reduction in the voxel level noise compared with the conventional spatiotemporal denoising methods while introducing a similar or lower amount of bias the better dae performance comes at the cost of lower generality and requiring appropriate training data

nonparametric density estimation the l 1 view differentiation of integrals consistency lower bounds for rates of convergence rates of convergence in l1 the automatic kernel estimate l1 and pointwise convergence estimates related to the kernel estimate and the histogram estimate simulation inequalities and random variate generation the transformed kernel estimate applications in discrimination operations on density estimates estimators based on orthogonal series index

a fast learning algorithm for deep belief nets we show how to use complementary priors to eliminate the explaining away effects that make inference difficult in densely connected belief nets that have many hidden layers using complementary priors we derive a fast greedy algorithm that can learn deep directed belief networks one layer at a time provided the top two layers form an undirected associative memory the fast greedy algorithm is used to initialize a slower learning procedure that fine tunes the weights using a contrastive version of the wake sleep algorithm after fine tuning a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels this generative model gives better digit classification than the best discriminative learning algorithms the low dimensional manifolds on which the digits lie are modeled by long ravines in the free energy landscape of the top level associative memory and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind

a fast fixed point algorithm for independent component analysis we introduce a novel fast algorithm for independent component analysis which can be used for blind source separation and feature extraction we show how a neural network learning rule can be transformed into a fixedpoint iteration which provides an algorithm that is very simple does not depend on any user defined parameters and is fast to converge to the most accurate solution allowed by the data the algorithm finds one at a time all nongaussian independent components regardless of their probability distributions the computations can be performed in either batch mode or a semiadaptive manner the convergence of the algorithm is rigorously proved and the convergence speed is shown to be cubic some comparisons to gradient based algorithms are made showing that the new algorithm is usually 10 to 100 times faster sometimes giving the solution in just a few iterations

progressive shape distribution encoder for learning 3d shape representation since there are complex geometric variations with 3d shapes extracting efficient 3d shape features is one of the most challenging tasks in shape matching and retrieval in this paper we propose a deep shape descriptor by learning shape distributions at different diffusion time via a progressive shape distribution encoder psde first we develop a shape distribution representation with the kernel density estimator to characterize the intrinsic geometry structures of 3d shapes then we propose to learn a deep shape feature through an unsupervised psde specially the unsupervised psde aims at modeling the complex non linear transform of the estimated shape distributions between consecutive diffusion time in order to characterize the intrinsic structures of 3d shapes more efficiently we stack multiple psdes to form a network structure finally we concatenate all neurons in the middle hidden layers of the unsupervised psde network to form an unsupervised shape descriptor for retrieval furthermore by imposing an additional constraint on the outputs of all hidden layers we propose a supervised psde to form a supervised shape descriptor for each hidden layer the similarity between a pair of outputs from the same class is as large as possible and the similarity between a pair of outputs from different classes is as small as possible the proposed method is evaluated on three benchmark 3d shape data sets with large geometric variations i e mcgill shrec 10 shapegoogle and shrec 14 human data sets and the experimental results demonstrate the superiority of the proposed method to the existing approaches

training with noise is equivalent to tikhonov regularization it is well known that the addition of noise to the input data of a neural network during training can in some circumstances lead to significant improvements in generalization performance previous work has shown that such training with noise is equivalent to a form of regularization in which an extra term is added to the error function however the regularization term which involves second derivatives of the error function is not bounded below and so can lead to difficulties if used directly in a learning algorithm based on error minimization in this paper we show that for the purposes of network training the regularization term can be reduced to a positive semi definite form that involves only first derivatives of the network mapping for a sum of squares error function the regularization term belongs to the class of generalized tikhonov regularizers direct minimization of the regularized error function provides a practical alternative to training with noise

learning with noise in a linear perceptron the learning of a set of p random patterns in a linear perceptron is studied in the limit of a large n of input units with noise on the weights inputs and output the problem is formulated in continuous time as a langevin equation and the first task is to evaluate the response or green function for the system white noise on the output is shown to correspond to spatially correlated weight noise acting only in a subspace of the weight space it is shown that the input noise acts as a simple weight decay with a size proportional to the load parameter alpha p n with no weight decay the relaxation time diverges at alpha 1 with a weight decay it becomes shorter and finite for alpha 1 but at the cost of a larger asymptotic learning error that is found analytically it is shown that a small weight decay decreases the effect of noise on the weights or outputs

generalization ability of perceptrons with continuous outputs in this paper we examine the influence of different input output relations on the generalization ability of a single layer perceptron the input output relations can be linear binary or sigmoid with this choice we take into account most of the cases which are of present interest the generalization problem will be realizable or unrealizable if the input output relations for teacher and student are identical or not we show that sometimes it can have a positive effect on the generalization ability if one learns with errors

walks in rigid environments continuous limits we derive the continuous limits of kinetic equations for spatially discrete systems generated by the motion of a particle in a random array of scatterers the type of scatterer at a vertex changes after the r th visit of the particle to this vertex where 1 r such deterministic cellular automata belong to the class of walks in rigid environments it has been recently shown that they form the simplest dynamical models with sub diffusive diffusive and super diffusive behaviour due to the deterministic character of the dynamics the continuous limit equations obtained for these models are of the euler type rather than the diffusive type the reason for that is that the fluctuations in these models are relatively small and there is no scaling of probabilities similar for example to those in the case of biased random walk that can account for them

representation and generalization properties of class entropy networks using conditional class entropy cce as a cost function allows feedforward networks to fully exploit classification relevant information cce based networks arrange the data space into partitions which are assigned unambiguous symbols and are labeled by class information by this labeling mechanism the network can model the empirical data distribution at the local level region labeling evolves with the network training process which follows a plastic algorithm the paper proves several theoretical properties about the performance of cce based networks and considers both convergence during training and generalization ability at run time in addition analytical criteria and practical procedures are proposed to enhance the generalization performance of the trained networks experiments on artificial and real world domains confirm the accuracy of this class of networks and witness the validity of the described methods

nonlinear autoassociation is not equivalent to pca a common misperception within the neural network community is that even with nonlinearities in their hidden layer autoassociators trained with backpropagation are equivalent to linear methods such as principal component analysis pca our purpose is to demonstrate that nonlinear autoassociators actually behave differently from linear methods and that they can outperform these methods when used for latent extraction projection and classification while linear autoassociators emulate pca and thus exhibit a flat or unimodal reconstruction error surface autoassociators with nonlinearities in their hidden layer learn domains by building error reconstruction surfaces that depending on the task contain multiple local valleys this interpolation bias allows nonlinear autoassociators to represent appropriate classifications of nonlinear multimodal domains in contrast to linear autoassociators which are inappropriate for such tasks in fact autoassociators with hidden unit nonlinearities can be shown to perform nonlinear classification and nonlinear recognition

dimensionality reduction using similarity induced embeddings the vast majority of dimensionality reduction dr techniques rely on the second order statistics to define their optimization objective even though this provides adequate results in most cases it comes with several shortcomings the methods require carefully designed regularizers and they are usually prone to outliers in this paper a new dr framework that can directly model the target distribution using the notion of similarity instead of distance is introduced the proposed framework called similarity embedding framework sef can overcome the aforementioned limitations and provides a conceptually simpler way to express optimization targets similar to existing dr techniques deriving a new dr technique using the sef becomes simply a matter of choosing an appropriate target similarity matrix a variety of classical tasks such as performing supervised dr and providing out of sample extensions as well as new novel techniques such as providing fast linear embeddings for complex techniques are demonstrated in this paper using the proposed framework six data sets from a diverse range of domains are used to evaluate the proposed method and it is demonstrated that it can outperform many existing dr techniques

using additive noise in back propagation training the possibility of improving the generalization capability of a neural network by introducing additive noise to the training samples is discussed the network considered is a feedforward layered neural network trained with the back propagation algorithm back propagation training is viewed as nonlinear least squares regression and the additive noise is interpreted as generating a kernel estimate of the probability density that describes the training vector distribution two specific application types are considered pattern classifier networks and estimation of a nonstochastic mapping from data corrupted by measurement errors it is not proved that the introduction of additive noise to the training vectors always improves network generalization however the analysis suggests mathematically justified rules for choosing the characteristics of noise if additive noise is used in training results of mathematical statistics are used to establish various asymptotic consistency results for the proposed method numerical simulations support the applicability of the training method

positive solutions for singular m point boundary value problems with positive parameter in this paper some new existence results for positive solutions for a class of singular m point boundary value problems with parameter be obtained

noise injection theoretical prospects noise injection consists of adding noise to the inputs during neural network training experimental results suggest that it might improve the generalization ability of the resulting neural network a justification of this improvement remains elusive describing analytically the average perturbed cost function is difficult and controlling the fluctuations of the random perturbed cost function is hard hence recent papers suggest replacing the random perturbed cost by a deterministic taylor approximation of the average perturbed cost function this article takes a different stance when the injected noise is gaussian noise injection is naturally connected to the action of the heat kernel this provides indications on the relevance domain of traditional taylor expansions and shows the dependence of the quality of taylor approximations on global smoothness properties of neural networks under consideration the connection between noise injection and heat kernel also enables controlling the fluctuations of

anisotropic noise injection for input variables relevance determination there are two archetypal ways to control the complexity of a flexible regressor subset selection and ridge regression in neural networks jargon they are respectively known as pruning and weight decay these techniques may also be adapted to estimate which features of the input space are relevant for predicting the output variables relevance is given by a binary indicator for subset selection and by a continuous rating for ridge regression this paper shows how to achieve such a rating for a multilayer perceptron trained with noise or jitter noise injection ni is modified in order to penalize heavily irrelevant features the proposed algorithm is attractive as it requires the tuning of a single parameter this parameter controls the complexity of the model effective number of parameters together with the rating of feature relevances effective input space dimension bounds on the effective number of parameters support that the stability of this adaptive scheme is enforced by the constraints applied to the admissible set of relevance indices the good properties of the algorithm are confirmed by satisfactory experimental results on simulated data sets

nonnegative solutions of singular boundary value problems with sign changing nonlinearities the paper presents sufficient conditions for the existence of positive solutions of the equation x t q t f t x t x t 0 with the dirichlet conditions x 0 0 x 1 0 and of the equation p t x t p t q t f t x t p t x t 0 with the boundary conditions limt o p t x t 0 x 1 0 our nonlinearity f is allowed to change sign and f may be singular at x 0 the proofs are based on a combination of the regularity and sequential techniques and the method of lower and upper functions

propagation dispersion equation a propagation dispersion equation is derived for the first passage distribution function of a particle moving on a substrate with time delays the equation is obtained as the hydrodynamic limit of the first visit equation an exact microscopic finite difference equation describing the motion of a particle on a lattice whose sites operate as time delayers the propagation dispersion equation should be contrasted with the advection diffusion equation or the classical fokker planck equation as it describes a dispersion process in time instead of diffusion in space with a drift expressed by a propagation speed with non zero bounded values the temporal dispersion coefficient is shown to exhibit a form analogous to taylor s dispersivity physical systems where the propagation dispersion equation applies are discussed

stacked denoising autoencoders learning useful representations in a deep network with a local denoising criterion we explore an original strategy for building deep networks based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs the resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders it is however shown on a benchmark of classification problems to yield significantly lower classification error thus bridging the performance gap with deep belief networks dbn and in several cases surpassing it higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent svm classifiers qualitative experiments show that contrary to ordinary autoencoders denoising autoencoders are able to learn gabor like edge detectors from natural image patches and larger stroke detectors from digit images this work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations

criteria for multichannel signal separation we consider the problem in which we want to separate two or more signals that are coupled to each other through an unknown multiple input multiple output linear system channel we prove that the signals can be decoupled or separated using only the condition that they are statistically independent and find even weaker sufficient conditions involving their cross polyspectra by imposing these conditions on the reconstructed signals we obtain a class of criteria for signal separation these criteria are universal in the sense that they do not require any prior knowledge or information concerning the nature of the source signals they may be communication signals or speech signals or any other 1 d or multidimensional signals e g images computationally efficient algorithms for implementing the proposed criteria that only involve the iterative solution to a linear least squares problem are presented

ensembles of random sphere cover classifiers we propose and evaluate a new set of ensemble methods for the randomised sphere cover rsc classifier rsc is a classifier using the sphere cover method that bases classification on distance to spheres rather than distance to instances the randomised nature of rsc makes it ideal for use in ensembles we propose two ensemble methods tailored to the rsc classifier rse an ensemble based on instance resampling and rsse a subspace ensemble we compare rse and rsse to tree based ensembles on a set of uci data sets and demonstrate that rsc ensembles perform significantly better than some of these ensembles and not significantly worse than the others we demonstrate via a case study on six gene expression data sets that rsse can outperform other subspace ensemble methods on high dimensional data when used in conjunction with an attribute filter finally we perform a set of bias variance decomposition experiments to analyse the source of improvement in comparison to a base classifier highlightswe developed novel ensemble algorithms for generalised instance based classifiers the base classifier used is a randomised sphere cover classifier the ensemble take advantage of the random selection of spheres and the parameters to add diversity to the ensemble the subspace method for ensemble design is used successfully for the randomised sphere cover we tested the algorithms on various data sets including a case study using gene expression data sets

training products of experts by minimizing contrastive divergence it is possible to combine multiple latent variable models of the same data by multiplying their probability distributions together and then renormalizing this way of combining individual expert models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data a product of experts poe is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary training a poe by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule fortunately a poe can be trained using a different objective function called contrastive divergence whose derivatives with regard to the parameters can be approximated accurately and efficiently examples are presented of contrastive divergence learning using several types of expert on several types of data

an asymptotically efficient solution to the bandwidth problem of kernel density estimation a data driven method of choosing the bandwidth h of a kernel density estimator is heuristically motivated by considering modifications of the kullback leibler or pseudo likelihood cross validation function it is seen that this means of choosing h is asymptotically equivalent to taking the h that minimizes some compelling error criteria such as the average squared error and the integrated squared error thus for a given kernel function the bandwidth can be chosen optimally without making precise smoothness assumptions on the underlying density

statistical independence and novelty detection with information preserving nonlinear maps according to barlow 1989 feature extraction can be understood as finding a statistically independent representation of the probability distribution underlying the measured signals the search for a statistically independent representation can be formulated by the criterion of minimal mutual information which reduces to decorrelation in the case of gaussian distributions if nongaussian distributions are to be considered minimal mutual information is the appropriate generalization of decorrelation as used in linear principal component analyses pca we also generalize to nonlinear transformations by only demanding perfect transmission of information this leads to a general class of nonlinear transformations namely symplectic maps conservation of information allows us to consider only the statistics of single coordinates the resulting factorial representation of the joint probability distribution gives a density estimation we apply this concept to the real world problem of electrical motor fault detection treated as a novelty detection task

an analysis of noise in recurrent neural networks convergence and generalization concerns the effect of noise on the performance of feedforward neural nets we introduce and analyze various methods of injecting synaptic noise into dynamically driven recurrent nets during training theoretical results show that applying a controlled amount of noise during training may improve convergence and generalization performance we analyze the effects of various noise parameters and predict that best overall performance can be achieved by injecting additive noise at each time step noise contributes a second order gradient term to the error function which can be viewed as an anticipatory agent to aid convergence this term appears to find promising regions of weight space in the beginning stages of training when the training error is large and should improve convergence on error surfaces with local minima the first order term is a regularization term that can improve generalization specifically it can encourage internal representations where the state nodes operate in the saturated regions of the sigmoid discriminant function while this effect can improve performance on automata inference problems with binary inputs and target outputs it is unclear what effect it will have on other types of problems to substantiate these predictions we present simulations on learning the dual parity grammar from temporal strings for all noise models and present simulations on learning a randomly generated six state grammar using the predicted best noise model

robust transfer metric learning for image classification metric learning has attracted increasing attention due to its critical role in image analysis and classification conventional metric learning always assumes that the training and test data are sampled from the same or similar distribution however to build an effective distance metric we need abundant supervised knowledge i e side label information which is generally inaccessible in practice because of the expensive labeling cost in this paper we develop a robust transfer metric learning rtml framework to effectively assist the unlabeled target learning by transferring the knowledge from the well labeled source domain specifically rtml exploits knowledge transfer to mitigate the domain shift in two directions i e sample space and feature space in the sample space domain wise and class wise adaption schemes are adopted to bridge the gap of marginal and conditional distribution disparities across two domains in the feature space our metric is built in a marginalized denoising fashion and low rank constraint which make it more robust to tackle noisy data in reality furthermore we design an explicit rank constraint regularizer to replace the rank minimization np hard problem to guide the low rank metric learning experimental results on several standard benchmarks demonstrate the effectiveness of our proposed rtml by comparing it with the state of the art transfer learning and metric learning algorithms

generalization in a linear perceptron in the presence of noise the authors study the evolution of the generalization ability of a simple linear perceptron with n inputs which learns to imitate a teacher perceptron the system is trained on p alpha n example inputs drawn from some distribution and the generalization ability is measured by the average agreement with the teacher on test examples drawn from the same distribution the dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at alpha 1 when there are no errors static noise in the training examples if the examples are produced by an erroneous teacher overfitting is observed i e the generalization error starts to increase after a finite time of training it is shown that a weight decay of the same size as the variance of the noise errors on the teacher improves on the generalization and suppresses the overfitting the generalization error as a function of time is calculated numerically for various values of the parameters finally dynamic noise in the training is considered white noise on the input corresponds on average to a weight decay and can thus improve generalization whereas white noise on the weights or the output degrades generalization generalization is particularly sensitive to noise on the weights for alpha 1 where it makes the error constantly increase with time but this effect is also shown to be damped by a weight decay weight noise and output noise acts similarly above the transition at alpha 1

a new approach to fuzzy modeling of nonlinear dynamic systems with noise relevance vector learning mechanism this paper presents a new fuzzy inference system for modeling of nonlinear dynamic systems based on input and output data with measurement noise the proposed fuzzy system has a number of fuzzy rules and parameter values of membership functions which are automatically generated using the extended relevance vector machine rvm the rvm has a probabilistic bayesian learning framework and has good generalization capability the rvm consists of the sum of product of weight and kernel function which projects input space into high dimensional feature space the structure of proposed fuzzy system is same as that of the takagi sugeno fuzzy model however in the proposed method the number of fuzzy rules can be reduced under the process of optimizing a marginal likelihood by adjusting parameter values of kernel functions using the gradient ascent method after a fuzzy system is determined coefficients in consequent part are found by the least square method examples illustrate effectiveness of the proposed new fuzzy inference system

positive solutions of sublinear sturm liouville problems with changing sign nonlinearity in this paper the following nonlinear sturm liouville problem p x u x q x u x email protected x u x email protected email protected 1 a 0u 0 email protected 0u 0 0 a 1u 1 email protected 1u 1 0 is discussed by topological methods in the case that the nonlinear term is non singular or singular a global structure of the positive solution set of the above problem is obtained and the existence of positive solutions is proved under the condition that the nonlinear term is allowed to change sign

convolutional covariance features conception integration and performance in person re identification abstract this paper introduces a novel type of features based on covariance descriptors the convolutional covariance features ccf differently from the traditional and handcrafted way to obtain covariance descriptors ccf is computed from adaptive and trainable features which come from a coarse to fine transfer learning cfl strategy cfl provides a generic to specific knowledge and noise invariant information for person re identification after training the deep features convolutional and flat features are extracted from respectively intermediate and top layers of a hybrid deep network intermediate layer features are then wrapped in covariance matrices composing the so called ccf which are integrated to the top layer features called here flat features integration of ccf and flat features demonstrated to improve the proposed person re identification in comparison with the use of the component features alone our person re identification method achieved the best top 1 performance when compared with other 18 state of the art methods over viper i lids cuhk01 and cuhk03 data sets the compared methods are based on deep learning covariance descriptors or handcrafted features and similarity functions

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

positive solutions of superlinear semipositone singular dirichlet boundary value problems in this paper we study a class of superlinear semipositone singular second order dirichlet boundary value problem a sufficient condition for the existence of positive solution is obtained under the more simple assumptions

how fast does langton s ant move the automaton known as langton s ant exhibits a propagation phase where the particle dynamics the ant produces a regular periodic pattern called highway despite the simplicity of its basic algorithm langton s ant has remained a puzzle in terms of analytical description here i show that propagation dynamics obeys a general difference equation for a class of automata which includes 1 d 2 d triangular and square lattice models in the case of langton s ant the speed of the ant in the highway c 2 52 follows exactly from the equation

on langevin updating in multilayer perceptrons the langevin updating rule in which noise is added to the weights during learning is presented and shown to improve learning on problems with initially ill conditioned hessians this is particularly important for multilayer perceptrons with many hidden layers that often have ill conditioned hessians in addition manhattan updating is shown to have a similar effect

geodesic invariant feature a local descriptor in depth different from the photometric images depth images resolve the distance ambiguity of the scene while the properties such as weak texture high noise and low resolution may limit the representation ability of the well developed descriptors which are elaborately designed for the photometric images in this paper a novel depth descriptor geodesic invariant feature gif is presented for representing the parts of the articulate objects in depth images gif is a multilevel feature representation framework which is proposed based on the nature of depth images low level geodesic gradient is introduced to obtain the invariance to the articulate motion such as scale and rotation variation midlevel superpixel clustering is applied to reduce depth image redundancy resulting in faster processing speed and better robustness to noise high level deep network is used to exploit the nonlinearity of the data which further improves the classification accuracy the proposed descriptor is capable of encoding the local structures in the depth data effectively and efficiently comparisons with the state of the art methods reveal the superiority of the proposed method

convergent algorithm for sensory receptive field development an unsupervised developmental algorithm for linear maps is derived which reduces the pixel entropy using the measure introduced in previous work at every update and thus removes pairwise correlations between pixels since the measure of pixel entropy has only a global minimum the algorithm is guaranteed to converge to the minimum entropy map such optimal maps have recently been shown to possess cognitively desirable properties and are likely to be used by the nervous system to organize sensory information the algorithm derived here turns out to be one proposed by goodall for pairwise decorrelation it is biologically plausible since in a neural network implementation it requires only data available locally to a neuron in training over ensembles of two dimensional input signals with the same spatial power spectrum as natural scenes networks develop output neurons with center surround receptive fields similar to those of ganglion cells in the retina some technical issues pertinent to developmental algorithms of this sort such as symmetry fixing are also discussed

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

clustering of cell populations in flow cytometry data using a combination of gaussian mixtures we propose a supervised learning approach to automatic quantification of cell populations in flow cytometric samples one sample contains up to millions of measurement vectors with a dimensionality between 10 and 20 normally each measurement vector corresponds to a single cell in the biological sample identifying biologically meaningful cell populations is essentially a clustering problem however standard clustering methods are impractical because size shape and location of corresponding clusters may vary strongly between samples mainly due to phenotypic differences and inter laboratory variations in our holistic approach we implicitly employ the structural information such as relative locations and shape of sub populations a new input sample is reconstructed by a linear combination of artificial reference samples each represented by a gaussian mixture model gmm in which for each gaussian component the class label of the corresponding cluster of observations is known the reference samples are calculated from a larger set of training samples by non negative matrix factorization and can be regarded as the basis of a lower dimensional feature space in which input samples are reconstructed we show a method for calculating the feature space transformation based on minimization the l2 distance defined between two gmm the feature space representation of the sample is then used to assign each observation to one of the specified sub populations by a bayes decision we present classification results on a database of about 170 patients with acute lymphoblastic leukemia all where high accuracy in the prediction of relatively small leukemic populations is crucial the approach is not limited to our application it can be employed wherever analysis of large multi dimensional numerical data of a specific class of samples with related structure has to be performed highlightsa density model based on the interpolation of gaussian mixture models is presented non negative matrix factorization is used to compress the model affine registration of gaussian mixture models using the l2 distance is performed the applicability of the method is demonstrated on flow cytometry data

a connection between score matching and denoising autoencoders denoising autoencoders have been previously shown to be competitive alternatives to restricted boltzmann machines for unsupervised pretraining of each layer of a deep architecture we show that a simple denoising autoencoder training criterion is equivalent to matching the score with respect to the data of a specific energy based model to that of a nonparametric parzen density estimator of the data this yields several useful insights it defines a proper probabilistic model for the denoising autoencoder technique which makes it in principle possible to sample from them or rank examples by their energy it suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives it justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy based models