decreasing the size of the restricted boltzmann machine in this letter we propose a method to decrease the number of hidden units of the restricted boltzmann machine while avoiding a decrease in the performance quantified by the kullback leibler divergence our algorithm is then demonstrated by numerical simulations

bounding the bias of contrastive divergence learning optimization based on k step contrastive divergence cd has become a common way to train restricted boltzmann machines rbms the k step cd is a biased estimator of the log likelihood gradient relying on gibbs sampling we derive a new upper bound for this bias its magnitude depends on k the number of variables in the rbm and the maximum change in energy that can be produced by changing a single variable the last reflects the dependence on the absolute values of the rbm parameters the magnitude of the bias is also affected by the distance in variation between the modeled distribution and the starting distribution of the gibbs chain

large margin classification in infinite neural networks we introduce a new family of positive definite kernels for large margin classification in support vector machines svms these kernels mimic the computation in large neural networks with one layer of hidden units we also show how to derive new kernels by recursive composition that may be viewed as mapping their inputs through a series of nonlinear feature spaces these recursively derived kernels mimic the computation in deep networks with multiple hidden layers we evaluate svms with these kernels on problems designed to illustrate the advantages of deep architectures compared to previous benchmarks we find that on some problems these svms yield state of the art results beating not only other svms but also deep belief nets

representation dimension and quasi hereditary algebras we study auslander s representation dimension of artin algebras which is by definition the minimal projective dimension of coherent functors on modules which are both generators and cogenerators we show the following statements 1 if an artin algebra a is stably hereditary then the representation dimension of a is at most 3 2 if two artin algebras are stably equivalent of morita type then they have the same representation dimension particularly if two self injective algebras are derived equivalent then they have the same representation dimension 3 any incidence algebra of a finite partially ordered set over a field has finite representation dimension moreover we use results on quasi hereditary algebras to show that 4 the auslander algebra of a nakayama algebra has finite representation dimension

deep tsk fuzzy classifier with stacked generalization and triplely concise interpretability guarantee for large data although takagi sugeno kang tsk fuzzy classifier has been applied to a wide range of practical scenarios how to enhance its classification accuracy and interpretability simultaneously is still a challenging task in this paper based on the powerful stacked generalization principle a deep tsk fuzzy classifier d tsk fc is proposed to achieve the enhanced classification accuracy and triplely concise interpretability for fuzzy rules d tsk fc consists of base building units just like the existing popular deep learning d tsk fc can be built in a layer by layer way in terms of the stacked generalization principle the training set plus random shifts obtained from random projections of prediction results of current base building unit are presented as the input of the next base building unit the hidden layer in each base building unit of d tsk fc is represented by triplely concise interpretable fuzzy rules in the sense of randomly selected features with the fixed five fuzzy partitions random rule combinations and the same input space kept in every base building unit of d tsk fc the output layer of each base building unit can be learnt quickly by least learning machine llm besides benefiting from llm d tsk fc s deep learning can be well scaled up for large datasets our extensive experimental results witness the power of the proposed deep tsk fuzzy classifier

learning algorithm for boltzmann machines with additive weight and bias noise this brief presents analytical results on the effect of additive weight bias noise on a boltzmann machine bm in which the unit output is in 1 1 instead of 0 1 with such noise it is found that the state distribution is yet another boltzmann distribution but the temperature factor is elevated thus the desired gradient ascent learning algorithm is derived and the corresponding learning procedure is developed this learning procedure is compared with the learning procedure applied to train a bm with noise it is found that these two procedures are identical therefore the learning algorithm for noise free bms is suitable for implementing as an online learning algorithm for an analog circuit implemented bm even if the variances of the additive weight noise and bias noise are unknown

uniform manifold sampling ums sampling the maximum entropy pdf maximum entropy pdf projection mepp is a way to construct generative models from feature transformations corresponding to each dimension reducing feature mapping such as a feed forward neural network or an algorithm to calculate linear prediction coefficients from time series and given a prior distribution for the features there exists a unique generative model for the input data which subject to mild requirements is maximum entropy maxent among all probability density functions pdfs that are consistent with the given feature prior in this paper we consider the problem of sampling from these maxent projected pdfs the sampling process consists of drawing a sample from the given feature prior distribution then drawing samples uniformly distributed on the inversion set set of input samples consistent with the drawn feature value usually a manifold the process is called uniform manifold sampling ums we describe ums for simple nonlinear and iterative feature transformations then focus on linear transformations with input data constraints x i 0 or 0 leq x i leq 1 which require mcmc based sampling we discover that the manifold centroid sample mean for a fixed feature value is useful as a deterministic maxent feature inversion solution we show how to predict the centroid efficiently without sampling and demonstrate its usefulness in speeding up mcmc by an order of magnitude and in spectral estimation and image reconstruction finally we provide an example of ums in a classification experiment in which we use monte carlo integration to create true generative models from arbitrary classifiers

large margin low rank tensor analysis we present a supervised model for tensor dimensionality reduction which is called large margin low rank tensor analysis lmlrta in contrast to traditional vector representation based dimensionality reduction methods lmlrta can take any order of tensors as input and unlike previous tensor dimensionality reduction methods which can learn only the low dimensional embeddings with a priori specified dimensionality lmlrta can automatically and jointly learn the dimensionality and the low dimensional representations from data moreover lmlrta delivers low rank projection matrices while it encourages data of the same class to be close and of different classes to be separated by a large margin of distance in the low dimensional tensor space lmlrta can be optimized using an iterative fixed point continuation algorithm which is guaranteed to converge to a local optimal solution of the optimization problem we evaluate lmlrta on an object recognition application where the data are represented as 2d tensors and a face recognition application where the data are represented as 3d tensors experimental results show the superiority of lmlrta over state of the art approaches

mesh convolutional restricted boltzmann machines for unsupervised learning of features with structure preservation on 3 d meshes discriminative features of 3 d meshes are significant to many 3 d shape analysis tasks however handcrafted descriptors and traditional unsupervised 3 d feature learning methods suffer from several significant weaknesses 1 the extensive human intervention is involved 2 the local and global structure information of 3 d meshes cannot be preserved which is in fact an important source of discriminability 3 the irregular vertex topology and arbitrary resolution of 3 d meshes do not allow the direct application of the popular deep learning models 4 the orientation is ambiguous on the mesh surface and 5 the effect of rigid and nonrigid transformations on 3 d meshes cannot be eliminated as a remedy we propose a deep learning model with a novel irregular model structure called mesh convolutional restricted boltzmann machines mcrbms mcrbm aims to simultaneously learn structure preserving local and global features from a novel raw representation local function energy distribution in addition multiple mcrbms can be stacked into a deeper model called mesh convolutional deep belief networks mcdbns mcdbn employs a novel local structure preserving convolution lspc strategy to convolve the geometry and the local structure learned by the lower mcrbm to the upper mcrbm lspc facilitates resolving the challenging issue of the orientation ambiguity on the mesh surface in mcdbn experiments using the proposed mcrbm and mcdbn were conducted on three common aspects global shape retrieval partial shape retrieval and shape correspondence results show that the features learned by the proposed methods outperform the other state of the art 3 d shape features

justifying and generalizing contrastive divergence we study an expansion of the log likelihood in undirected graphical models such as the restricted boltzmann machine rbm where each term in the expansion is associated with a sample in a gibbs chain alternating between two random variables the visible vector and the hidden vector in rbms we are particularly interested in estimators of the gradient of the log likelihood obtained through this expansion we show that its residual term converges to zero justifying the use of a truncation running only a short gibbs chain which is the main idea behind the contrastive divergence cd estimator of the log likelihood gradient by truncating even more we obtain a stochastic reconstruction error related through a mean field approximation to the reconstruction error often used to train autoassociators and stacked autoassociators the derivation is not specific to the particular parametric forms used in rbms and requires only convergence of the gibbs chain we present theoretical and empirical evidence linking the number of gibbs steps k and the magnitude of the rbm parameters to the bias in the cd estimator these experiments also suggest that the sign of the cd estimator is correct most of the time even when the bias is large so that cd k is a good descent direction even for small k

deep transfer metric learning conventional metric learning methods usually assume that the training and test samples are captured in similar scenarios so that their distributions are assumed to be the same this assumption does not hold in many real visual recognition applications especially when samples are captured across different data sets in this paper we propose a new deep transfer metric learning dtml method to learn a set of hierarchical nonlinear transformations for cross domain visual recognition by transferring discriminative knowledge from the labeled source domain to the unlabeled target domain specifically our dtml learns a deep metric network by maximizing the inter class variations and minimizing the intra class variations and minimizing the distribution divergence between the source domain and the target domain at the top layer of the network to better exploit the discriminative information from the source domain we further develop a deeply supervised transfer metric learning dstml method by including an additional objective on dtml where the output of both the hidden layers and the top layer are optimized jointly to preserve the local manifold of input data points in the metric space we present two new methods dtml with autoencoder regularization and dstml with autoencoder regularization experimental results on face verification person re identification and handwritten digit recognition validate the effectiveness of the proposed methods

graph regularized restricted boltzmann machine the restricted boltzmann machine rbm has received an increasing amount of interest in recent years it determines good mapping weights that capture useful latent features in an unsupervised manner the rbm and its generalizations have been successfully applied to a variety of image classification and speech recognition tasks however most of the existing rbm based models disregard the preservation of the data manifold structure in many real applications the data generally reside on a low dimensional manifold embedded in high dimensional ambient space in this brief we propose a novel graph regularized rbm to capture features and learning representations explicitly considering the local manifold structure of the data by imposing manifold based locality that preserves constraints on the hidden layer of the rbm the model ultimately learns sparse and discriminative representations the representations can reflect data distributions while simultaneously preserving the local manifold structure of data we test our model using several benchmark image data sets for unsupervised clustering and supervised classification problem the results demonstrate that the performance of our method exceeds the state of the art alternatives

assessing generalization ability of majority vote point classifiers classification algorithms have been traditionally designed to simultaneously reduce errors caused by bias as well by variance however there occur many situations where low generalization error becomes extremely crucial to getting tangible classification solutions and even slight overfitting causes serious consequences in the test results in such situations classifiers with low vapnik chervonenkis vc dimension can bring out positive differences due to two main advantages 1 the classifier manages to keep the test error close to training error and 2 the classifier learns effectively with small number of samples this paper shows that a class of classifiers named majority vote point mvp classifiers on account of very low vc dimension can exhibit a generalization error that is even lower than that of linear classifiers this paper proceeds by theoretically formulating an upper bound on the vc dimension of the mvp classifier later through empirical analysis the trend of exact values of vc dimension is estimated finally case studies on machine fault diagnosis problems and prostate tumor detection problem revalidate the fact that an mvp classifier can achieve a lower generalization error than most other classifiers

stable equivalences of graded algebras abstract we extend the notion of stable equivalence to the class of locally finite graded algebras for such an algebra we focus on the krull schmidt category gr of finitely generated z graded modules with degree 0 maps and the stable category gr obtained by factoring out those maps that factor through a graded projective module we say that and are graded stably equivalent if there is an equivalence gr gr that commutes with the grading shift adapting arguments of auslander and reiten involving functor categories we show that a graded stable equivalence commutes with the syzygy operator where defined and preserves finitely presented modules as a result we see that if is right noetherian resp right graded coherent then so is any graded stably equivalent algebra furthermore if is right noetherian or k is artinian we use almost split sequences to show that a graded stable equivalence preserves finite length modules of particular interest in the nonartinian case we prove that any graded stable equivalence involving an algebra with soc 0 must be a graded morita equivalence

conditional high order boltzmann machines for supervised relation learning relation learning is a fundamental problem in many vision tasks recently high order boltzmann machine and its variants have shown their great potentials in learning various types of data relation in a range of tasks but most of these models are learned in an unsupervised way i e without using relation class labels which are not very discriminative for some challenging tasks e g face verification in this paper with the goal to perform supervised relation learning we introduce relation class labels into conventional high order multiplicative interactions with pairwise input samples and propose a conditional high order boltzmann machine chbm which can learn to classify the data relation in a binary classification way to be able to deal with more complex data relation we develop two improved variants of chbm 1 latent chbm which jointly performs relation feature learning and classification by using a set of latent variables to block the pathway from pairwise input samples to output relation labels and 2 gated chbm which untangles factors of variation in data relation by exploiting a set of latent variables to multiplicatively gate the classification of chbm to reduce the large number of model parameters generated by the multiplicative interactions we approximately factorize high order parameter tensors into multiple matrices then we develop efficient supervised learning algorithms by first pretraining the models using joint likelihood to provide good parameter initialization and then finetuning them using conditional likelihood to enhance the discriminant ability we apply the proposed models to a series of tasks including invariant recognition face verification and action similarity labeling experimental results demonstrate that by exploiting supervised relation labels our models can greatly improve the performance

comparing classification methods for longitudinal fmri studies we compare 10 methods of classifying fmri volumes by applying them to data from a longitudinal study of stroke recovery adaptive fisher s linear and quadratic discriminant gaussian naive bayes support vector machines with linear quadratic and radial basis function rbf kernels logistic regression two novel methods based on pairs of restricted boltzmann machines rbm and k nearest neighbors all methods were tested on three binary classification tasks and their out of sample classification accuracies are compared the relative performance of the methods varies considerably across subjects and classification tasks the best overall performers were adaptive quadratic discriminant support vector machines with rbf kernels and generatively trained pairs of rbms

feature extraction with deep neural networks by a generalized discriminant analysis we present an approach to feature extraction that is a generalization of the classical linear discriminant analysis lda on the basis of deep neural networks dnns as for lda discriminative features generated from independent gaussian class conditionals are assumed this modeling has the advantages that the intrinsic dimensionality of the feature space is bounded by the number of classes and that the optimal discriminant function is linear unfortunately linear transformations are insufficient to extract optimal discriminative features from arbitrarily distributed raw measurements the generalized discriminant analysis gerda proposed in this paper uses nonlinear transformations that are learnt by dnns in a semisupervised fashion we show that the feature extraction based on our approach displays excellent performance on real world recognition and detection tasks such as handwritten digit recognition and face detection in a series of experiments we evaluate gerda features with respect to dimensionality reduction visualization classification and detection moreover we show that gerda dnns can preprocess truly high dimensional input data to low dimensional representations that facilitate accurate predictions even if simple linear predictors or measures of similarity are used

learning rotation invariant local binary descriptor in this paper we propose a rotation invariant local binary descriptor ri lbd learning method for visual recognition compared with hand crafted local binary descriptors such as local binary pattern and its variants which require strong prior knowledge local binary feature learning methods are more efficient and data adaptive unlike existing learning based local binary descriptors such as compact binary face descriptor and simultaneous local binary feature learning and encoding which are susceptible to rotations our ri lbd first categorizes each local patch into a rotational binary pattern rbp and then jointly learns the orientation for each pattern and the projection matrix to obtain ri lbds as all the rotation variants of a patch belong to the same rbp they are rotated into the same orientation and projected into the same binary descriptor then we construct a codebook by a clustering method on the learned binary codes and obtain a histogram feature for each image as the final representation in order to exploit higher order statistical information we extend our ri lbd to the triple rotation invariant co occurrence local binary descriptor trico lbd learning method which learns a triple co occurrence binary code for each local patch extensive experimental results on four different visual recognition tasks including image patch matching texture classification face recognition and scene classification show that our ri lbd and trico lbd outperform most existing local descriptors

image classification using boosted local features with random orientation and location selection we propose an image classification method using boosted randomly selected local features we jointly consider local feature extraction codebook generation and classifier training the proposed method trains a series of classifiers using the boosting strategy experimental results demonstrate effectiveness and efficiency of the proposed method the combination of local features with sparse technique has improved image classification performance dramatically in recent years although very effective this strategy still has two shortcomings first local features are often extracted in a pre defined way e g sift with dense sampling without considering the classification task second the codebook is generated by sparse coding or its variants by minimizing the reconstruction error which has no direct relationships with the classification process to alleviate the two problems we propose a novel boosted local features method with random orientation and location selection we first extract local features with random orientation and location using a weighting strategy this randomization process makes us to extract more types of information for image representation than pre defined methods these extracted local features are then encoded by sparse representation instead of generating the codebook in a single process we construct a series of codebooks and the corresponding encoding parameters of local features using a boosting strategy the weights of local features are determined by the classification performances of learned classifiers in this way we are able to combine the local feature extraction and encoding with classifier training into a unified framework and gradually improve the image classification performance experiments on several public image datasets prove the effectiveness and efficiency of the proposed method

how to center deep boltzmann machines this work analyzes centered restricted boltzmann machines rbms and centered deep boltzmann machines dbms where centering is done by subtracting offset values from visible and hidden variables we show analytically that i centered and normal boltzmann machines bms and thus rbms and dbms are different parameterizations of the same model class such that any normal bm rbm dbm can be transformed to an equivalent centered bm rbm dbm and vice versa and that this equivalence generalizes to artificial neural networks in general ii the expected performance of centered binary bms rbms dbms is invariant under simultaneous flip of data and offsets for any off set value in the range of zero to one iii centering can be reformulated as a different update rule for normal bms rbms dbms and iv using the enhanced gradient is equivalent to setting the offset values to the average over model and data mean furthermore we present numerical simulations suggesting that i optimal generative performance is achieved by subtracting mean values from visible as well as hidden variables ii centered binary rbms dbms reach significantly higher log likelihood values than normal binary rbms dbms iii centering variants whose offsets depend on the model mean like the enhanced gradient suffer from severe divergence problems iv learning is stabilized if an exponentially moving average over the batch means is used for the offset values instead of the current batch mean which also prevents the enhanced gradient from severe divergence v on a similar level of log likelihood values centered binary rbms dbms have smaller weights and bigger bias parameters than normal binary rbms dbms vi centering leads to an update direction that is closer to the natural gradient which is extremely efficient for training as we show for small binary rbms vii centering eliminates the need for greedy layer wise pre training of dbms which often even deteriorates the results independently of whether centering is used or not and ix centering is also beneficial for auto encoders

graph regularized auto encoders for image representation image representation has been intensively explored in the domain of computer vision for its significant influence on the relative tasks such as image clustering and classification it is valuable to learn a low dimensional representation of an image which preserves its inherent information from the original image space at the perspective of manifold learning this is implemented with the local invariant idea to capture the intrinsic low dimensional manifold embedded in the high dimensional input space inspired by the recent successes of deep architectures we propose a local invariant deep nonlinear mapping algorithm called graph regularized auto encoder gae with the graph regularization the proposed method preserves the local connectivity from the original image space to the representation space while the stacked auto encoders provide explicit encoding model for fast inference and powerful expressive capacity for complex modeling theoretical analysis shows that the graph regularizer penalizes the weighted frobenius norm of the jacobian matrix of the encoder mapping where the weight matrix captures the local property in the input space furthermore the underlying effects on the hidden representation space are revealed providing insightful explanation to the advantage of the proposed method finally the experimental results on both clustering and classification tasks demonstrate the effectiveness of our gae as well as the correctness of the proposed theoretical analysis and it also suggests that gae is a superior solution to the current deep representation learning techniques comparing with variant auto encoders and existing local invariant methods

derf distinctive efficient robust features from the biological modeling of the p ganglion cells studies in neuroscience and biological vision have shown that the human retina has strong computational power and its information representation supports vision tasks on both ventral and dorsal pathways in this paper a new local image descriptor termed distinctive efficient robust features derf is derived by modeling the response and distribution properties of the parvocellular projecting ganglion cells in the primate retina derf features exponential scale distribution exponential grid structure and circularly symmetric function difference of gaussian dog used as a convolution kernel all of which are consistent with the characteristics of the ganglion cell array found in neurophysiology anatomy and biophysics in addition a new explanation for local descriptor design is presented from the perspective of wavelet tight frames dog is naturally a wavelet and the structure of the grid points array in our descriptor is closely related to the spatial sampling of wavelets the dog wavelet itself forms a frame and when we modulate the parameters of our descriptor to make the frame tighter the performance of the derf descriptor improves accordingly this is verified by designing a tight frame dog which leads to much better performance extensive experiments conducted in the image matching task on the multiview stereo correspondence data set demonstrate that derf outperforms state of the art methods for both hand crafted and learned descriptors while remaining robust and being much faster to compute

oblique random forest ensemble via least square estimation for time series forecasting abstract recent studies in machine learning indicates that the classifiers most likely to be the bests are the random forests as an ensemble classifier random forest combines multiple decision trees to significant decrease the overall variances conventional random forest employs orthogonal decision tree which selects one optimal feature to split the data instances within a non leaf node according to impurity criteria such as gini impurity information gain and so on however orthogonal decision tree may fail to capture the geometrical structure of the data samples motivated by this we make the first attempt to study the oblique random forest in the context of time series forecasting in each node of the decision tree instead of the single optimal feature based orthogonal classification algorithms used by standard random forest a least square classifier is employed to perform partition the proposed method is advantageous with respect to both efficiency and accuracy we empirically evaluate the proposed method on eight generic time series datasets and five electricity load demand time series datasets from the australian energy market operator and compare with several other benchmark methods

dynamics of learning in mlp natural gradient and singularity revisited the dynamics of supervised learning play a main role in deep learning which takes place in the parameter space of a multilayer perceptron mlp we review the history of supervised stochastic gradient learning focusing on its singular structure and natural gradient the parameter space includes singular regions in which parameters are not identifiable one of our results is a full exploration of the dynamical behaviors of stochastic gradient learning in an elementary singular network the bad news is its pathological nature in which part of the singular region becomes an attractor and another part a repulser at the same time forming a milnor attractor a learning trajectory is attracted by the attractor region staying in it for a long time before it escapes the singular region through the repulser region this is typical of plateau phenomena in learning we demonstrate the strange topology of a singular region by introducing blow down coordinates which are useful for analyzing the natural gradient dynamics we confirm that the natural gradient dynamics are free of critical slowdown the second main result is the good news the interactions of elementary singular networks eliminate the attractor part and the milnor type attractors disappear this explains why large scale networks do not suffer from serious critical slowdowns due to singularities we finally show that the unit wise natural gradient is effective for learning in spite of its low computational cost

a fuzzy restricted boltzmann machine novel learning algorithms based on the crisp possibilistic mean value of fuzzy numbers a fuzzy restricted boltzmann machine frbm is extended from a restricted boltzmann machine rbm by replacing all the real valued parameters with fuzzy numbers a new frbm that employs the crisp possibilistic mean value of a fuzzy number to defuzzify the fuzzy free energy function is presented this approach is much clearer and easier to obtain the expression of the defuzzified free energy function and its approximation than the centroid method several theorems that discuss the error bounds of the approximation to ensure the rationality and validity are also investigated learning algorithms are given for the designed frbm with symmetric triangular fuzzy numbers stfns asymmetric triangular fuzzy numbers and gaussian fuzzy numbers by appropriately choosing the parameters a theorem is concluded that all frbms with symmetric fuzzy numbers will have identical learning algorithm to that of frbms with stfns this is illustrated by a case of frbm with gaussian fuzzy numbers two experiments including the mnist handwriting recognition and the bars and stripes benchmark are carried out the results show that the proposed frbms significantly outperform rbms in learning accuracy and generalization ability especially when encountering unlearned samples and recovering incomplete images

universal approximation depth and errors of narrow belief networks with discrete units we generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well we relax the setting of binary units sutskever hinton 2008 le roux bengio 2008 2010 montufar ay 2011 to units with arbitrary finite state spaces and the vanishing approximation error to an arbitrary approximation error tolerance for example we show that a q ary deep belief network with layers of width for some can approximate any probability distribution on without exceeding a kullback leibler divergence of our analysis covers discrete restricted boltzmann machines and naive bayes models as special cases

discrete restricted boltzmann machines we describe discrete restricted boltzmann machines probabilistic graphical models with bipartite interactions between visible and hidden discrete variables examples are binary restricted boltzmann machines and discrete naive bayes models we detail the inference functions and distributed representations arising in these models in terms of configurations of projected products of simplices and normal fans of products of simplices we bound the number of hidden variables depending on the cardinalities of their state spaces for which these models can approximate any probability distribution on their visible states to any given accuracy in addition we use algebraic methods and coding theory to compute their dimension

deep narrow sigmoid belief networks are universal approximators in this note we show that exponentially deep belief networks can approximate any distribution over binary vectors to arbitrary accuracy even when the width of each layer is limited to the dimensionality of the data we further show that such networks can be greedily learned in an easy yet impractical way

deep linear discriminant analysis on fisher networks person re identification is to seek a correct match for a person of interest across different camera views among a large number of impostors it typically involves two procedures of non linear feature extractions against dramatic appearance changes and subsequent discriminative analysis in order to reduce intra personal variations while enlarging inter personal differences in this paper we introduce a hybrid deep architecture which combines fisher vectors and deep neural networks to learn non linear transformations of pedestrian images to a deep space where data can be linearly separable the proposed method starts from fisher vector encoding which computes a sequence of local feature extraction aggregation and encoding the resulting fisher vector output are fed into stacked supervised layer to seek non linear transformation into a deep space on top of the deep neural network linear discriminant analysis lda is reinforced such that linearly separable latent representations can be learned in an end to end fashion by optimizing an objective function modified from lda the network is enforced to produce feature distributions which have a low variance within the same class and high variance between classes the objective is essentially derived from the general lda eigenvalue problem and allows to train the network with stochastic gradient descent and back propagate lda gradients to compute gaussian mixture model gmm gradients in fisher vector encoding for empirical evaluations we test our approach on four benchmark data sets in person re identification viper 1 cuhk03 2 cuhk01 3 and market 1501 4 extensive experiments on these benchmarks show that our method can achieve state of the art results highlightsa hybrid architecture that combines fisher vectors and deep neural networks an end to end training with linear discriminant analysis as objective deep features are linearly separable and class separability is maximally preserved

a sparse response deep belief network based on rate distortion theory abstract deep belief networks dbns are currently the dominant technique for modeling the architectural depth of brain and can be trained efficiently in a greedy layer wise unsupervised learning manner however dbns without a narrow hidden bottleneck typically produce redundant continuous valued codes and unstructured weight patterns taking inspiration from rate distortion rd theory which encodes original data using as few bits as possible we introduce in this paper a variant of dbn referred to as sparse response dbn sr dbn in this approach kullback leibler divergence between the distribution of data and the equilibrium distribution defined by the building block of dbn is considered as a distortion function and the sparse response regularization induced by l 1 norm of codes is used to achieve a small code rate several experiments by extracting features from different scale image datasets show that our approach sr dbn learns codes with small rate extracts features at multiple levels of abstraction mimicking computations in the cortical hierarchy and obtains more discriminative representation than pca and several basic algorithms of dbns

neural networks retrieving boolean patterns in a sea of gaussian ones restricted boltzmann machines are key tools in machine learning and are described by the energy function of bipartite spin glasses from a statistical mechanical perspective they share the same gibbs measure of hopfield networks for associative memory in this equivalence weights in the former play as patterns in the latter as boltzmann machines usually require real weights to be trained with gradient descent like methods while hopfield networks typically store binary patterns to be able to retrieve the investigation of a mixed hebbian network equipped with both real e g gaussian and discrete e g boolean patterns naturally arises we prove that in the challenging regime of a high storage of real patterns where retrieval is forbidden an additional load of boolean patterns can still be retrieved as long as the ratio between the overall load and the network size does not exceed a critical threshold that turns out to be the same of the standard amit gutfreund sompolinsky theory assuming replica symmetry we study the case of a low load of boolean patterns combining the stochastic stability and hamilton jacobi interpolating techniques the result can be extended to the high load by a non rigorous but standard replica computation argument

a fast learning algorithm for deep belief nets we show how to use complementary priors to eliminate the explaining away effects that make inference difficult in densely connected belief nets that have many hidden layers using complementary priors we derive a fast greedy algorithm that can learn deep directed belief networks one layer at a time provided the top two layers form an undirected associative memory the fast greedy algorithm is used to initialize a slower learning procedure that fine tunes the weights using a contrastive version of the wake sleep algorithm after fine tuning a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels this generative model gives better digit classification than the best discriminative learning algorithms the low dimensional manifolds on which the digits lie are modeled by long ravines in the free energy landscape of the top level associative memory and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind

energy based models for sparse overcomplete representations we present a new way of extending independent components analysis ica to overcomplete representations in contrast to the causal generative extensions of ica which maintain marginal independence of sources we define features as deterministic linear functions of the inputs this assumption results in marginal dependencies among the features but conditional independence of the features given the inputs by assigning energies to the features a probability distribution over the input states is defined through the boltzmann distribution free parameters of this model are trained using the contrastive divergence objective hinton 2002 when the number of features is equal to the number of input dimensions this energy based model reduces to noiseless ica and we show experimentally that the proposed learning algorithm is able to perform blind source separation on speech data in additional experiments we train overcomplete energy based models to extract features from various standard data sets containing speech natural images hand written digits and faces

on the representational power of restricted boltzmann machines for symmetric functions and boolean functions restricted boltzmann machines rbms are used to build deep belief networks that are widely thought to be one of the first effective deep learning neural networks this paper studies the ability of rbms to represent distributions over 0 1 n via softplus hardplus rbm networks it is shown that any distribution whose density depends on the number of 1 s in their input can be approximated with arbitrarily high accuracy by an rbm of size 2n 1 which improves the result of a previous study by reducing the size from n 2 to 2n 1 a theorem for representing partially symmetric boolean functions by softplus rbm networks is established accordingly the representational power of rbms for distributions whose mass represents the boolean functions is investigated in comparison with that of threshold circuits and polynomial threshold functions it is shown that a distribution over 0 1 n whose mass represents a boolean function can be computed with a given margin delta by an rbm of size and parameters bounded by polynomials in n if and only if it can be computed by a depth 2 threshold circuit with size and parameters bounded by polynomials in n

scalability of using restricted boltzmann machines for combinatorial optimization abstract estimation of distribution algorithms edas require flexible probability models that can be efficiently learned and sampled restricted boltzmann machines rbms are generative neural networks with these desired properties we integrate an rbm into an eda and evaluate the performance of this system in solving combinatorial optimization problems with a single objective we assess how the number of fitness evaluations and the cpu time scale with problem size and complexity the results are compared to the bayesian optimization algorithm boa a state of the art multivariate eda and the dependency tree algorithm dta which uses a simpler probability model requiring less computational effort for training the model although rbm eda requires larger population sizes and a larger number of fitness evaluations than boa it outperforms boa in terms of cpu times in particular if the problem is large or complex this is because rbm eda requires less time for model building than boa dta with its restricted model is a good choice for small problems but fails for larger and more difficult problems these results highlight the potential of using generative neural networks for combinatorial optimization

universal approximation results for the temporal restricted boltzmann machine and the recurrent temporal restricted boltzmann machine the restricted boltzmann machine rbm has proved to be a powerful tool in machine learning both on its own and as the building block for deep belief networks multi layer generative graphical models the rbm and deep belief network have been shown to be universal approximators for probability distributions on binary vectors in this paper we prove several similar universal approximation results for two variations of the restricted boltzmann machine with time dependence the temporal restricted boltzmann machine trbm and the recurrent temporal restricted boltzmann machine rtrbm we show that the trbm is a universal approximator for markov chains and generalize the theorem to sequences with longer time dependence we then prove that the rtrbm is a universal approximator for stochastic processes with nite time dependence we conclude with a discussion on efficiency and how the constructions developed could explain some previous experimental results

deep restricted kernel machines using conjugate feature duality the aim of this letter is to propose a theory of deep restricted kernel machines offering new foundations for deep learning with kernel machines from the viewpoint of deep learning it is partially related to restricted boltzmann machines which are characterized by visible and hidden units in a bipartite graph without hidden to hidden connections and deep learning extensions as deep belief networks and deep boltzmann machines from the viewpoint of kernel machines it includes least squares support vector machines for classification and regression kernel principal component analysis pca matrix singular value decomposition and parzen type models a key element is to first characterize these kernel machines in terms of so called conjugate feature duality yielding a representation with visible and hidden units it is shown how this is related to the energy form in restricted boltzmann machines with continuous variables in a nonprobabilistic setting in this new framework of so called restricted kernel machine rkm representations the dual variables correspond to hidden features deep rkm are obtained by coupling the rkms the method is illustrated for deep rkm consisting of three levels with a least squares support vector machine regression level and two kernel pca levels in its primal form also deep feedforward neural networks can be trained within this framework

a spatio temporal rbm based model for facial expression recognition the ability to recognize facial expressions will be an important characteristic of next generation human computer interfaces towards this goal we propose a novel rbm based model to learn effectively the relationships or transformations between image pairs associated with different facial expressions the proposed model has the ability to disentangle these transformations e g pose variations and facial expressions by encoding them into two different hidden sets namely facial expression morphlets and non facial expression morphlets the first hidden set is used to encode facial expression morphlets through a factored four way sub model conditional to label units the second hidden set is used to encode non facial expression morphlets through a factored three way sub model with such a strategy the proposed model can learn transformations between image pairs while disentangling facial expression transformations from non facial expression transformations this is achieved using an algorithm dubbed quadripartite contrastive divergence reported experiments demonstrate the superior performance of the proposed model compared to state of the art highlightsintroducing a novel rbm based model to capture transformations between image pairs disentangling fer transformations from other transformations using two hidden sets introducing a quadripartite contrastive divergence algorithm to learn our model

what regularized auto encoders learn from the data generating distribution what do auto encoders learn about the underlying data generating distribution recent work suggests that some auto encoder variants do a good job of capturing the local manifold structure of data this paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density we show that the auto encoder captures the score derivative of the log density with respect to the input it contradicts previous interpretations of reconstruction error as an energy function unlike previous results the theorems provided here are completely generic and do not depend on the parameterization of the auto encoder they show what the auto encoder would tend to if given enough capacity and examples these results are for a contractive training criterion we show to be similar to the denoising auto encoder training criterion with small corruption noise but with contraction applied on the whole reconstruction function rather than just encoder similarly to score matching one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function finally we show how an approximate metropolis hastings mcmc can be setup to recover samples from the estimated distribution and this is confirmed in sampling experiments

quickly generating representative samples from an rbm derived process two recently proposed learning algorithms herding and fast persistent contrastive divergence fpcd share the following interesting characteristic they exploit changes in the model parameters while sampling in order to escape modes and mix better during the sampling process that is part of the learning algorithm we justify such approaches as ways to escape modes while keeping approximately the same asymptotic distribution of the markov chain in that spirit we extend fpcd using an idea borrowed from herding in order to obtain a pure sampling algorithm which we call the rates fpcd sampler interestingly this sampler can improve the model as we collect more samples since it optimizes a lower bound on the log likelihood of the training data we provide empirical evidence that this new algorithm displays substantially better and more robust mixing than gibbs sampling

relative stable equivalences of morita type in the representation theory of finite groups the stable equivalence of morita type plays a prominent role however except for self injective algebras one does not know much on existence of such equivalences between arbitrary algebras moreover this notion seems to be not general enough to be preserved by classical algebraic constructions in the paper there is introduced a notion of relative stable equivalence of morita type such an equivalence appears in the context of stable equivalence of morita type as is showed in theorem 0 1

resolution of singularities introduced by hierarchical structure in deep neural networks we present a theoretical analysis of singular points of artificial deep neural networks resulting in providing deep neural network models having no critical points introduced by a hierarchical structure it is considered that such deep neural network models have good nature for gradient based optimization first we show that there exist a large number of critical points introduced by a hierarchical structure in deep neural networks as straight lines depending on the number of hidden layers and the number of hidden neurons second we derive a sufficient condition for deep neural networks having no critical points introduced by a hierarchical structure which can be applied to general deep neural networks it is also shown that the existence of critical points introduced by a hierarchical structure is determined by the rank and the regularity of weight matrices for a specific class of deep neural networks finally two kinds of implementation methods of the sufficient conditions to have no critical points are provided one is a learning algorithm that can avoid critical points introduced by the hierarchical structure during learning called avoidant learning algorithm the other is a neural network that does not have some critical points introduced by the hierarchical structure as an inherent property called avoidant neural network

combining multiple dynamic models and deep learning architectures for tracking the left ventricle endocardium in ultrasound data we present a new statistical pattern recognition approach for the problem of left ventricle endocardium tracking in ultrasound data the problem is formulated as a sequential importance resampling algorithm such that the expected segmentation of the current time step is estimated based on the appearance shape and motion models that take into account all previous and current images and previous segmentation contours produced by the method the new appearance and shape models decouple the affine and nonrigid segmentations of the left ventricle to reduce the running time complexity the proposed motion model combines the systole and diastole motion patterns and an observation distribution built by a deep neural network the functionality of our approach is evaluated using a dataset of diseased cases containing 16 sequences and another dataset of normal cases comprised of four sequences where both sets present long axis views of the left ventricle using a training set comprised of diseased and healthy cases we show that our approach produces more accurate results than current state of the art endocardium tracking methods in two test sequences from healthy subjects using three test sequences containing different types of cardiopathies we show that our method correlates well with interuser statistics produced by four cardiologists

mst fitness index and implicit data narratives a comparative test on alternative unsupervised algorithms abstract in this paper we introduce a new methodology for the evaluation of alternative algorithms in capturing the deep statistical structure of datasets of different types and nature called mst fitness and based on the notion of minimum spanning tree mst we test this methodology on six different databases some of which artificial and widely used in similar experimentations and some related to real world phenomena our test set consists of eight different algorithms including some widely known and used such as principal component analysis linear correlation or euclidean distance we moreover consider more sophisticated artificial neural network based algorithms such as the self organizing map som and a relatively new algorithm called auto contractive map autocm we find that for our benchmark of datasets autocm performs consistently better than all other algorithms for all of the datasets and that its global performance is superior to that of the others of several orders of magnitude it is to be checked in future research if autocm can be considered a truly general purpose algorithm for the analysis of heterogeneous categories of datasets

learning invariant color features for person reidentification matching people across multiple camera views known as person reidentification is a challenging problem due to the change in visual appearance caused by varying lighting conditions the perceived color of the subject appears to be different under different illuminations previous works use color as it is or address these challenges by designing color spaces focusing on a specific cue in this paper we propose an approach for learning color patterns from pixels sampled from images across two camera views the intuition behind this work is that even though varying lighting conditions across views affect the pixel values of the same color the final representation of a particular color should be stable and invariant to these variations i e they should be encoded with the same values we model color feature generation as a learning problem by jointly learning a linear transformation and a dictionary to encode pixel values we also analyze different photometric invariant color spaces as well as popular color constancy algorithm for person reidentification using color as the only cue we compare our approach with all the photometric invariant color spaces and show superior performance over all of them combining with other learned low level and high level features we obtain promising results in viper person re id 2011 and caviar4reid data sets

regularized bundle methods for convex and non convex risks machine learning is most often cast as an optimization problem ideally one expects a convex objective function to rely on efficient convex optimizers with nice guarantees such as no local optima yet non convexity is very frequent in practice and it may sometimes be inappropriate to look for convexity at any price alternatively one can decide not to limit a priori the modeling expressivity to models whose learning may be solved by convex optimization and rely on non convex optimization algorithms the main motivation of this work is to provide efficient and scalable algorithms for non convex optimization we focus on regularized unconstrained optimization problems which cover a large number of modern machine learning problems such as logistic regression conditional random fields large margin estimation etc we propose a novel algorithm for minimizing a regularized objective that is able to handle convex and non convex smooth and non smooth risks the algorithm is based on the cutting plane technique and on the idea of exploiting the regularization term in the objective function it may be thought as a limited memory extension of convex regularized bundle methods for dealing with convex and non convex risks in case the risk is convex the algorithm is proved to converge to a stationary solution with accuracy e with a rate o 1 e where is the regularization parameter of the objective function under the assumption of a lipschitz empirical risk in case the risk is not convex getting such a proof is more difficult and requires a stronger and more disputable assumption yet we provide experimental results on artificial test problems and on five standard and difficult machine learning problems that are cast as convex and non convex optimization problems that show how our algorithm compares well in practice with state of the art optimization algorithms

discriminative deep metric learning for face and kinship verification this paper presents a new discriminative deep metric learning ddml method for face and kinship verification in wild conditions while metric learning has achieved reasonably good performance in face and kinship verification most existing metric learning methods aim to learn a single mahalanobis distance metric to maximize the inter class variations and minimize the intra class variations which cannot capture the nonlinear manifold where face images usually lie on to address this we propose a ddml method to train a deep neural network to learn a set of hierarchical nonlinear transformations to project face pairs into the same latent feature space under which the distance of each positive pair is reduced and that of each negative pair is enlarged to better use the commonality of multiple feature descriptors to make all the features more robust for face and kinship verification we develop a discriminative deep multi metric learning method to jointly learn multiple neural networks under which the correlation of different features of each sample is maximized and the distance of each positive pair is reduced and that of each negative pair is enlarged extensive experimental results show that our proposed methods achieve the acceptable results in both face and kinship verification

deep belief networks are compact universal approximators deep belief networks dbn are generative models with many layers of hidden causal variables recently introduced by hinton osindero and teh 2006 along with a greedy layer wise unsupervised learning algorithm building on le roux and bengio 2008 and sutskever and hinton 2008 we show that deep but narrow generative networks do not require more parameters than shallow ones to achieve universal approximation exploiting the proof technique we prove that deep but narrow feedforward neural networks with sigmoidal units can represent any boolean expression

shape matching and object recognition using shape contexts we present a novel approach to measuring similarity between shapes and exploit it for object recognition in our framework the measurement of similarity is preceded by 1 solving for correspondences between points on the two shapes 2 using the correspondences to estimate an aligning transform in order to solve the correspondence problem we attach a descriptor the shape context to each point the shape context at a reference point captures the distribution of the remaining points relative to it thus offering a globally discriminative characterization corresponding points on two similar shapes will have similar shape contexts enabling us to solve for correspondences as an optimal assignment problem given the point correspondences we estimate the transformation that best aligns the two shapes regularized thin plate splines provide a flexible class of transformation maps for this purpose the dissimilarity between the two shapes is computed as a sum of matching errors between corresponding points together with a term measuring the magnitude of the aligning transform we treat recognition in a nearest neighbor classification framework as the problem of finding the stored prototype shape that is maximally similar to that in the image results are presented for silhouettes trademarks handwritten digits and the coil data set

latent structure preserving hashing aiming at efficient similarity search hash functions are designed to embed high dimensional feature descriptors to low dimensional binary codes such that similar descriptors will lead to binary codes with a short distance in the hamming space it is critical to effectively maintain the intrinsic structure and preserve the original information of data in a hashing algorithm in this paper we propose a novel hashing algorithm called latent structure preserving hashing lsph with the target of finding a well structured low dimensional data representation from the original high dimensional data through a novel objective function based on nonnegative matrix factorization nmf with their corresponding kullback leibler divergence of data distribution as the regularization term via exploiting the joint probabilistic distribution of data lsph can automatically learn the latent information and successfully preserve the structure of high dimensional data to further achieve robust performance with complex and nonlinear data in this paper we also contribute a more generalized multi layer lsph ml lsph framework in which hierarchical representations can be effectively learned by a multiplicative up propagation algorithm once obtaining the latent representations the hash functions can be easily acquired through multi variable logistic regression experimental results on three large scale retrieval datasets i e sift 1m gist 1m and 500 k tinyimage show that ml lsph can achieve better performance than the single layer lsph and both of them outperform existing hashing techniques on large scale data

stacked denoising autoencoders learning useful representations in a deep network with a local denoising criterion we explore an original strategy for building deep networks based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs the resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders it is however shown on a benchmark of classification problems to yield significantly lower classification error thus bridging the performance gap with deep belief networks dbn and in several cases surpassing it higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent svm classifiers qualitative experiments show that contrary to ordinary autoencoders denoising autoencoders are able to learn gabor like edge detectors from natural image patches and larger stroke detectors from digit images this work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations

training products of experts by minimizing contrastive divergence it is possible to combine multiple latent variable models of the same data by multiplying their probability distributions together and then renormalizing this way of combining individual expert models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data a product of experts poe is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary training a poe by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule fortunately a poe can be trained using a different objective function called contrastive divergence whose derivatives with regard to the parameters can be approximated accurately and efficiently examples are presented of contrastive divergence learning using several types of expert on several types of data

convergence of contrastive divergence algorithm in exponential family the contrastive divergence cd algorithm has achieved notable success in training energy based models including restricted boltzmann machines and played a key role in the emergence of deep learning the idea of this algorithm is to approximate the intractable term in the exact gradient of the log likelihood function by using short markov chain monte carlo mcmc runs the approximate gradient is computationally cheap but biased whether and why the cd algorithm provides an asymptotically consistent estimate are still open questions this paper studies the asymptotic properties of the cd algorithm in canonical exponential families which are special cases of the energy based model suppose the cd algorithm runs m mcmc transition steps at each iteration t and iteratively generates a sequence of parameter estimates theta t t ge 0 given an i i d data sample x i i 1 n sim p theta star under conditions which are commonly obeyed by the cd algorithm in practice we prove the existence of some bounded m such that any limit point of the time average sum s 0 t 1 theta s t as t to infty is a consistent estimate for the true parameter theta star our proof is based on the fact that theta t t ge 0 is a homogenous markov chain conditional on the data sample x i i 1 n this chain meets the foster lyapunov drift criterion and converges to a random walk around the maximum likelihood estimate the range of the random walk shrinks to zero at rate mathcal o 1 sqrt 3 n as the sample size n to infty

action recognition by spatio temporal oriented energies in this paper we present a unified representation based on the spatio temporal steerable pyramid stsp for the holistic representation of human actions a video sequence is viewed as a spatio temporal volume preserving all the appearance and motion information of an action in it by decomposing the spatio temporal volumes into band passed sub volumes the spatio temporal laplacian pyramid provides an effective technique for multi scale analysis of video sequences and spatio temporal patterns with different scales could be well localized and captured to efficiently explore the underlying local spatio temporal orientation structures at multiple scales a bank of three dimensional separable steerable filters are conducted on each of the sub volume from the laplacian pyramid the outputs of the quadrature pair of steerable filters are squared and summed to yield a more robust oriented energy representation to be further invariant and compact a spatio temporal max pooling operation is performed between responses of the filtering at adjacent scales and over spatio temporal neighbourhoods in order to capture the appearance local geometric structure and motion of an action we apply the stsp on the intensity 3d gradients and optical flow of video sequences yielding a unified holistic representation of human actions taking advantage of multi scale multi orientation analysis and feature pooling stsp produces a compact but informative and invariant representation of human actions we conduct extensive experiments on the kth ucf sports and hmdb51 datasets which shows the unified stsp achieves comparable results with the state of the art methods

kernel reconstruction ica for sparse representation independent component analysis with soft reconstruction cost rica has been recently proposed to linearly learn sparse representation with an overcomplete basis and this technique exhibits promising performance even on unwhitened data however linear rica may not be effective for the majority of real world data because nonlinearly separable data structure pervasively exists in original data space meanwhile rica is essentially an unsupervised method and does not employ class information motivated by the success of the kernel trick that maps a nonlinearly separable data structure into a linearly separable case in a high dimensional feature space we propose a kernel rica krica model to nonlinearly capture sparse representation in feature space furthermore we extend the unsupervised krica to a supervised one by introducing a class driven discrimination constraint such that the data samples from the same class are well represented on the basis of the corresponding subset of basis vectors this discrimination constraint minimizes inhomogeneous representation energy and maximizes homogeneous representation energy simultaneously which is essentially equivalent to maximizing between class scatter and minimizing within class scatter at the same time in an implicit manner experimental results demonstrate that the proposed algorithm is more effective than other state of the art methods on several datasets

on the finitistic dimension conjecture ii related to finite global dimension abstract in this paper we study the finitistic dimensions of artin algebras by establishing a relationship between the global dimensions of the given algebras on the one hand and the finitistic dimensions of their subalgebras on the other hand this is a continuation of the project in j pure appl algebra 193 2004 287 305 for an artin algebra a we denote by gl dim a fin dim a and rep dim a the global dimension finitistic dimension and representation dimension of a respectively the jacobson radical of a is denoted by rad a the main results in the paper are as follows let b be a subalgebra of an artin algebra a such that rad b is a left ideal in a then 1 if gl dim a 4 and rad a rad b a then fin dim b 2 if rep dim a 3 then fin dim b the results are applied to pullbacks of algebras over semi simple algebras moreover we have also the following dual statement 3 let b a be a surjective homomorphism between two algebras b and a suppose that the kernel of is contained in the socle of the right b module b b if gl dim a 4 or rep dim a 3 then fin dim b finally we provide a class of algebras with representation dimension at most three 4 if a is stably hereditary and rad b is an ideal in a then rep dim b 3

construction of approximation spaces for reinforcement learning linear reinforcement learning rl algorithms like least squares temporal difference learning lstd require basis functions that span approximation spaces of potential value functions this article investigates methods to construct these bases from samples we hypothesize that an ideal approximation spaces should encode diffusion distances and that slow feature analysis sfa constructs such spaces to validate our hypothesis we provide theoretical statements about the lstd value approximation error and induced metric of approximation spaces constructed by sfa and the state of the art methods krylov bases and proto value functions pvf in particular we prove that sfa minimizes the average over all tasks in the same environment bound on the above approximation error compared to other methods sfa is very sensitive to sampling and can sometimes fail to encode the whole state space we derive a novel importance sampling modification to compensate for this effect finally the lstd and least squares policy iteration lspi performance of approximation spaces constructed by krylov bases pvf sfa and pca is compared in benchmark tasks and a visual robot navigation experiment both in a realistic simulation and with a robot the results support our hypothesis and suggest that i sfa provides subspace invariant features for mdps with self adjoint transition operators which allows strong guarantees on the approximation error ii the modified sfa algorithm is best suited for lspi in both discrete and continuous state spaces and iii approximation spaces encoding diffusion distances facilitate lspi performance

refinements of universal approximation results for deep belief networks and restricted boltzmann machines we improve recently published results about resources of restricted boltzmann machines rbm and deep belief networks dbn required to make them universal approximators we show that any distribution on the set of binary vectors of length can be arbitrarily well approximated by an rbm with hidden units where is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of in important cases this number is half the cardinality of the support set of given in le roux bengio 2008 we construct a dbn with hidden layers of width that is capable of approximating any distribution on arbitrarily well this confirms a conjecture presented in le roux and bengio 2010

covariance regularized regression and classification for high dimensional problems in recent years many methods have been developed for regression in high dimensional settings we propose covariance regularized regression a family of methods that use a shrunken estimate of the inverse covariance matrix of the features in order to achieve superior prediction an estimate of the inverse covariance matrix is obtained by maximizing its log likelihood under a multivariate normal model subject to a constraint on its elements this estimate is then used to estimate coefficients for the regression of the response onto the features we show that ridge regression the lasso and the elastic net are special cases of covariance regularized regression and we demonstrate that certain previously unexplored forms of covariance regularized regression can outperform existing methods in a range of situations the covariance regularized regression framework is extended to generalized linear models and linear discriminant analysis and is used to analyze gene expression data sets with multiple class and survival outcomes

parameter inference in a probabilistic model from data regulation of transition rate in the monte carlo method abstract we consider the inference of parameters in a probabilistic model from a data set which is generated by an unknown probabilistic model the monte carlo method is a tool for obtaining a data set obeying a given probability distribution a set of transition rates is required to satisfy three conditions irreducible aperiodic and stationary for a sampled data set to represent a probability distribution we utilize the stationary condition of a probability distribution with respect to transition rates to infer parameters a frequency distribution by a data set substitutes for an unknown probability distribution in the condition our method includes minimum probability flow as a special case and becomes superior to it as the number of samples increases

construction of stable equivalences of morita type for finite dimensional algebras i in the representation theory of finite groups the stable equivalence of morita type plays an important role for general finite dimensional algebras this notion is still of particular interest however except for the class of self injective algebras one does not know much on the existence of such equivalences between two finite dimensional algebras in fact even a non trivial example is not known in this paper we provide two methods to produce new stable equivalences of morita type from given ones the main results are corollary 1 2 and theorem 1 3 here the algebras considered are not necessarily self injective as a consequence of our constructions we give an example of a stable equivalence of morita type between two algebras of global dimension 4 such that one of them is quasi hereditary and the other is not this shows that stable equivalences of morita type do not preserve the quasi heredity of algebras as another by product we construct a morita equivalence inside each given stable equivalence of morita type between algebras a and b this leads not only to a general formulation of a result by linckelmann 1996 but also to a nice correspondence of some torsion pairs in a mod with those in b mod if both a and b are symmetric algebras moreover under the assumption of symmetric algebras we can get a new stable equivalence of morita type finally we point out that stable equivalences of morita type are preserved under separable extensions of ground fields

approximation algorithms for training one node relu neural networks training a one node neural network with the relu activation function via optimization which we refer to as the on relu problem is a fundamental problem in machine learning in this paper we begin by proving the np hardness of the on relu problem we then present an approximation algorithm to solve the on relu problem whose running time is mathcal o n k where n is the number of samples and k is a predefined integral constant as an algorithm parameter we analyze the performance of this algorithm under two regimes and show that 1 given any arbitrary set of training samples the algorithm guarantees an n k approximation for the on relu problem to the best of our knowledge this is the first time that an algorithm guarantees an approximation ratio for arbitrary data scenario thus in the ideal case i e when the training error is zero the approximation algorithm achieves the globally optimal solution for the on relu problem and 2 given training sample with gaussian noise the same approximation algorithm achieves a much better asymptotic approximation ratio which is independent of the number of samples n extensive numerical studies show that our approximation algorithm can perform better than the gradient descent algorithm our numerical results also show that the solution of the approximation algorithm can provide a good initialization for gradient descent which can significantly improve the performance

modeling natural images using gated mrfs this paper describes a markov random field for real valued image modeling that has two sets of latent variables one set is used to gate the interactions between all pairs of pixels while the second set determines the mean intensities of each pixel this is a powerful model with a conditional distribution over the input that is gaussian with both mean and covariance determined by the configuration of latent variables which is unlike previous models that were restricted to using gaussians with either a fixed mean or a diagonal covariance matrix thanks to the increased flexibility this gated mrf can generate more realistic samples after training on an unconstrained distribution of high resolution natural images furthermore the latent variables of the model can be inferred efficiently and can be used as very effective descriptors in recognition tasks both generation and discrimination drastically improve as layers of binary latent variables are added to the model yielding a hierarchical model called a deep belief network

representational power of restricted boltzmann machines and deep belief networks deep belief networks dbn are generative neural network models with many layers of hidden explanatory factors recently introduced by hinton osindero and teh 2006 along with a greedy layer wise unsupervised learning algorithm the building block of a dbn is a probabilistic model called a restricted boltzmann machine rbm used to represent one layer of the model restricted boltzmann machines are interesting because inference is easy in them and because they have been successfully used as building blocks for training deeper models we first prove that adding hidden units yields strictly improved modeling power while a second theorem shows that rbms are universal approximators of discrete distributions we then study the question of whether dbns with more layers are strictly more powerful in terms of representational power this suggests a new and less greedy criterion for training rbms within dbns

neighborhood discriminant hashing for large scale image retrieval with the proliferation of large scale community contributed images hashing based approximate nearest neighbor search in huge databases has aroused considerable interest from the fields of computer vision and multimedia in recent years because of its computational and memory efficiency in this paper we propose a novel hashing method named neighborhood discriminant hashing ndh for short to implement approximate similarity search different from the previous work we propose to learn a discriminant hashing function by exploiting local discriminative information i e the labels of a sample can be inherited from the neighbor samples it selects the hashing function is expected to be orthogonal to avoid redundancy in the learned hashing bits as much as possible while an information theoretic regularization is jointly exploited using maximum entropy principle as a consequence the learned hashing function is compact and nonredundant among bits while each bit is highly informative extensive experiments are carried out on four publicly available data sets and the comparison results demonstrate the outperforming performance of the proposed ndh method over state of the art hashing techniques

applicability of n vicinity method for calculation of free energy of ising model abstract here we apply the n vicinity method of approximate calculation of the partition function to an ising model with the nearest neighbor interaction on d dimensional hypercube lattice we solve the equation of state for an arbitrary dimension d and analyze the behavior of the free energy as expected for large dimensions d 3 the system demonstrates a phase transition of the second kind in this case we obtain a simple analytical expression for the critical value of the inverse temperature when 3 d 7 this expression is in a very good agreement with the results of computer simulations in the case of small dimensions d 1 2 there is a noticeable discrepancy with the known exact results

a novel pixel neighborhood differential statistic feature for pedestrian and face detection abstract motivated by the successful application of local binary pattern lbp in this paper we propose a novel pixel neighborhood differential statistic feature for pedestrian and face detection based on the multiple channel maps the calculation of lbp comprises of two steps pixel differential feature pdf calculation and pdf sign encoding the pdf distills discriminative information of local region that can improve the performance of the pedestrian detector but the encoding step degrades the performance due to the quantization error although pdf is more discriminative than original channel maps it has a much higher dimension than the original feature maps and consequently requiring large computation cost to address this issue the pixel neighborhood differential pattern is learned with both supervised and unsupervised learning methods which allow discovering discriminative pixel differential patterns in local area and achieving state of the art results specifically our method firstly aggregates the image channel maps into cell maps with max pooling then pixel neighborhood differential feature based on each channel cell maps are calculated which contributes to encoding discriminative information in each local area and benefits the performance improvements in addition we attempt to learn discriminative differential statistic patterns by using linear discriminative analysis lda and principle component analysis pca for further performance improvement two sets of experiments are conducted on pedestrian detection and face detection respectively the inria caltech and eth datasets are used for pedestrian detection and the fddb and afw datasets for multi view face detection the experimental results show that our method achieves superior performance in comparison with the state of the arts while running at 20 fps for 480 640 images

variational aspects of the reaction in the method of moments the reaction r j sub 1 j sub 2 between two electric surface current densities j sub 1 and j sub 2 is the integral of the dot product of j sub 1 with the electric field produced by j sub 2 jack h richmond see ibid vol 39 no 4 p 473 479 1991 has correctly shown that the reaction is variational a formula is variational if it has a stationary point at the quantity of interest with galerkin s method but is not necessarily variational when the non galerkin element method is applied in the manner where one set of functions is used to expand both j sub 1 and j sub 2 and another set of functions is used to test both the integral equation for j sub 1 and that for j sub 2 in this paper the reaction r j sub 1 j sub 2 is shown to be variational when the non galerkin method is applied in the manner of operation where the set of functions used to expand j sub 2 is the set of functions used to test the integral equation for j sub 1 and the set of functions used to test the integral equation for j sub 2 is the set of functions used to expand j sub 1 hence the manner of operation just described may be more appropriate than richmond s if a variational result is desired a few numerical results are included to compare the performance of both manners of operation

information geometric optimization algorithms a unifying picture via invariance principles we present a canonical way to turn any smooth parametric family of probability distributions on an arbitrary search space x into a continuous time black box optimization method on x the information geometric optimization igo method invariance as a major design principle keeps the number of arbitrary choices to a minimum the resulting igo flow is the flow of an ordinary differential equation conducting the natural gradient ascent of an adaptive time dependent transformation of the objective function it makes no particular assumptions on the objective function to be optimized the igo method produces explicit igo algorithms through time discretization it naturally recovers versions of known algorithms and offers a systematic way to derive new ones in continuous search spaces igo algorithms take a form related to natural evolution strategies nes the cross entropy method is recovered in a particular case with a large time step and can be extended into a smoothed parametrization independent maximum likelihood update igo ml when applied to the family of gaussian distributions on rd the igo framework recovers a version of the well known cma es algorithm and of xnes for the family of bernoulli distributions on 0 1 d we recover the seminal pbil algorithm and cga for the distributions of restricted boltzmann machines we naturally obtain a novel algorithm for discrete optimization on 0 1 d all these algorithms are natural instances of and unified under the single information geometric optimization framework the igo method achieves thanks to its intrinsic formulation maximal invariance properties invariance under reparametrization of the search space x under a change of parameters of the probability distribution and under increasing transformation of the function to be optimized the latter is achieved through an adaptive quantile based formulation of the objective theoretical considerations strongly suggest that igo algorithms are essentially characterized by a minimal change of the distribution over time therefore they have minimal loss in diversity through the course of optimization provided the initial diversity is high first experiments using restricted boltzmann machines confirm this insight as a simple consequence igo seems to provide from information theory an elegant way to simultaneously explore several valleys of a fitness landscape in a single run

unsupervised 3d local feature learning by circle convolutional restricted boltzmann machine extracting local features from 3d shapes is an important and challenging task that usually requires carefully designed 3d shape descriptors however these descriptors are hand crafted and require intensive human intervention with prior knowledge to tackle this issue we propose a novel deep learning model namely circle convolutional restricted boltzmann machine ccrbm for unsupervised 3d local feature learning ccrbm is specially designed to learn from raw 3d representations it effectively overcomes obstacles such as irregular vertex topology orientation ambiguity on the 3d surface and rigid or slightly non rigid transformation invariance in the hierarchical learning of 3d data that cannot be resolved by the existing deep learning models specifically by introducing the novel circle convolution ccrbm holds a novel ring like multi layer structure to learn 3d local features in a structure preserving manner circle convolution convolves across 3d local regions via rotating a novel circular sector convolution window in a consistent circular direction in the process of circle convolution extra points are sampled in each 3d local region and projected onto the tangent plane of the center of the region in this way the projection distances in each sector window are employed to constitute a novel local raw 3d representation called projection distance distribution pdd in addition to eliminate the initial location ambiguity of a sector window the fourier transform modulus is used to transform the pdd into the fourier domain which is then conveyed to ccrbm experiments using the learned local features are conducted on three aspects global shape retrieval partial shape retrieval and shape correspondence the experimental results show that the learned local features outperform other state of the art 3d shape descriptors

greedy learning of binary latent trees inferring latent structures from observations helps to model and possibly also understand underlying data generating processes a rich class of latent structures is the latent trees i e tree structured distributions involving latent variables where the visible variables are leaves these are also called hierarchical latent class hlc models zhang and kocka check end of sentence proposed a search algorithm for learning such models in the spirit of bayesian network structure learning while such an approach can find good solutions it can be computationally expensive as an alternative we investigate two greedy procedures the bin g algorithm determines both the structure of the tree and the cardinality of the latent variables in a bottom up fashion the bin a algorithm first determines the tree structure using agglomerative hierarchical clustering and then determines the cardinality of the latent variables as for bin g we show that even with restricting ourselves to binary trees we obtain hlc models of comparable quality to zhang s solutions in terms of cross validated log likelihood while being generally faster to compute this claim is validated by a comprehensive comparison on several data sets furthermore we demonstrate that our methods are able to estimate interpretable latent structures on real world data with a large number of variables by applying our method to a restricted version of the 20 newsgroups data these models turn out to be related to topic models and on data from the pascal visual object classes voc 2007 challenge we show how such tree structured models help us understand how objects co occur in images for reproducibility of all experiments in this paper all code and data sets or links to data are available at http people kyb tuebingen mpg de harmeling code ltt 1 4 tar

derived equivalences for φ auslander yoneda algebras in this paper we introduce auslander yoneda algebras in a triangulated category with a parameter set in n and present a method to construct new derived equivalences between these auslander yoneda algebras not necessarily artin algebras or their quotient algebras from a given almost stable derived equivalence as consequences of our method we have 1 suppose that a and b are representation finite self injective artin algebras with ax and by additive generators for a and b respectively if a and b are derived equivalent then the auslander yoneda algebras of x and y are derived equivalent for every admissible set in particular the auslander algebras of a and b are both derived equivalent and stably equivalent 2 for a self injective

higman ideal stable hochschild homology and auslander reiten conjecture let a and b be two finite dimensional algebras over an algebraically closed field related to each other by a stable equivalence of morita type we prove that a and b have the same number of isomorphism classes of simple modules if and only if their 0 degree hochschild homology groups hh0 a and hh0 b have the same dimension the first of these two equivalent conditions is claimed by the auslander reiten conjecture for symmetric algebras we will show that the auslander reiten conjecture is equivalent to other dimension equalities involving the centers and the projective centers of a and b this motivates our detailed study of the projective center which now appears to contain the main obstruction to proving the auslander reiten conjecture for symmetric algebras as a by product we get several new invariants of stable equivalences of morita type

latent log linear models for handwritten digit classification we present latent log linear models an extension of log linear models incorporating latent variables and we propose two applications thereof log linear mixture models and image deformation aware log linear models the resulting models are fully discriminative can be trained efficiently and the model complexity can be controlled log linear mixture models offer additional flexibility within the log linear modeling framework unlike previous approaches the image deformation aware model directly considers image deformations and allows for a discriminative training of the deformation parameters both are trained using alternating optimization for certain variants convergence to a stationary point is guaranteed and in practice even variants without this guarantee converge and find models that perform well we tune the methods on the usps data set and evaluate on the mnist data set demonstrating the generalization capabilities of our proposed models our models although using significantly fewer parameters are able to obtain competitive results with models proposed in the literature

a connection between score matching and denoising autoencoders denoising autoencoders have been previously shown to be competitive alternatives to restricted boltzmann machines for unsupervised pretraining of each layer of a deep architecture we show that a simple denoising autoencoder training criterion is equivalent to matching the score with respect to the data of a specific energy based model to that of a nonparametric parzen density estimator of the data this yields several useful insights it defines a proper probabilistic model for the denoising autoencoder technique which makes it in principle possible to sample from them or rank examples by their energy it suggests a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives it justifies the use of tied weights between the encoder and decoder and suggests ways to extend the success of denoising autoencoders to a larger family of energy based models

tractable multivariate binary density estimation and the restricted boltzmann forest we investigate the problem of estimating the density function of multivariate binary data in particular we focus on models for which computing the estimated probability of any data point is tractable in such a setting previous work has mostly concentrated on mixture modeling approaches we argue that for the problem of tractable density estimation the restricted boltzmann machine rbm provides a competitive framework for multivariate binary density modeling with this in mind we also generalize the rbm framework and present the restricted boltzmann forest rbforest which replaces the binary variables in the hidden layer of rbms with groups of tree structured binary variables this extension allows us to obtain models that have more modeling capacity but remain tractable in experiments on several data sets we demonstrate the competitiveness of this approach and study some of its properties