exponentiated gradient algorithms for conditional random fields and max margin markov networks log linear and maximum margin models are two commonly used methods in supervised machine learning and are frequently used in structured prediction problems efficient learning of parameters in these models is therefore an important problem and becomes a key factor when learning from very large data sets this paper describes exponentiated gradient eg algorithms for training such models where eg updates are applied to the convex dual of either the log linear or max margin objective function the dual in both the log linear and max margin cases corresponds to minimizing a convex function with simplex constraints we study both batch and online variants of the algorithm and provide rates of convergence for both cases in the max margin case o 1 e eg updates are required to reach a given accuracy e in the dual in contrast for log linear models only o log 1 e updates are required for both the max margin and log linear cases our bounds suggest that the online eg algorithm requires a factor of n less computation to reach a desired accuracy than the batch eg algorithm where n is the number of training examples our experiments confirm that the online algorithms are much faster than the batch algorithms in practice we describe how the eg updates factor in a convenient way for structured prediction problems allowing the algorithms to be efficiently applied to problems such as sequence learning or natural language parsing we perform extensive evaluation of the algorithms comparing them to l bfgs and stochastic gradient descent for log linear models and to svm struct for max margin models the algorithms are applied to a multi class problem as well as to a more complex large scale parsing task in all these settings the eg algorithms presented here outperform the other methods

fredholm operators semigroups and the asymptotic and boundary behavior of solutions of pdes abstract every semigroup t on a banach space x can be used to define elements u x of exponential type relative to t by requiring that u t s v for some s 0 and v x let then x and y be banach spaces in which the exponential type is characterized by the semigroups t and s respectively and let l l x y be fredholm it is shown that if l satisfies some compatibility conditions with respect to t and s and if f y has exponential type then every solution u x of lu f has exponential type as well when l is a differential operator it is often compatible in this sense and in suitable spaces with semigroups that embody an asymptotic or boundary behavior this yields a way to study such a behavior in solutions of pdes which is technically simple very general and delivers rather sharp results furthermore this approach is easily generalized to the nonlinear setting

the complexity of the homeomorphism relation between compact metric spaces abstract we determine the exact complexity of classifying compact metric spaces up to homeomorphism more precisely the homeomorphism relation on compact metric spaces is borel bi reducible with the complete orbit equivalence relation of polish group actions consequently the same holds for the isomorphism relation between separable commutative c algebras and the isometry relation between c k spaces

towards a theory of classification abstract the well known difficulties arising in a classification which is not set theoretically trivial involving what is sometimes called a non smooth quotient have been overcome in a striking way in the theory of operator algebras by the use of what might be called a classification functor the very existence of which is already a surprise here the notion of such a functor is developed abstractly and a number of examples are considered including those which have arisen for various classes of operator algebras

nonnegative elastic net and application in index tracking this paper deals with the model selection consistency of nonnegative elastic net proposed by imposing nonnegative constraint to the regression parameters in general setting where p the number of predictors q the number of predictors with non zero coefficients in the true linear model and n sample size all go to infinity we prove that this method has nice property of variable selection consistency under neic condition comparing with nonnegative lasso nonnegative elastic net can select the true variables even when nonnegative lasso cannot in empirical part this method is applied to the constrained index tracking problem in stock market without short sales i e tracking csi 300 index1csi 300 index is a capitalization weighted stock market index designed to replicate the performance of 300 stocks traded in the shanghai and shenzhen stock exchanges 1 and sse 180 index2sse 180 index selects constituents with best representation through scientific and objective method sse is a benchmark index reflecting shanghai market and serving as a performance benchmark for investment and a basis for financial innovation 2 by selecting about 30 stocks the results indicate that nonnegative elastic net outperforms nonnegative lasso in asset selection a two step method nonnegative elastic net combined with ols produce better results than simple nonnegative elastic net method

unified development of multiplicative algorithms for linear and quadratic nonnegative matrix factorization multiplicative updates have been widely used in approximative nonnegative matrix factorization nmf optimization because they are convenient to deploy their convergence proof is usually based on the minimization of an auxiliary upper bounding function the construction of which however remains specific and only available for limited types of dissimilarity measures here we make significant progress in developing convergent multiplicative algorithms for nmf first we propose a general approach to derive the auxiliary function for a wide variety of nmf problems as long as the approximation objective can be expressed as a finite sum of monomials with real exponents multiplicative algorithms with theoretical guarantee of monotonically decreasing objective function sequence can thus be obtained the solutions of nmf based on most commonly used dissimilarity measures such as and divergence as well as many other more comprehensive divergences can be derived by the new unified principle second our method is extended to a nonseparable case that includes e g divergence and renyi divergence third we develop multiplicative algorithms for nmf using second order approximative factorizations in which each factorizing matrix may appear twice preliminary numerical experiments demonstrate that the multiplicative algorithms developed using the proposed procedure can achieve satisfactory karush kuhn tucker optimality we also demonstrate nmf problems where algorithms by the conventional method fail to guarantee descent at each iteration but those by our principle are immune to such violation

turbulence orbit equivalence and the classification of nuclear c algebras we bound the borel cardinality of the isomorphism relation for nuclear simple separable c algebras it is turbulent yet borel reducible to the action of the automorphism group of the cuntz algebra o2 on its closed subsets the same bounds are obtained for affine homeomorphism of metrizable choquet simplexes as a by product we recover a result of kechris and solecki namely that homeomorphism of compacta in the hilbert cube is borel reducible to a polish group action these results depend intimately on the classification theory of nuclear simple c algebras by k theory and traces both of necessity and in order to lay the groundwork for further study on the borel complexity of c algebras we prove that many standard c algebra constructions and relations are borel and we prove borel versions of kirchberg s o2 stability and embedding theorems we also find a c algebraic witness for a k hard equivalence relation the authors dedicate this article to the memory of greg hjorth

exponentiated gradient versus gradient descent for linear predictors we consider two algorithm for on line prediction based on a linear model the algorithms are the well known gradient descent gd algorithm and a new algorithm which we call eg they both maintain a weight vector using simple updates for the gd algorithm the update is based on subtracting the gradient of the squared error made on a prediction the eg algorithm uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively we present worst case loss bounds for eg and compare them to previously known bounds for the gd algorithm the bounds suggest that the losses of the algorithms are in general incomparable but eg has a much smaller loss if only a few components of the input are relevant for the predictions we have performed experiments which show that our worst case upper bounds are quite tight already on simple artificial data

multiplicative update rules for concurrent nonnegative matrix factorization and maximum margin classification the state of the art classification methods which employ nonnegative matrix factorization nmf employ two consecutive independent steps the first one performs data transformation dimensionality reduction and the second one classifies the transformed data using classification methods such as nearest neighbor centroid or support vector machines svms in the following we focus on using nmf factorization followed by svm classification typically the parameters of these two steps e g the nmf bases coefficients and the support vectors are optimized independently thus leading to suboptimal classification performance in this paper we merge these two steps into one by incorporating maximum margin classification constraints into the standard nmf optimization the notion behind the proposed framework is to perform nmf while ensuring that the margin between the projected data of the two classes is maximal the concurrent nmf factorization and support vector optimization are performed through a set of multiplicative update rules in the same context the maximum margin classification constraints are imposed on the nmf problem with additional discriminant constraints and respective multiplicative update rules are extracted the impact of the maximum margin classification constraints on the nmf factorization problem is addressed in section vi experimental results in several databases indicate that the incorporation of the maximum margin classification constraints into the nmf and discriminant nmf objective functions improves the accuracy of the classification

inductive limits of interval algebras the tracial state space introduction this is the first in a series of papers where we investigate the structure of the c algebras obtained as inductive limits of interval algebras c 0 1 0m the purpose here is to show that these c algebras comprise a large stock of simple c algebras and that some of them allow endomorphisms with chaotic actions on the tracial state space to be more specific we show that to any dense subgroup g of q containing z and any metrizable choquet simplex s there is a sequence

complete analytic equivalence relations we prove that various concrete analytic equivalence relations arising in model theory or analysis are complete i e maximum in the borel reducibility ordering the proofs use some general results concerning the wider class of analytic quasi orders

multiplicative update rules for incremental training of multiclass support vector machines we present a new method for the incremental training of multiclass support vector machines that can simultaneously modify each class separating hyperplane and provide computational efficiency for training tasks where the training data collection is sequentially enriched and dynamic adaptation of the classifier is required over time an auxiliary function has been designed that incorporates some desired characteristics in order to provide an upper bound for the objective function which summarizes the multiclass classification task a novel set of multiplicative update rules is proposed which is independent from any kind of learning rate parameter provides computational efficiency compared to the conventional batch training approach and is easy to implement convergence to the global minimum is guaranteed since the optimization problem is convex and the global minimizer for the enriched dataset is found using a warm start algorithm experimental evidence on various data collections verified that our method is faster than retraining the classifier from scratch while the achieved classification accuracy rate is maintained at the same level

a proof of convergence of the concave convex procedure using zangwill s theory the concave convex procedure cccp is an iterative algorithm that solves d c difference of convex functions programs as a sequence of convex programs in machine learning cccp is extensively used in many learning algorithms including sparse support vector machines svms transductive svms and sparse principal component analysis though cccp is widely used in many applications its convergence behavior has not gotten a lot of specific attention yuille and rangarajan analyzed its convergence in their original paper however we believe the analysis is not complete the convergence of cccp can be derived from the convergence of the d c algorithm dca proposed in the global optimization literature to solve general d c programs whose proof relies on d c duality in this note we follow a different reasoning and show how zangwill s global convergence theory of iterative algorithms provides a natural framework to prove the convergence of cccp this underlines zangwill s theory as a powerful and general framework to deal with the convergence issues of iterative algorithms after also being used to prove the convergence of algorithms like expectation maximization and generalized alternating minimization in this note we provide a rigorous analysis of the convergence of cccp by addressing two questions when does cccp find a local minimum or a stationary point of the d c program under consideration and when does the sequence generated by cccp converge we also present an open problem on the issue of local convergence of cccp

multi class support vector machine classifiers using intrinsic and penalty graphs in this paper a new multi class classification framework incorporating geometric data relationships described in both intrinsic and penalty graphs in multi class support vector machine is proposed direct solutions are derived for the proposed optimization problem in both the input and arbitrary dimensional hilbert spaces for linear and non linear multi class classification respectively in addition it is shown that the proposed approach constitutes a general framework for svm based multi class classification exploiting geometric data relationships which includes several svm based classification schemes as special cases the power of the proposed approach is demonstrated in the problem of human action recognition in unconstrained environments as well as in facial image and standard classification problems experiments indicate that by exploiting geometric data relationships described in both intrinsic and penalty graphs the svm classification performance can be enhanced highlightswe propose a general class of multiclass svm classifiers we use generic intrinsic and penalty graphs for multiclass svm regularization a new direct solution of the regularized multiclass svm problem is proposed

turbulence representations and trace preserving actions we establish criteria for turbulence in certain spaces of c algebra repre sentations and apply this to the problem of nonclassiability by countable structures for group actions on a standard atomless probability space x and on the hypernite ii1 factor r we also prove that the conjugacy action on the space of free actions of a countably innite amenable group on r is turbulent and that the conjugacy action on the space of ergodic measure preserving ows on x is generically turbulent

nonlinear hyperspectral unmixing with robust nonnegative matrix factorization we introduce a robust mixing model to describe hyperspectral data resulting from the mixture of several pure spectral signatures the new model extends the commonly used linear mixing model by introducing an additional term accounting for possible nonlinear effects that are treated as sparsely distributed additive outliers with the standard nonnegativity and sum to one constraints inherent to spectral unmixing our model leads to a new form of robust nonnegative matrix factorization with a group sparse outlier term the factorization is posed as an optimization problem which is addressed with a block coordinate descent algorithm involving majorization minimization updates simulation results obtained on synthetic and real data show that the proposed strategy competes with the state of the art linear and nonlinear unmixing methods

nonnegative sparse locality preserving hashing it is a np hard problem to optimize the objective function of hash based similarity search algorithms such as spectral hashing and self taught hashing to make the problem solvable existing methods have relaxed the constraints on hash codes from binary values discrete to real values continuous then eigenvalue decomposition was employed to achieve the relaxed real solution the main problem is that the signs of the relaxed continuous solution are mixed such results may deviate severely from the true solution which has lead to significant semantic loss moreover eigenvalue decomposition confronts singularity problem when the dimension of the data is larger than the sample size to address these problems we propose a novel method named nonnegative sparse locality preserving hashing nslph nonnegative and sparse constraints are imposed for a more accurate solution which preserves semantic information well then we have applied nonnegative quadratic programming and multiplicative updating to solve the optimization problem which successfully avoids the singularity problem of the eigenvalue decomposition the extensive experiments presented in this paper demonstrate that the proposed approach outperforms the state of the art algorithms

fredholm and regularity theory of douglis nirenberg elliptic systems on mathbb r n we give a fairly complete exposition of the fredholm properties of the douglis nirenberg elliptic systems on mathbb r n in the classical unweighted lp sobolev spaces and under minimal assumptions about the coefficients these assumptions rule out the use of classical pseudodifferential operator theory although it is indirectly of assistance in places after generalizing a necessary and sufficient condition for fredholmness already known in special cases various invariance properties are established index null space etc with respect to p and the douglis nirenberg numbers among other things this requires getting around the problem that the lp spaces are not ordered by inclusion in turn with some work invariance leads to a regularity theory more general than what can be obtained by the method of differential quotients

projection free parallel quadratic programming for linear model predictive control a key component in enabling the application of model predictive control mpc in fields such as automotive aerospace and factory automation is the availability of low complexity fast optimisation algorithms to solve the mpc finite horizon optimal control problem in architectures with reduced computational capabilities in this paper we introduce a projection free iterative optimisation algorithm and discuss its application to linear mpc the algorithm originally developed by brand for non negative quadratic programs is based on a multiplicative update rule and it is shown to converge to a fixed point which is the optimum an acceleration technique based on a projection free line search is also introduced to speed up the convergence to the optimum the algorithm is applied to mpc through the dual of the quadratic program qp formulated from the mpc finite time optimal control problem we discuss how termination conditions with guaranteed degree of suboptimality can be enforced and how the algorithm

symmetric nonnegative matrix factorization algorithms and applications to probabilistic clustering nonnegative matrix factorization nmf is an unsupervised learning method useful in various applications including image processing and semantic analysis of documents this paper focuses on symmetric nmf snmf which is a special case of nmf decomposition three parallel multiplicative update algorithms using level 3 basic linear algebra subprograms directly are developed for this problem first by minimizing the euclidean distance a multiplicative update algorithm is proposed and its convergence under mild conditions is proved based on it we further propose another two fast parallel methods snmf and snmf algorithms all of them are easy to implement these algorithms are applied to probabilistic clustering we demonstrate their effectiveness for facial image clustering document categorization and pattern clustering in gene expression

polish groupoids and functorial complexity we introduce and study the notion of functorial borel complexity for polish groupoids such a notion aims to measure the complexity of classifying the objects of a category in a constructive and functorial way in the particular case of principal groupoids such a notion coincides with the usual borel complexity of equivalence relations our main result is that on one hand for polish groupoids with an essentially treeable orbit equivalence relation functorial borel complexity coincides with the borel complexity of the associated orbit equivalence relation on the other hand for every countable equivalence relation e that is not treeable there are polish groupoids with different functorial borel complexity both having e as orbit equivalence relation in order to obtain such a conclusion we generalize some fundamental results about the descriptive set theory of polish group actions to actions of polish groupoids answering a question of arlan ramsay these include the becker kechris results on polishability of borel g spaces existence of universal borel g spaces and characterization of borel g spaces with borel orbit equivalence relations

regularity properties in the classification program for separable amenable c algebras we report on recent progress in the program to classify separable amenable c algebras our emphasis is on the newly apparent role of regularity properties such as finite decomposition rank strict comparison of positive elements and z stability and on the importance of the cuntz semigroup we include a brief history of the program s successes since 1989 a more detailed look at the villadsen type algebras which have so dramatically changed the landscape and a collection of announcements on the structure and properties of the cuntz semigroup

measure reducibility of countable borel equivalence relations we show that every basis for the countable borel equivalence relations strictly above mathbb e 0 under measure reducibility is uncountable thereby ruling out natural generalizations of the glimm effros dichotomy we also push many known results concerning the abstract structure of the measure reducibility hierarchy to its base using arguments substantially simpler than those previously employed

completeness of the isomorphism problem for separable c algebras this paper studies the descriptive set theoretical complexity of the isomorphism problem for separable c algebras we prove that the isomorphism problem for separable simple ai c algebras is complete in the class of orbit equivalence relations this means that any isomorphism problem arising from a continuous action of a separable completely metrizable group can be reduced to the isomorphism of simple separable ai c algebras

unitary equivalence of automorphisms of separable c algebras abstract we prove that the automorphisms of any separable c algebra that does not have continuous trace are not classifiable by countable structures up to unitary equivalence this implies a dichotomy for the borel complexity of the relation of unitary equivalence of automorphisms of a separable unital c algebra such relation is either smooth or not even classifiable by countable structures

actions of rigid groups on uhf algebras let be a countably infinite property t group and let d be uhf algebra of infinite type we prove that there exists a continuum of pairwise non weakly cocycle conjugate strongly outer actions of on d the proof consists in assigning to any second countable abelian pro p group g a strongly outer action of on d whose weak cocycle conjugacy class completely remembers the group g the group g is reconstructed from the action through its weak 1 cohomology set endowed with a canonical pairing function our construction also shows the following stronger statement the relations of conjugacy cocycle conjugacy and weak cocycle conjugacy of strongly outer actions of on d are complete analytic sets and in particular not borel the same conclusions hold more generally when is only assumed to contain an infinite subgroup with relative property t and for actions on not necessarily simple separable nuclear uhf absorbing self absorbing c algebras with at least one trace finally we use the techniques of this paper to construct outer actions on r with prescribed cohomology precisely for every infinite property t group and for every countable abelian group we construct an outer action of on r whose 1 cohomology is isomorphic to

nonnegative tensor cofactorization and its unified solution in this paper we present a new joint factorization algorithm called nonnegative tensor co factorization ntcof the key idea is to simultaneously factorize multiple visual features of the same data into nonnegative dimensionality reduced representations and meanwhile to maximize the correlations of the low dimensional representations the data is generally encoded as tensors of arbitrary order rather than vectors to preserve the original data structures ntcof provides a simple and efficient way to fuse multiple complementary features for enhancing the discriminative power of the desired rank reduced representations under the nonnegative constraints we formulate the related objectives with a block wise quadratic nonnegative function to optimize a unified convergence provable solution is developed this solution is applicable for any nonnegative optimization problems with block wise quadratic objective functions and thus offer an unified platform based on which specific solution can be directly derived by skipping over tedious proof about algorithmic convergence we apply the proposed algorithm and solution on three image tasks face recognition multi class image categorization and multi label image annotation results with comparisons on public challenging datasets show that the proposed algorithm can outperform both the traditional nonnegative methods and the popular feature combination methods

multiplicative updates for nonnegative quadratic programming many problems in neural computation and statistical learning involve optimizations with nonnegativity constraints in this article we study convex problems in quadratic programming where the optimization is confined to an axis aligned region in the nonnegative orthant for these problems we derive multiplicative updates that improve the value of the objective function at each iteration and converge monotonically to the global minimum the updates have a simple closed form and do not involve any heuristics or free parameters that must be tuned to ensure convergence despite their simplicity they differ strikingly in form from other multiplicative updates used in machine learning we provide complete proofs of convergence for these updates and describe their application to problems in signal processing and pattern recognition

local coordinate concept factorization for image representation learning sparse representation of high dimensional data is a state of the art method for modeling data matrix factorization based techniques such as nonnegative matrix factorization and concept factorization cf have shown great advantages in this area especially useful for image representation both of them are linear learning problems and lead to a sparse representation of the images however the sparsity obtained by these methods does not always satisfy locality conditions for example the learned new basis vectors may be relatively far away from the original data thus we may not be able to achieve the optimal performance when using the new representation for other learning tasks such as classification and clustering in this paper we introduce a locality constraint into the traditional cf by requiring the concepts basis vectors to be as close to the original data points as possible each datum can be represented by a linear combination of only a few basis concepts thus our method is able to achieve sparsity and locality simultaneously we analyze the complexity of our novel algorithm and demonstrate the effectiveness in comparison with the state of the art approaches through a set of evaluations based on real world applications