arbitrary norm support vector machines support vector machines svm are state of the art classifiers typically l2 norm or l1 norm is adopted as a regularization term in svms while other norm based svms for example the l0 norm svm or even the l norm svm are rarely seen in the literature the major reason is that l0 norm describes a discontinuous and nonconvex term leading to a combinatorially np hard optimization problem in this letter motivated by bayesian learning we propose a novel framework that can implement arbitrary norm based svms in polynomial time one significant feature of this framework is that only a sequence of sequential minimal optimization problems needs to be solved thus making it practical in many real applications the proposed framework is important in the sense that bayesian priors can be efficiently plugged into most learning methods without knowing the explicit form hence this builds a connection between bayesian learning and the kernel machines we derive the theoretical framework demonstrate how our approach works on the l0 norm svm as a typical example and perform a series of experiments to validate its advantages experimental results on nine benchmark data sets are very encouraging the implemented l0 norm is competitive with or even better than the standard l2 norm svm in terms of accuracy but with a reduced number of support vectors 9 46 of the number on average when compared with another sparse model the relevance vector machine our proposed algorithm also demonstrates better sparse properties with a training speed over seven times faster

structured multicategory support vector machines with analysis of variance decomposition the support vector machine has been a popular choice of classification method for many applications in machine learning while it often outperforms other methods in terms of classification accuracy the implicit nature of its solution renders the support vector machine less attractive in providing insights into the relationship between covariates and classes use of structured kernels can remedy the drawback borrowing the flexible model building idea of functional analysis of variance decomposition we consider multicategory support vector machines with analysis of variance kernels in this paper an additional penalty is imposed on the sum of weights of functional subspaces which encourages a sparse representation of the solution incorporation of the additional penalty enhances the interpretability of a resulting classifier with often improved accuracy the proposed method is demonstrated through simulation studies and an application to real data

double regularization methods for robust feature selection and svm classification via dc programming abstract in this work two novel formulations for embedded feature selection are presented a second order cone programming approach for support vector machines is extended by adding a second regularizer to encourage feature elimination the one and the zero norm penalties are used in combination with the tikhonov regularization under a robust setting designed to correctly classify instances up to a predefined error rate even for the worst data distribution the use of the zero norm leads to a nonconvex formulation which is solved by using difference of convex dc functions extending dc programming to second order cones experiments on high dimensional microarray datasets were performed and the best performance was obtained with our approaches compared with well known feature selection methods for support vector machines

a threshold of ln n for approximating set cover given a collection f of subsets of s 1 n setcover is the problem of selecting as few as possiblesubsets from f such that their union covers s and maxk cover is the problem of selecting k subsets from f such that their union has maximum cardinality both these problems arenp hard we prove that 1 o 1 ln n is a threshold below which setcover cannot be approximated efficiently unless np has slightlysuperpolynomial time algorithms this closes the gap up to low orderterms between the ratio of approximation achievable by the greedyalogorithm which is 1 o 1 lnn and provious results of lund and yanakakis that showed hardness ofapproximation within a ratio of log 2 n 2s0 72 ln n for max k cover we show an approximationthreshold of 1 1 e up tolow order terms under assumption that p np

feature selection using structural similarity a new method of feature selection is developed based on structural similarity the topological neighborhood information about pairs of objects or patterns to partition s is taken into consideration while computing a measure of structural similarity this is termed proximity and is defined in terms of membership values multi objective evolutionary optimization is employed to arrive at a consensus solution in terms of the contradictory criteria pair involving fuzzy proximity and feature set cardinality results for real and synthetic datasets of low medium and high dimensionality show that the method led to a correct selection of the reduced feature subset comparative study is also provided and quantified in terms of accuracy of classification and clustering validity indices

on the hardness of approximating minimization problems we prove results indicating that it is hard to compute efficiently good approximate solutions to the graph coloring set covering and other related minimization problems specifically there is an e 0 such that graph coloring cannot be approximated with ratio n e unless p np set covering cannot be approximated with ratio c log n for any c 1 4 unless np is contained in dtime n poly log n similar results follow for related problems such as clique cover fractional chromatic number dominating set and others

on a nonlinear elliptic eigenvalue problem abstract let n be the number of the solutions of the equation u f u in u 0 on where is a bounded domain in r n n 2 with smooth boundary under suitable conditions on f we proved that n as

minimization on spheres of coercive functionals on w01 2 ω abstract we deal with the minimization on spheres of functionals defined on the space w 0 1 2 we apply the main results to establish a multiplicity theorem for a dirichlet problem involving equations of the type u h x u s 2 u g x u here is the laplacian operator is a real parameter s 1 2 h c is a function which is allowed to be sign changing and g r r r n is a caratheodory function having no growth conditions with respect to the second variable

ep gig priors and applications in bayesian sparse learning in this paper we propose a novel framework for the construction of sparsity inducing priors in particular we define such priors as a mixture of exponential power distributions with a generalized inverse gaussian density ep gig ep gig is a variant of generalized hyperbolic distributions and the special cases include gaussian scale mixtures and laplace scale mixtures furthermore laplace scale mixtures can subserve a bayesian framework for sparse learning with nonconvex penalization the densities of ep gig can be explicitly expressed moreover the corresponding posterior distribution also follows a generalized inverse gaussian distribution we exploit these properties to develop em algorithms for sparse empirical bayesian learning we also show that these algorithms bear an interesting resemblance to iteratively reweighted l2 or l1 methods finally we present two extensions for grouped variable selection and logistic regression

sparse multiple kernel learning for signal processing applications in many signal processing applications grouping of features during model development and the selection of a small number of relevant groups can be useful to improve the interpretability of the learned parameters while a lot of work based on linear models has been reported to solve this problem in the last few years multiple kernel learning has come up as a candidate to solve this problem in nonlinear models since all of the multiple kernel learning algorithms to date use convex primal problem formulations the kernel weights selected by these algorithms are not strictly the sparsest possible solution the main reason for using a convex primal formulation is that efficient implementations of kernel based methods invariably rely on solving the dual problem this work proposes the use of an additional log based concave penalty term in the primal problem to induce sparsity in terms of groups of parameters a generalized iterative learning algorithm which can be used with a linear combination of this concave penalty term with other penalty terms is given for model parameter estimation in the primal space it is then shown that a natural extension of the method to nonlinear models using the kernel trick results in a new algorithm called sparse multiple kernel learning smkl which generalizes group feature selection to kernel selection smkl is capable of exploiting existing efficient single kernel algorithms while providing a sparser solution in terms of the number of kernels used as compared to the existing multiple kernel learning framework a number of signal processing examples based on the use of mass spectra for cancer detection hyperspectral imagery for land cover classification and nir spectra from wheat fescue grass and diesel are given to highlight the ability of smkl to achieve a very high accuracy with a very few kernels

variational elliptic problems which are nonquadratic at infinity p co if n 1 2 then it is well known that the weak solutions u e h 2 of p are the critical points of the c functional j u ulvul dx jcf x u dx where f x s jif x t dt in this variational setting the literature usually distinguishes between two situations for prob lem p the subquadratic situation where the potential f satisfies lim suplsl f x s s2 i c m and the superquadratic situation where f satisfies limlsl m f x s s2 00 the main goal of this paper is to present a unified approach to both situations by means of a condition of nonquadraticity at infinity on f more precisely our hypotheses are as follows

stratified sampling for feature subspace selection in random forests for high dimensional data for high dimensional data a large portion of features are often not informative of the class of the objects random forest algorithms tend to use a simple random sampling of features in building their decision trees and consequently select many subspaces that contain few if any informative features in this paper we propose a stratified sampling method to select the feature subspaces for random forests with high dimensional data the key idea is to stratify features into two groups one group will contain strong informative features and the other weak informative features then for feature subspace selection we randomly select features from each group proportionally the advantage of stratified sampling is that we can ensure that each subspace contains enough informative features for classification in high dimensional data testing on both synthetic data and various real data sets in gene classification image categorization and face recognition data sets consistently demonstrates the effectiveness of this new method the performance is shown to better that of state of the art algorithms including svm the four variants of random forests rf ert enrich rf and oblique rf and nearest neighbor nn algorithms

a thermal perceptron learning rule the thermal perceptron is a simple extension to rosenblatt s perceptron learning rule for training individual linear threshold units it finds stable weights for nonseparable problems as well as separable ones experiments indicate that if a good initial setting for a temperature parameter to has been found then the thermal perceptron outperforms the pocket algorithm and methods based on gradient descent the learning rule stabilizes the weights learns over a fixed training period for separable problems it finds separating weights much more quickly than the usual rules

radial solutions of dirichlet problems with concave convex nonlinearities abstract we prove the existence of a double infinite sequence of radial solutions for a dirichlet concave convex problem associated with an elliptic equation in a ball of r n we are interested in relaxing the classical positivity condition on the weights by allowing the weights to vanish the idea is to develop a topological method and to use the concept of rotation number the solutions are characterized by their nodal properties

speech recognition apparatus having a speech coder outputting acoustic prototype ranks a speech coding and speech recognition apparatus the value of at least one feature of an utterance is measured over each of a series of successive time intervals to produce a series of feature vector signals the closeness of the feature value of each feature vector signal to the parameter value of each of a set of prototype vector signals is determined to obtain prototype match scores for each vector signal and each prototype vector signal for each feature vector signal first rank and second rank scores are associated with the prototype vector signals having the best and second best prototype match scores respectively for each feature vector signal at least the identification value and the rank score of the first ranked and second ranked prototype vector signals are output as a coded utterance representation signal of the feature vector signal to produce a series of coded utterance representation signals for each of a plurality of speech units a probabilistic model has a plurality of model outputs and output probabilities for each model output each model output comprises the identification value of a prototype vector and a rank score for each speech unit a match score comprises an estimate of the probability that the probabilistic model of the speech unit would output a series of model outputs matching a reference series comprising the identification value and rank score of at least one prototype vector from each coded utterance representation signal in the series of coded utterance representation signals

empirical risk minimization for support vector classifiers in this paper we propose a general technique for solving support vector classifiers svcs for an arbitrary loss function relying on the application of an iterative reweighted least squares irwls procedure we further show that three properties of the svc solution can be written as conditions over the loss function this technique allows the implementation of the empirical risk minimization erm inductive principle on large margin classifiers obtaining at the same time very compact in terms of number of support vectors solutions the improvements obtained by changing the svc loss function are illustrated with synthetic and real data examples

feature selection for high dimensional class imbalanced data sets using support vector machines feature selection and classification of imbalanced data sets are two of the most interesting machine learning challenges attracting a growing attention from both industry and academia feature selection addresses the dimensionality reduction problem by determining a subset of available features to build a good model for classification or prediction while the class imbalance problem arises when the class distribution is too skewed both issues have been independently studied in the literature and a plethora of methods to address high dimensionality as well as class imbalance has been proposed the aim of this work is to simultaneously explore both issues proposing a family of methods that select those attributes that are relevant for the identification of the target class in binary classification we propose a backward elimination approach based on successive holdout steps whose contribution measure is based on a balanced loss function obtained on an independent subset our experiments are based on six highly imbalanced microarray data sets comparing our methods with well known feature selection techniques and obtaining a better prediction with consistently fewer relevant features

an evidential classifier based on feature selection and two step classification strategy in this paper we investigate ways to learn efficiently from uncertain data using belief functions in order to extract more knowledge from imperfect and insufficient information and to improve classification accuracy we propose a supervised learning method composed of a feature selection procedure and a two step classification strategy using training information the proposed feature selection procedure automatically determines the most informative feature subset by minimizing an objective function the proposed two step classification strategy further improves the decision making accuracy by using complementary information obtained during the classification process the performance of the proposed method was evaluated on various synthetic and real datasets a comparison with other classification methods is also presented highlightsa classifier is based on belief functions to tackle uncertain data the classifier composed by feature selection and a two step classification a new combination rule to better represent data uncertainty a new feature selection is based on minimizing uncertainty with sparse constraint two step classification improving accuracy of decision making

non differentiable functionals and applications to elliptic problems with discontinuous nonlinearities abstract multiple critical points theorems for non differentiable functionals are established applications both to elliptic variational hemivariational inequalities and eigenvalue problems with discontinuous nonlinearities are then presented

continuous speech recognition method and device in a system for recognizing a continuous speech pattern based on a speech pattern entered through a microphone a feature vector alpha i is extracted by a feature extraction unit and a feature vector beta jn is read out from a reference pattern memory a first recursive operation unit dpm1 computes a set of similarity measures sn i j between the feature vectors a maximum similarity measure at a time point i is determined and produced by a first decision unit dsc1 and is stored in a maximum similarity memory max a second recursive operation unit dmp2 computes a reversed similarity measure based on a computed result g v l and the output from said maximum similarity measure memory a second decision unit determines a boundary vmax a word wu based on data u vmax 1 obtained by the boundary v is stored as nx x 1 y in an order reversing unit rev the order reversing unit finally reverses the order of data and produces an output nx x 1 y

efficient sparse generalized multiple kernel learning kernel methods have been successfully applied in various applications to succeed in these applications it is crucial to learn a good kernel representation whose objective is to reveal the data similarity precisely in this paper we address the problem of multiple kernel learning mkl searching for the optimal kernel combination weights through maximizing a generalized performance measure most mkl methods employ the norm simplex constraints on the kernel combination weights which therefore involve a sparse but non smooth solution for the kernel weights despite the success of their efficiency they tend to discard informative complementary or orthogonal base kernels and yield degenerated generalization performance alternatively imposing the norm constraint on the kernel weights will keep all the information in the base kernels this leads to non sparse solutions and brings the risk of being sensitive to noise and incorporating redundant information to tackle these problems we propose a generalized mkl gmkl model by introducing an elastic net type constraint on the kernel weights more specifically it is an mkl model with a constraint on a linear combination of the norm and the squared norm on the kernel weights to seek the optimal kernel combination weights therefore previous mkl problems based on the norm or the norm constraints can be regarded as special cases furthermore our gmkl enjoys the favorable sparsity property on the solution and also facilitates the grouping effect moreover the optimization of our gmkl is a convex optimization problem where a local solution is the global optimal solution we further derive a level method to efficiently solve the optimization problem a series of experiments on both synthetic and real world datasets have been conducted to show the effectiveness and efficiency of our gmkl

dissimilarity metric learning in the belief function framework the evidential k nearest neighbor ek nn method provided a global treatment of imperfect knowledge regarding the class membership of training patterns it has outperformed traditional k nn rules in many applications but still shares some of their basic limitations e g 1 classification accuracy depends heavily on how to quantify the dissimilarity between different patterns and 2 no guarantee for satisfactory performance when training patterns contain unreliable imprecise and or uncertain input features in this paper we propose to address these issues by learning a suitable metric using a low dimensional transformation of the input space so as to maximize both the accuracy and efficiency of the ek nn classification to this end a novel loss function to learn the dissimilarity metric is constructed it consists of two terms the first one quantifies the imprecision regarding the class membership of each training pattern while by means of feature selection the second one controls the influence of unreliable input features on the output linear transformation the proposed method has been compared with some other metric learning methods on several synthetic and real datasets it consistently led to comparable performance with regard to testing accuracy and class structure visualization

dc programming and dca thirty years of developments the year 2015 marks the 30th birthday of dc difference of convex functions programming and dca dc algorithms which constitute the backbone of nonconvex programming and global optimization in this article we offer a short survey on thirty years of developments of these theoretical and algorithmic tools the survey is comprised of three parts in the first part we present a brief history of the field while in the second we summarize the state of the art results and recent advances we focus on main theoretical results and dca solvers for important classes of difficult nonconvex optimization problems and then give an overview of real world applications whose solution methods are based on dca the third part is devoted to new trends and important open issues as well as suggestions for future developments

feature selection via dependence maximization we introduce a framework for feature selection based on dependence maximization between the selected features and the labels of an estimation problem using the hilbert schmidt independence criterion the key idea is that good features should be highly dependent on the labels our approach leads to a greedy procedure for feature selection we show that a number of existing feature selectors are special cases of this framework experiments on both artificial and real world data show that our feature selector works well in practice

normalization of speech by adaptive labelling in a speech processor system in which prototype vectors of speech are generated by an acoustic processor under reference noise and known ambient conditions and in which feature vectors of speech are generated during varying noise and other ambient and recording conditions normalized vectors are generated to reflect the form the feature vectors would have if generated under the reference conditions the normalized vectors are generated by a applying an operator function a i to a set of feature vectors x occurring at or before time interval i to yield a normalized vector y i a i x b determining a distance error vector e i by which the normalized vector is projectively moved toward the closest prototype vector to the normalized vector y i c up dating the operator function for next time interval to correspond to the most recently determined distance error vector and d incrementing i to the next time interval and repeating steps a through d wherein the feature vector corresponding to the incremented i value has the most recent up dated operator function applied thereto with successive time intervals successive normalized vectors are generated based on a successively up dated operator function for each normalized vector the closest prototype thereto is associated therewith the string of normalized vectors or the string of associated prototypes or respective label identifiers thereof or both provide output from the acoustic processor

perturbations from symmetric elliptic boundary value problems abstract we study the multiplicity of solutions for the elliptic problem u f x u eg x u in and u 0 on where e is a parameter is a smooth bounded domain in r n f c r f x t is odd with respect to t and g c r under suitable conditions only on f we prove that for any j n there exists ej 0 such that if e ej then the above problem possesses at least j distinct solutions

on the complexity of learning lexicographic strategies fast and frugal heuristics are well studied models of bounded rationality psychological research has proposed the take the best heuristic as a successful strategy in decision making with limited resources take the best searches for a sufficiently good ordering of cues or features in a task where objects are to be compared lexicographically we investigate the computational complexity of finding optimal cue permutations for lexicographic strategies and prove that the problem is np complete it follows that no efficient that is polynomial time algorithm computes optimal solutions unless p np we further analyze the complexity of approximating optimal cue permutations for lexicographic strategies we show that there is no efficient algorithm that approximates the optimum to within any constant factor unless p np the results have implications for the complexity of learning lexicographic strategies from examples they show that learning them in polynomial time within the model of agnostic probably approximately correct pac learning is impossible unless rp np we further consider greedy approaches for building lexicographic strategies and determine upper and lower bounds for the performance ratio of simple algorithms moreover we present a greedy algorithm that performs provably better than take the best tight bounds on the sample complexity for learning lexicographic strategies are also given in this article

method of noise reduction using correction and scaling vectors with partitioning of the acoustic space in the domain of noisy speech a method and apparatus are provided for reducing noise in a training signal and or test signal the noise reduction technique uses a stereo signal formed of two channel signals each channel containing the same pattern signal one of the channel signals is clean and the other includes additive noise using feature vectors from these channel signals a collection of noise correction and scaling vectors is determined when a feature vector of a noisy pattern signal is later received it is multiplied by the best scaling vector for that feature vector and the best correction vector is added to the product to produce a noise reduced feature vector under one embodiment the best scaling and correction vectors are identified by choosing an optimal mixture component for the noisy feature vector the optimal mixture component being selected based on a distribution of noisy channel feature vectors associated with each mixture component

variable selection using svm based criteria we propose new methods to evaluate variable subset relevance with a view to variable selection relevance criteria are derived from support vector machines and are based on weight vector w 2 or generalization error bounds sensitivity with respect to a variable experiments on linear and non linear toy problems and real world datasets have been carried out to assess the effectiveness of these criteria results show that the criterion based on weight vector derivative achieves good results and performs consistently well over the datasets we used

arbitrarily many solutions for a perturbed p x laplacian equation involving oscillatory terms this paper deals with a perturbed p x laplacian equation involving oscillatory terms div u p x email protected u u p x 2u q x f u email protected u email protected r n u 0 u x 0as x by using variational methods and the non smooth version symmetric criticality principle we establish a the unperturbed problem p 0 has infinitely many distinct solutions b the number of distinct solutions for p e becomes greater and greater whenever e is smaller and smaller our results are a generalization of the case of laplacian from kristaly to the case of p x laplacian

top k feature selection framework using robust 0 1 integer programming feature selection fs which identifies the relevant features in a data set to facilitate subsequent data analysis is a fundamental problem in machine learning and has been widely studied in recent years most fs methods rank the features in order of their scores based on a specific criterion and then select the k top ranked features where k is the number of desired features however these features are usually not the top k features and may present a suboptimal choice to address this issue we propose a novel fs framework in this article to select the exact top k features in the unsupervised semisupervised and supervised scenarios the new framework utilizes the l0 2 norm as the matrix sparsity constraint rather than its relaxations such as the l1 2 norm since the l0 2 norm constrained problem is difficult to solve we transform the discrete l0 2 norm based constraint into an equivalent 0 1 integer constraint and replace the 0 1 integer constraint with two continuous constraints the obtained top k fs framework with two continuous constraints is theoretically equivalent to the l0 2 norm constrained problem and can be optimized by the alternating direction method of multipliers admm unsupervised and semisupervised fs methods are developed based on the proposed framework and extensive experiments on real world data sets are conducted to demonstrate the effectiveness of the proposed fs framework

existence of solutions for a perturbed dirichlet problem without growth conditions we present some results on the existence and multiplicity of solutions for boundary value problems involving equations of the type u f x u g x u where is the laplacian operator is a real parameter and f g r r rn are two caratheodory functions having no growth conditions with respect to the second variable the approach is variational and mainly based on a critical point theorem by b ricceri

local causal and markov blanket induction for causal discovery and feature selection for classification part i algorithms and empirical evaluation we present an algorithmic framework for learning local causal structure around target variables of interest in the form of direct causes effects and markov blankets applicable to very large data sets with relatively small samples the selected feature sets can be used for causal discovery and classification the framework generalized local learning or gll can be instantiated in numerous ways giving rise to both existing state of the art as well as novel algorithms the resulting algorithms are sound under well defined sufficient conditions in a first set of experiments we evaluate several algorithms derived from this framework in terms of predictivity and feature set parsimony and compare to other local causal discovery methods and to state of the art non causal feature selection methods using real data a second set of experimental evaluations compares the algorithms in terms of ability to induce local causal neighborhoods using simulated and resimulated data and examines the relation of predictivity with causal induction performance our experiments demonstrate consistently with causal feature selection theory that local causal feature selection methods under broad assumptions encompassing appropriate family of distributions types of classifiers and loss functions exhibit strong feature set parsimony high predictivity and local causal interpretability although non causal feature selection methods are often used in practice to shed light on causal relationships we find that they cannot be interpreted causally even when they achieve excellent predictivity therefore we conclude that only local causal techniques should be used when insight into causal structure is sought in a companion paper we examine in depth the behavior of gll algorithms provide extensions and show how local techniques can be used for scalable and accurate global causal graph learning

sparse density estimation on the multinomial manifold a new sparse kernel density estimator is introduced based on the minimum integrated square error criterion for the finite mixture model since the constraint on the mixing coefficients of the finite mixture model is on the multinomial manifold we use the well known riemannian trust region rtr algorithm for solving this problem the first and second order riemannian geometry of the multinomial manifold are derived and utilized in the rtr algorithm numerical examples are employed to demonstrate that the proposed approach is effective in constructing sparse kernel density estimators with an accuracy competitive with those of existing kernel density estimators

acoustic category mean value calculating apparatus and adaptation apparatus each time a matching unit 22 finishes a matching at a time i between feature vectors of input speech which have been converted by an analyzer 1 and feature vectors of reference patterns stored in a reference pattern memory 2 and goes to a matching at a next time i 1 a feature vector integrator 24 multiplies the feature vectors of the input speech or the reference patterns by a weight w1 in each of acoustic categories successively integrates products stores integrated values in respective frames of the reference patterns in feature vector accumulating buffers 26 and integrates and stores weights w1 in feature vector weight counters 27 corresponding to the respective acoustic categories when the feature vector integrator 24 finishes integrating and storing processes corresponding to the final matching process a mean value calculator 28 divides the values stored in the final frames in the feature vector accumulating buffers 26 by the values stored in the feature vector weight counters 27 and outputs the quotients as mean values in the respective acoustic categories of the input speech or the reference patterns an adaptation unit 55 adapts one or both of the input speech or the reference patterns using the mean values

feature selection for linear svm with provable guarantees we give two provably accurate feature selection techniques for the linear svm the algorithms run in deterministic and randomized time respectively our algorithms can be used in an unsupervised or supervised setting the supervised approach is based on sampling features from support vectors we prove that the margin in the feature space is preserved to within relative error of the margin in the full feature space in the worst case in the unsupervised setting we also provide worst case guarantees of the radius of the minimum enclosing ball thereby ensuring comparable generalization as in the full feature space and resolving an open problem posed in dasgupta et al 2007 7 we present extensive experiments on real world datasets to support our theory and to demonstrate that our method is competitive and often better than prior state of the art for which there are no known provable guarantees highlightswe give two provably accurate feature selection techniques for the linear svm algorithms can be used in supervised or unsupervised setting we prove margin is preserved to within e relative error in the full feature space in unsupervised case we provide worst case guarantees of margin and radius of minimum enclosing ball extensive experiments demonstrate that our method is competitive and often better than prior art

the existence of multiple equilibrium points in a global attractor for some p laplacian equation abstract in this paper we are mainly concerned with some properties of the global attractor for some p laplacian equation with a lyapunov function in a banach space under some suitable assumptions we prove the existence of multiple equilibrium points in a global attractor for some p laplacian equation

an optimization based method for feature ranking in nonlinear regression problems in this paper we consider the feature ranking problem where given a set of training instances the task is to associate a score with the features in order to assess their relevance feature ranking is a very important tool for decision support systems and may be used as an auxiliary step of feature selection to reduce the high dimensionality of real world data we focus on regression problems by assuming that the process underlying the generated data can be approximated by a continuous function for instance a feedforward neural network we formally state the notion of relevance of a feature by introducing a minimum zero norm inversion problem of a neural network which is a nonsmooth constrained optimization problem we employ a concave approximation of the zero norm function and we define a smooth global optimization problem to be solved in order to assess the relevance of the features we present the new feature ranking method based on the solution of instances of the global optimization problem depending on the available training data computational experiments on both artificial and real data sets are performed and point out that the proposed feature ranking method is a valid alternative to existing methods in terms of effectiveness the obtained results also show that the method is costly in terms of cpu time and this may be a limitation in the solution of large dimensional problems

distributed nonconvex constrained optimization over time varying digraphs this paper considers nonconvex distributed constrained optimization over networks modeled as directed possibly time varying graphs we introduce the first algorithmic framework for the minimization of the sum of a smooth nonconvex nonseparable function the agent s sum utility plus a difference of convex function with nonsmooth convex part this general formulation arises in many applications from statistical machine learning to engineering the proposed distributed method combines successive convex approximation techniques with a judiciously designed perturbed push sum consensus mechanism that aims to track locally the gradient of the smooth part of the sum utility sublinear convergence rate is proved when a fixed step size possibly different among the agents is employed whereas asymptotic convergence to stationary solutions is proved using a diminishing step size numerical results show that our algorithms compare favorably with current schemes on both convex and nonconvex problems

primal explicit max margin feature selection for nonlinear support vector machines embedding feature selection in nonlinear support vector machines svms leads to a challenging non convex minimization problem which can be prone to suboptimal solutions this paper develops an effective algorithm to directly solve the embedded feature selection primal problem we use a trust region method which is better suited for non convex optimization compared to line search methods and guarantees convergence to a minimizer we devise an alternating optimization approach to tackle the problem efficiently breaking it down into a convex subproblem corresponding to standard svm optimization and a non convex subproblem for feature selection importantly we show that a straightforward alternating optimization approach can be susceptible to saddle point solutions we propose a novel technique which shares an explicit margin variable to overcome saddle point convergence and improve solution quality experiment results show our method outperforms the state of the art embedded svm feature selection method as well as other leading filter and wrapper approaches highlightswe solve a primal embedded feature selection problem for nonlinear svms use smooth hinge loss and trust region algorithm for bound constrained minimization propose alternating optimization approach to break problem into two smaller ones propose explicit margin maximization to solve feature selection subproblem show our approach improves state of art and other algorithms on various data

fractional norm regularization learning with very few relevant features learning in the presence of a large number of irrelevant features is an important problem in high dimensional tasks previous studies have shown that l1 norm regularization can be effective in such cases while l2 norm regularization is not furthermore work in compressed sensing suggests that regularization by nonconvex e g fractional semi norms may outperform l1 regularization however for classification it is largely unclear when this may or may not be the case in addition the nonconvex problem is harder to solve than the convex l1 problem in this paper we provide a more in depth analysis to elucidate the potential advantages and pitfalls of nonconvex regularization in the context of logistic regression where the regularization term employs the family of lq semi norms first using results from the phenomenon of concentration of norms and distances in high dimensions we gain intuition about the working of sparse estimation when the dimensionality is very high second using the probably approximately correct pac bayes methodology we give a data dependent bound on the generalization error of lq regularized logistic regression which is applicable to any algorithm that implements this model and may be used to predict its generalization behavior from the training set alone third we demonstrate the usefulness of our approach by experiments and applications where the pac bayes bound is used to guide the choice of semi norm in the regularization term the results support the conclusion that the optimal choice of regularization depends on the relative fraction of relevant versus irrelevant features and a fractional norm with a small exponent is most suitable when the fraction of relevant features is very small

margin maximizing feature elimination methods for linear and nonlinear kernel based discriminant functions feature selection for classification in high dimensional spaces can improve generalization reduce classifier complexity and identify important discriminating feature markers for support vector machine svm classification a widely used technique is recursive feature elimination rfe we demonstrate that rfe is not consistent with margin maximization central to the svm learning approach we thus propose explicit margin based feature elimination mfe for svms and demonstrate both improved margin and improved generalization compared with rfe moreover for the case of a nonlinear kernel we show that rfe assumes that the squared weight vector 2 norm is strictly decreasing as features are eliminated we demonstrate this is not true for the gaussian kernel and consequently rfe may give poor results in this case mfe for nonlinear kernels gives better margin and generalization we also present an extension which achieves further margin gains by optimizing only two degrees of freedom the hyperplane s intercept and its squared 2 norm with the weight vector orientation fixed we finally introduce an extension that allows margin slackness we compare against several alternatives including rfe and a linear programming method that embeds feature selection within the classifier design on high dimensional gene microarray data sets university of california at irvine uci repository data sets and alzheimer s disease brain image data mfe methods give promising results

robust joint graph sparse coding for unsupervised spectral feature selection in this paper we propose a new unsupervised spectral feature selection model by embedding a graph regularizer into the framework of joint sparse regression for preserving the local structures of data to do this we first extract the bases of training data by previous dictionary learning methods and then map original data into the basis space to generate their new representations by proposing a novel joint graph sparse coding jgsc model in jgsc we first formulate its objective function by simultaneously taking subspace learning and joint sparse regression into account then design a new optimization solution to solve the resulting objective function and further prove the convergence of the proposed solution furthermore we extend jgsc to a robust jgsc rjgsc via replacing the least square loss function with a robust loss function for achieving the same goals and also avoiding the impact of outliers finally experimental results on real data sets showed that both jgsc and rjgsc outperformed the state of the art algorithms in terms of k nearest neighbor classification performance

discriminative semi supervised feature selection via manifold regularization feature selection has attracted a huge amount of interest in both research and application communities of data mining we consider the problem of semi supervised feature selection where we are given a small amount of labeled examples and a large amount of unlabeled examples since a small number of labeled samples are usually insufficient for identifying the relevant features the critical problem arising from semi supervised feature selection is how to take advantage of the information underneath the unlabeled data to address this problem we propose a novel discriminative semi supervised feature selection method based on the idea of manifold regularization the proposed approach selects features through maximizing the classification margin between different classes and simultaneously exploiting the geometry of the probability distribution that generates both labeled and unlabeled data in comparison with previous semi supervised feature selection algorithms our proposed semi supervised feature selection method is an embedded feature selection method and is able to find more discriminative features we formulate the proposed feature selection method into a convex concave optimization problem where the saddle point corresponds to the optimal solution to find the optimal solution the level method a fairly recent optimization method is employed we also present a theoretic proof of the convergence rate for the application of the level method to our problem empirical evaluation on several benchmark data sets demonstrates the effectiveness of the proposed semi supervised feature selection method

completeness in approximation classes we introduce a formal framework for studying approximation properties of np optimization npo problems the classes we consider are those appearing in the literature namely the class of approximable problems within a constant e apx the class of problems having a polynomial time approximation scheme pas and the class of problems having a fully polynomial time approximation scheme fpas we define natural approximation preserving reductions and obtain completeness results for these classes a complete problem in a class can not have stronger approximation properties unless p np we also show that the degree structure of npo allows intermediate degrees that is if p np there are problems which are neither complete nor belong to a lower class

estimation of speech spectral parameters in the presence of noise a speech analysis system includes a codebook 230 that stores speech spectral parameters corresponding to a speech spectral hypothesis the speech spectral hypothesis may be combined with a noise spectral hypothesis and the combined hypothesis compared to a sensed signal via a discriminant function 240 the discriminant function may be evaluated using the preconditioned conjugate gradient pcg process

sparse and robust normal and t portfolios by penalized lq likelihood minimization abstract two important problems arising in traditional asset allocation methods are the sensitivity to estimation error of portfolio weights and the high dimensionality of the set of candidate assets in this paper we address both issues by proposing a new criterion for portfolio selection the new criterion is a two stage description of the available information where the q entropy a generalized measure of information is used to code the uncertainty of the data given the parametric model and the uncertainty related to the model choice the information about the model is coded in terms of a prior distribution that promotes asset weights sparsity our approach carries out model selection and estimation in a single step by selecting a few assets and estimating their portfolio weights simultaneously the resulting portfolios are doubly robust in the sense that they can tolerate deviations from both assumed data model and prior distribution for model parameters empirical results on simulated and real world data support the validity of our approach

robust clustering using outlier sparsity regularization notwithstanding the popularity of conventional clustering algorithms such as k means and probabilistic clustering their clustering results are sensitive to the presence of outliers in the data even a few outliers can compromise the ability of these algorithms to identify meaningful hidden structures rendering their outcome unreliable this paper develops robust clustering algorithms that not only aim to cluster the data but also to identify the outliers the novel approaches rely on the infrequent presence of outliers in the data which translates to sparsity in a judiciously chosen domain leveraging sparsity in the outlier domain outlier aware robust k means and probabilistic clustering approaches are proposed their novelty lies on identifying outliers while effecting sparsity in the outlier domain through carefully chosen regularization a block coordinate descent approach is developed to obtain iterative algorithms with convergence guarantees and small excess computational complexity with respect to their non robust counterparts kernelized versions of the robust clustering algorithms are also developed to efficiently handle high dimensional data identify nonlinearly separable clusters or even cluster objects that are not represented by vectors numerical tests on both synthetic and real datasets validate the performance and applicability of the novel algorithms

adaptive unsupervised feature selection with structure regularization feature selection is one of the most important dimension reduction techniques for its efficiency and interpretation since practical data in large scale are usually collected without labels and labeling these data are dramatically expensive and time consuming unsupervised feature selection has become a ubiquitous and challenging problem without label information the fundamental problem of unsupervised feature selection lies in how to characterize the geometry structure of original feature space and produce a faithful feature subset which preserves the intrinsic structure accurately in this paper we characterize the intrinsic local structure by an adaptive reconstruction graph and simultaneously consider its multiconnected components multicluster structure by imposing a rank constraint on the corresponding laplacian matrix to achieve a desirable feature subset we learn the optimal reconstruction graph and selective matrix simultaneously instead of using a predetermined graph we exploit an efficient alternative optimization algorithm to solve the proposed challenging problem together with the theoretical analyses on its convergence and computational complexity finally extensive experiments on clustering task are conducted over several benchmark data sets to verify the effectiveness and superiority of the proposed unsupervised feature selection algorithm

incremental p margin algorithm for classification with arbitrary norm this paper presents a new algorithm to approximate large margin solutions in binary classification problems with arbitrary q norm or p margin where p and q are holder conjugates we begin by presenting the online fixed p margin perceptron algorithm fmpp that solves linearly separable classification problems in primal variables and consists of a generalization of the fixed margin perceptron algorithm fmp this algorithm is combined with an incremental margin strategy called imap which computes an approximation of the maximal p margin to achieve this goal imap executes fmpp several times with increasing p margin values one of the main advantages of this approach is its flexibility which allows the use of different p norms in the same primal formulation for non linearly separable problems fmpp can be used with a soft margin in primal variables the incremental learning strategy always guarantees a good approximation of the optimal p margin and avoids the use of linear or higher order programming methods imap was tested in different datasets obtaining similar results when compared to classical l1 and l linear programming formulations also the algorithm was compared to almap and presents superior results highlightswe propose a novel algorithm for large p margin classification problems for 1 p the approach is based on an unified perceptron based formulation soft margin in primal variables is introduced for non linearly separable problems an efficient incremental strategy is used to construct the large p margin solution

multiple solutions for two nonlinear problems involving the p laplacian abstract using variational arguments we prove the existence of at least two solutions for two elliptic problems involving the p laplacian dirichlet and neumann boundary conditions will be considered in the two problems where a positive parameter appears

low bit rate speech coder using lpc data reduction processing a speech coder employs vector quantization of lpc parameters interpolation and trellis coding for improved speech coding at low bit rates 400 bps the speech coder has an lpc analysis module for converting input speech to lpc parameters an lsp conversion module for converting lpc parameters into line spectrum frequencies lsp data and a vector quantization and interpolation vq i module for encoding the lsp data into vector indexes for transmission by applying lpc spectral amplitude as weighting coefficients to the lsp data the vq i module outputs one vector index for every two lpc frames in order to reduce the transmission bit rate and the omitted frames are interpolated on the receiving end a decoder correspondingly decodes incoming indexes to lpc parameters and synthesizes them into output speech trellis coders with an adaptive tracking function encode the pitch and gain parameters of the lpc frames a universal codebook stores codewords according to a plurality of accents the speech coder automatically identifies a speaker s accent and selects the corresponding vocabulary of codewords in order to more intelligibly encode and decode the speaker s speech

perturbation from dirichlet problem involving oscillating nonlinearities in this paper we prove that if the potential f x t 0tf x s ds has a suitable oscillating behavior in any neighborhood of the origin respectively then under very mild conditions on the perturbation term g for every k n there exists bk 0 such that u f x u g x u in u 0on has at least k distinct weak solutions in w01 2 for every r with bk moreover information about the location of such solutions is also given in fact there exists a positive real number 0 which does not depend on such that the w01 2 norm of each of those k solutions is not greater than

multiclass feature selection with kernel gram matrix based criteria feature selection has been an important issue in recent decades to determine the most relevant features according to a given classification problem numerous methods have emerged that take into account support vector machines svms in the selection process such approaches are powerful but often complex and costly in this paper we propose new feature selection methods based on two criteria designed for the optimization of svm kernel target alignment and kernel class separability we demonstrate how these two measures when fully expressed can build efficient and simple methods easily applicable to multiclass problems and iteratively computable with minimal memory requirements an extensive experimental study is conducted both on artificial and real world datasets to compare the proposed methods to state of the art feature selection algorithms the results demonstrate the relevance of the proposed methods both in terms of performance and computational cost

parallel and distributed methods for constrained nonconvex optimization part ii applications in communications and machine learning in part i of this paper we proposed and analyzed a novel algorithmic framework for the minimization of a nonconvex objective function subject to nonconvex constraints based on inner convex approximations this part ii is devoted to the nontrivial application of the framework to the following relevant large scale problems ranging from communications to machine learning 1 generalizations of the rate profile maximization in mimo interference broadcast networks 2 the max min fair multicast multigroup beamforming problem in a multicell environment and 3 a general nonconvex constrained bi criteria formulation for k sparse variable selection in statistical learning the two criteria are a nonconvex loss objective function measuring the fitness of the model to data and the latter is a nonconvex sparsity inducing constraint in the general form of difference of convex dc functions which allows to accomodate in a unified fashion convex and nonconvex surrogates of the ell 0 function the proposed algorithms outperform current state of the art schemes for 1 3 both theoretically and numerically for instance they are the first distributed schemes for the class of problems 1 and 2 and they also lead to subproblems enjoying closed form solutions

three solutions for a perturbed dirichlet problem abstract in this paper we prove the existence of at least three distinct solutions to the following perturbed dirichlet problem u f x u g x u in u 0 on where r n is an open bounded set with smooth boundary and r under very mild conditions on g and some assumptions on the behaviour of the potential of f at 0 and our result assures the existence of at least three distinct solutions to the above problem for small enough moreover such solutions belong to a ball of the space w 0 1 2 centered in the origin and with radius not dependent on

the support feature machine classification with the least number of features and application to neuroimaging data by minimizing the zero norm of the separating hyperplane the support feature machine sfm finds the smallest subspace the least number of features of a data set such that within this subspace two classes are linearly separable without error this way the dimensionality of the data is more efficiently reduced than with support vector based feature selection which can be shown both theoretically and empirically in this letter we first provide a new formulation of the previously introduced concept of the sfm with this new formulation classification of unbalanced and nonseparable data is straightforward which allows using the sfm for feature selection and classification in a large variety of different scenarios to illustrate how the sfm can be used to identify both the smallest subset of discriminative features and the total number of informative features in biological data sets we apply repetitive feature selection based on the sfm to a functional magnetic resonance imaging data set we suggest that these capabilities qualify the sfm as a universal method for feature selection especially for high dimensional small sample size data sets that often occur in biological and medical applications

reference speech pattern generating method a time series of spectral parameters is extracted from a learning speech the spectral parameters are divided into a plurality of segments for each voice interval and the segments are clustered into a plurality of clusters for each cluster an initial reference pattern representing the cluster is computed the segment boundaries are corrected using the computed reference patterns a correcting step the segments of the corrected spectral parameter time series are clustered a clustering step and for each cluster a reference pattern representing the cluster is computed a reference pattern computing step the correcting step the clustering step and the reference pattern computing step are performed at least once and the reference patterns obtained by the last reference pattern computing step are regarded as reference patterns desired to be obtained

nonconvex regularizations for feature selection in ranking with sparse svm feature selection in learning to rank has recently emerged as a crucial issue whereas several preprocessing approaches have been proposed only a few have focused on integrating feature selection into the learning process in this paper we propose a general framework for feature selection in learning to rank using support vector machines with a sparse regularization term we investigate both classical convex regularizations such as ell 1 or weighted ell 1 and nonconvex regularization terms such as log penalty minimax concave penalty or ell p pseudo norm with p two algorithms are proposed the first an accelerated proximal approach for solving the convex problems and the second a reweighted ell 1 scheme to address nonconvex regularizations we conduct intensive experiments on nine datasets from letor 3 0 and letor 4 0 corpora numerical results show that the use of nonconvex regularizations we propose leads to more sparsity in the resulting models while preserving the prediction performance the number of features is decreased by up to a factor of 6 compared to the ell 1 regularization in addition the software is publicly available on the web 1 available at http remi flamary com soft soft ranksvm nc html

minimal kernel classifiers a finite concave minimization algorithm is proposed for constructing kernel classifiers that use a minimal number of data points both in generating and characterizing a classifier the algorithm is theoretically justified on the basis of linear programming perturbation theory and a leave one out error bound as well as effective computational results on seven real world datasets a nonlinear rectangular kernel is generated by systematically utilizing as few of the data as possible both in training and in characterizing a nonlinear separating surface this can result in substantial reduction in kernel data dependence over 94 in six of the seven public datasets tested on and with test set correctness equal to that obtained by using a conventional support vector machine classifier that depends on many more data points this reduction in data dependence results in a much faster classifier that requires less storage to eliminate data points the proposed approach makes use of a novel loss function the pound function which is a linear combination of the 1 norm and the step function that measures both the magnitude and the presence of any error

enhancement of speech coding in background noise for low rate speech coder a speech coding system employs measurements of robust features of speech frames whose distribution are not strongly affected by noise levels to make voicing decisions for input speech occurring in a noisy environment linear programing analysis of the robust features and respective weights are used to determine an optimum linear combination of these features the input speech vectors are matched to a vocabulary of codewords in order to select the corresponding optimally matching codeword adaptive vector quantization is used in which a vocabulary of words obtained in a quiet environment is updated based upon a noise estimate of a noisy environment in which the input speech occurs and the noisy vocabulary is then searched for the best match with an input speech vector the corresponding clean codeword index is then selected for transmission and for synthesis at the receiver end the results are better spectral reproduction and significant intelligibility enhancement over prior coding approaches robust features found to allow robust voicing decisions include low band energy zero crossing counts adapted for noise level amdf ratio speech periodicity measure low pass filtered backward correlation low pass filtered forward correlation inverse filtered backward correlation and inverse filtered pitch prediction gain measure

noise compensation arrangement adaptive speech recognition is improved by extracting an estimate of the background noise during unknown speech input and using the noise estimate to modify the noiseless prestored reference speech signals for comparison with the input averaged values of autocorrelation coefficient and toeplitz matrix elements representative of noise frames and prestored reference frames are used to calculate the lpc predictor signals

detection of arbitrarily many solutions for perturbed elliptic problems involving oscillatory terms we propose a direct approach for detecting arbitrarily many solutions for perturbed elliptic problems involving oscillatory terms although the method works in various frameworks we illustrate it on the problem

vector adaptive vector quantization with application to speech coding a vector adaptive vector quantization vavq scheme which may be viewed as a generalization of gain adaptive vector quantization is described the proposed scheme adjusts each component of the encoding signal vector according to a statistical estimate of the signal characteristics the vavq scheme can cope with large input dynamic ranges it can be used in either the time domain or the transform domain and exhibits approximately 4 db improvement in segmental signal to noise ratio snr over the fixed vq

structure preserving stability and accuracy properties of the energy conserving sampling and weighting method for the hyper reduction of nonlinear finite element dynamic models summary the computational efficiency of a typical projection based nonlinear model reduction method hinges on the efficient approximation for explicit computations of the scalar projections onto a subspace of a residual vector for implicit computations it also hinges on the additional efficient approximation of similar projections of the jacobian of this residual with respect to the solution the computation of both approximations is often referred to in the literature as hyper reduction to this effect this paper focuses on the analysis and comparative performance study for nonlinear finite element reduced order models of solids and structures of the recently developed energy conserving mesh sampling and weighting ecsw hyper reduction method unlike most alternative approaches this method approximates the scalar projections of residuals and or jacobians directly instead of approximating first these vectors and matrices then projecting the resulting approximations onto the subspaces of interest in this paper it is shown that ecsw distinguishes itself furthermore from other hyper reduction methods through its preservation of the lagrangian structure associated with hamilton s principle for second order dynamical systems this enables it to also preserve the numerical stability properties of the discrete system to which it is applied it is also shown that for a fixed set of parameter values the approximation error committed online by ecsw is bounded by its counterpart error committed off line during the training of this method therefore this error can be estimated in this case a priori and controlled the performance of ecsw is demonstrated first for two academic but nevertheless interesting nonlinear dynamic response problems for both of them ecsw is shown to preserve numerical stability and deliver the desired level of accuracy whereas the discrete empirical interpolation method and its recently introduced unassembled variant are shown to be susceptible to failure because of numerical instability the potential of ecsw for enabling the effective reduction of nonlinear finite element dynamic models of solids and structures is also highlighted with the realistic simulation of the nonlinear transient dynamic response of a complete car engine to thermal and combustion pressure loads using an implicit scheme for this simulation ecsw is reported to enable the reduction of the cpu time required by the high dimensional nonlinear finite element dynamic analysis by more than four orders of magnitude while achieving a very good level of accuracy copyright 2015 john wiley sons ltd

use of the zero norm with linear models and kernel methods we explore the use of the so called zero norm of the parameters of linear models in learning minimization of such a quantity has many uses in a machine learning context for variable or feature selection minimizing training error and ensuring sparsity in solutions we derive a simple but practical method for achieving these goals and discuss its relationship to existing techniques of minimizing the zero norm the method boils down to implementing a simple modification of vanilla svm namely via an iterative multiplicative rescaling of the training data applications we investigate which aid our discussion include variable and feature selection on biological microarray data and multicategory classification

a speaker independent label coding apparatus the present invention is related to speech recognition and particularly to a new type of vector quantizer and a new vector quantization technique in which the error rate of associating a sound with an incoming speech signal is drastically reduced to achieve this end the present invention technique groups the feature vectors in a space into different prototypes at least two of which represent a class of sound each of the prototypes may in turn have a number of subclasses or partitions each of the prototypes and their subclasses may be assigned respective identifying values to identify an incoming speech feature vector at least one of the feature values of the incoming feature vector is compared with the different values of the respective prototypes or the subclasses of the prototypes the class of sound whose group of prototypes or at least one of the prototypes whose combined value most closely matches the value of the feature value of the feature vector is deemed to be the class corresponding to the feature vector the feature vector is then labeled with the identifier associated with that class

on the approximability of minimizing nonzero variables or unsatisfied relations in linear systems we investigate the computational complexity of two closely related classes of combinatorial optimization problems for linear systems which arise in various fields such as machine learning operations research and pattern recognition in the first class min ulr one wishes given a possibly infeasible system of linear relations to find a solution that violates as few relations as possible while satisfying all the others in the second class min rvls the linear system is supposed to be feasible and one looks for a solution with as few nonzero variables as possible for both min ulr and min rvls the four basic types of relational operators and are considered while min rvls with equations was mentioned to be np hard in garey and johnson 1979 we established in amaldi 1992 amaldi and kann 1995 that min ulr with equalities and inequalities are np hard even when restricted to homogeneous systems with bipolar coefficients the latter problems have been shown hard to approximate in arora et al 1993 in this paper we determine strong bounds on the approximability of various variants of min rvls and min ulr including constrained ones where the variables are restricted to take binary values or where some relations are mandatory while others are optional the various np hard versions turn out to have different approximability properties depending on the type of relations and the additional constraints but none of them can be approximated within any constant factor unless p np particular attention is devoted to two interesting special cases that occur in discriminant analysis and machine learning in particular we disprove a conjecture of van horn and martinez 1992 regarding the existence of a polynomial time algorithm to design linear classifiers or perceptrons that involve a close to minimum number of features

very sparse lssvm reductions for large scale data least squares support vector machines lssvms have been widely applied for classification and regression with comparable performance with svms the lssvm model lacks sparsity and is unable to handle large scale data due to computational and memory constraints a primal fixed size lssvm pfs lssvm introduce sparsity using nystrom approximation with a set of prototype vectors pvs the pfs lssvm model solves an overdetermined system of linear equations in the primal however this solution is not the sparsest we investigate the sparsity error tradeoff by introducing a second level of sparsity this is done by means of l 0 norm based reductions by iteratively sparsifying lssvm and pfs lssvm models the exact choice of the cardinality for the initial pv set is not important then as the final model is highly sparse the proposed method overcomes the problem of memory constraints and high computational costs resulting in highly sparse reductions to lssvm models the approximations of the two models allow to scale the models to large scale datasets experiments on real world classification and regression data sets from the uci repository illustrate that these approaches achieve sparse models without a significant tradeoff in errors

a sparsity driven kernel machine based on minimizing a generalization error bound a new sparsity driven kernel classifier is presented based on the minimization of a recently derived data dependent generalization error bound the objective function consists of the usual hinge loss function penalizing training errors and a concave penalty function of the expansion coefficients the problem of minimizing the non convex bound is addressed by a successive linearization approach whereby the problem is transformed into a sequence of linear programs the algorithm produced comparable error rates to the standard support vector machine but significantly reduced the number of support vectors and the concomitant classification time

feature elimination in kernel machines in moderately high dimensions we develop an approach for feature elimination in statistical learning with kernel machines based on recursive elimination of features we present theoretical properties of this method and show that it is uniformly consistent in finding the correct feature space under certain generalized assumptions we present a few case studies to show that the assumptions are met in most practical situations and present simulation results to demonstrate performance of the proposed approach

effective discriminative feature selection with nontrivial solution feature selection and feature transformation the two main ways to reduce dimensionality are often presented separately in this paper a feature selection method is proposed by combining the popular transformation based dimensionality reduction method linear discriminant analysis lda and sparsity regularization we impose row sparsity on the transformation matrix of lda through ell 2 1 norm regularization to achieve feature selection and the resultant formulation optimizes for selecting the most discriminative features and removing the redundant ones simultaneously the formulation is extended to the ell 2 p norm regularized case which is more likely to offer better sparsity when 0 thus the formulation is a better approximation to the feature selection problem an efficient algorithm is developed to solve the ell 2 p norm based optimization problem and it is proved that the algorithm converges when 0 systematical experiments are conducted to understand the work of the proposed method promising experimental results on various types of real world data sets demonstrate the effectiveness of our algorithm

on the maximum feasible subsystem problem iiss and iis hypergraphs we consider the max fs problem for a given infeasible linear system ax b determine a feasible subsystem containing as many inequalities as possible this problem which is np hard and also difficult to approximate has a number of interesting applications in a wide range of fields in this paper we examine structural and algorithmic properties of max fs and of irreducible infeasible subsystems iiss which are intrinsically related since one must delete at least one constraint from each iis to attain feasibility first we provide a new simplex decomposition characterization of iiss and prove that finding a smallest cardinality iis is very difficult to approximate then we discuss structural properties of iis hypergraphs i e hypergraphs in which each edge corresponds to an iis and show that recognizing iis hypergraphs subsumes the steinitz problem for polytopes and hence is np hard finally we investigate rank facets of the feasible subsystem polytope whose vertices are incidence vectors of feasible subsystems of a given infeasible system in particular using the iis hypergraph structural result we show that only two very specific types of rank inequalities induced by generalized antiwebs which generalize cliques odd holes and antiholes to general independence systems can arise as facets

second order cone programming formulations for feature selection this paper addresses the issue of feature selection for linear classifiers given the moments of the class conditional densities the problem is posed as finding a minimal set of features such that the resulting classifier has a low misclassification error using a bound on the misclassification error involving the mean and covariance of class conditional densities and minimizing an l1 norm as an approximate criterion for feature selection a second order programming formulation is derived to handle errors in estimation of mean and covariances a tractable robust formulation is also discussed in a slightly different setting the fisher discriminant is derived feature selection for fisher discriminant is also discussed experimental results on synthetic data sets and on real life microarray data show that the proposed formulations are competitive with the state of the art linear programming formulation

three solutions for a perturbed dirichlet boundary value problem involving the p laplacian abstract in this paper we study the perturbed dirichlet problem p u a x u p 2 u f x u g x u in u 0 on some existence results of three solutions are obtained by using a three critical points theorem proved by g bonanno 2003 g bonanno some remarks on a three critical points theorem nonlinear anal 54 2003 651 665

model selection beyond the bayesian frequentist divide the principle of parsimony also known as ockham s razor has inspired many theories of model selection yet such theories all making arguments in favor of parsimony are based on very different premises and have developed distinct methodologies to derive algorithms we have organized challenges and edited a special issue of jmlr and several conference proceedings around the theme of model selection in this editorial we revisit the problem of avoiding overfitting in light of the latest results we note the remarkable convergence of theories as different as bayesian theory minimum description length bias variance tradeoff structural risk minimization and regularization in some approaches we also present new and interesting examples of the complementarity of theories leading to hybrid algorithms neither frequentist nor bayesian or perhaps both frequentist and bayesian

the complexity and approximability of finding maximum feasible subsystems of linear relations we study the combinatorial problem which consists given a system of linear relations of finding a maximum feasible subsystem that is a solution satisfying as many relations as possible the computational complexity of this general problem named max fls is investigated for the four types of relations and various constrained versions of max fls where a subset of relations must be satisfied or where the variables take bounded discrete values are also considered we establish the complexity of solving these problems optimally and whenever they are intractable we determine their degree of approximability max fls with or relations is np hard even when restricted to homogeneous systems with bipolar coefficients whereas it can be solved in polynomial time for relations with real coefficients the various np hard versions of max fls belong to different approximability classes depending on the type of relations and the additional constraints we show that the range of approximability stretches from apx complete problems which can be approximated within a constant but not within every constant unless p np to npo pb complete ones that are as hard to approximate as all np optimization problems with polynomially bounded objective functions while max fls with equations and integer coefficients cannot be approximated within pe for some e 0 where p is the number of relations the same problem over gf q for a prime q can be approximated within q but not within qe for some e 0 max fls with strict or nonstrict inequalities can be approximated within 2 but not within every constant factor our results also provide strong bounds on the approximability of two variants of max fls with and relations that arise when training perceptrons which are the building blocks of artificial neural networks and when designing linear classifiers

feature selection for support vector machines via mixed integer linear programming abstract the performance of classification methods such as support vector machines depends heavily on the proper choice of the feature set used to construct the classifier feature selection is an np hard problem that has been studied extensively in the literature most strategies propose the elimination of features independently of classifier construction by exploiting statistical properties of each of the variables or via greedy search all such strategies are heuristic by nature in this work we propose two different mixed integer linear programming formulations based on extensions of support vector machines to overcome these shortcomings the proposed approaches perform variable selection simultaneously with classifier construction using optimization models we ran experiments on real world benchmark datasets comparing our approaches with well known feature selection techniques and obtained better predictions with consistently fewer relevant features

three solutions to a perturbed nonlinear discrete dirichlet problem using critical point theory we study the existence of at least three solutions for a perturbed nonlinear dirichlet boundary value problem for difference equations depending on two positive parameters