incremental support vector learning for ordinal regression support vector ordinal regression svor is a popular method to tackle ordinal regression problems however until now there were no effective algorithms proposed to address incremental svor learning due to the complicated formulations of svor recently an interesting accurate on line algorithm was proposed for training nu support vector classification nu svc which can handle a quadratic formulation with a pair of equality constraints in this paper we first present a modified svor formulation based on a sum of margins strategy the formulation has multiple constraints and each constraint includes a mixture of an equality and an inequality then we extend the accurate on line nu svc algorithm to the modified formulation and propose an effective incremental svor algorithm the algorithm can handle a quadratic formulation with multiple constraints where each constraint is constituted of an equality and an inequality more importantly it tackles the conflicts between the equality and inequality constraints we also provide the finite convergence analysis for the algorithm numerical experiments on the several benchmark and real world data sets show that the incremental algorithm can converge to the optimal solution in a finite number of steps and is faster than the existing batch and incremental svor algorithms meanwhile the modified formulation has better accuracy than the existing incremental svor algorithm and is as accurate as the sum of margins based formulation of shashua and levin

wait and judge scenario optimization we consider convex optimization problems with uncertain probabilistically described constraints in this context scenario optimization is a well recognized methodology where a sample of the constraints is used to describe uncertainty one says that the scenario solution generalizes well or has a high robustness level if it satisfies most of the other constraints besides those in the sample over the past 10 years the main theoretical investigations on the scenario approach have related the robustness level of the scenario solution to the number of optimization variables this paper breaks into the new paradigm that the robustness level is a posteriori evaluated after the solution is computed and the actual number of the so called support constraints is assessed wait and judge a new theory is presented which shows that a posteriori observing k support constraints in dimension d k allows one to draw conclusions close to those obtainable when the problem is from the outset in dimension k this new theory provides evaluations of the robustness that largely outperform those carried out based on the number of optimization variables

a second order cone programming approach for semi supervised learning semi supervised learning ssl involves the training of a decision rule from both labeled and unlabeled data in this paper we propose a novel ssl algorithm based on the multiple clusters per class assumption the proposed algorithm consists of two stages in the first stage we aim to capture the local cluster structure of the training data by using the k nearest neighbor knn algorithm to split the data into a number of disjoint subsets in the second stage a maximal margin classifier based on the second order cone programming socp is introduced to learn an inductive decision function from the obtained subsets globally for linear classification problems once the knn algorithm has been performed the proposed algorithm trains a classifier using only the first and second order moments of the subsets without considering individual data points since the number of subsets is usually much smaller than the number of training points the proposed algorithm is efficient for handling big data sets with a large amount of unlabeled data despite its simplicity the classification performance of the proposed algorithm is guaranteed by the maximal margin classifier we demonstrate the efficiency and effectiveness of the proposed algorithm on both synthetic and real world data sets

adjustable robust solutions of uncertain linear programs we consider linear programs with uncertain parameters lying in some prescribed uncertainty set where part of the variables must be determined before the realization of the uncertain parameters non adjustable variables while the other part are variables that can be chosen after the realization adjustable variables we extend the robust optimization methodology 1 3 6 9 13 14 to this situation by introducing the adjustable robust counterpart arc associated with an lp of the above structure often the arc is significantly less conservative than the usual robust counterpart rc however in most cases the arc is computationally intractable np hard this difficulty is addressed by restricting the adjustable variables to be affine functions of the uncertain data the ensuing affinely adjustable robust counterpart aarc problem is then shown to be in certain important cases equivalent to a tractable optimization problem typically an lp or a semidefinite problem and in other cases having a tight approximation which is tractable the aarc approach is illustrated by applying it to a multi stage inventory management problem

distributionally robust optimization and its tractable approximations in this paper we focus on a linear optimization problem with uncertainties having expectations in the objective and in the set of constraints we present a modular framework to obtain an approximate solution to the problem that is distributionally robust and more flexible than the standard technique of using linear rules our framework begins by first affinely extending the set of primitive uncertainties to generate new linear decision rules of larger dimensions and is therefore more flexible next we develop new piecewise linear decision rules that allow a more flexible reformulation of the original problem the reformulated problem will generally contain terms with expectations on the positive parts of the recourse variables finally we convert the uncertain linear program into a deterministic convex program by constructing distributionally robust bounds on these expectations these bounds are constructed by first using different pieces of information on the distribution of the underlying uncertainties to develop separate bounds and next integrating them into a combined bound that is better than each of the individual bounds

relaxations and approximations of chance constraints under finite distributions optimization problems with constraints involving stochastic parameters that are required to be satisfied with a prespecified probability threshold arise in numerous applications such chance constrained optimization problems involve the dual challenges of stochasticity and nonconvexity in the setting of a finite distribution of the stochastic parameters an optimization problem with linear chance constraints can be formulated as a mixed integer linear program milp the natural milp formulation has a weak relaxation bound and is quite difficult to solve in this paper we review some recent results on improving the relaxation bounds and constructing approximate solutions for milp formulations of chance constraints we also discuss a recently introduced bicriteria approximation algorithm for covering type chance constrained problems this algorithm uses a relaxation to construct a solution whose constraint violation risk level may be larger than the pre specified threshold but is within a constant factor of it and whose objective value is also within a constant factor of the true optimal value finally we present some new results that improve on the bicriteria approximation factors in the finite scenario setting and shed light on the effect of strong relaxations on the approximation ratios

ridge regression biased estimation for nonorthogonal problems in multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory if not incorrect if the prediction vectors are not orthogonal proposed is an estimation procedure based on adding small positive quantities to the diagonal of x x introduced is the ridge trace a method for showing in two dimensions the effects of nonorthogonality it is then shown how to augment x x to obtain biased estimates with smaller mean square error

robust solutions of linear programming problems contaminated with uncertain data optimal solutions of linear programming problems may become severely infeasible if the nominal data is slightly perturbed we demonstrate this phenomenon by studying 90 lps from the well known netlib collection we then apply the robust optimization methodology ben tal and nemirovski 1 3 el ghaoui et al 5 6 to produce robust solutions of the above lps which are in a sense immuned against uncertainty surprisingly for the netlib problems these robust solutions nearly lose nothing in optimality

distributionally robust optimization under moment uncertainty with application to data driven problems stochastic programming can effectively describe many decision making problems in uncertain environments unfortunately such programs are often computationally demanding to solve in addition their solution can be misleading when there is ambiguity in the choice of a distribution for the random parameters in this paper we propose a model that describes uncertainty in both the distribution form discrete gaussian exponential etc and moments mean and covariance matrix we demonstrate that for a wide range of cost functions the associated distributionally robust or min max stochastic program can be solved efficiently furthermore by deriving a new confidence region for the mean and the covariance matrix of a random vector we provide probabilistic arguments for using our model in problems that rely heavily on historical data these arguments are confirmed in a practical example of portfolio selection where our framework leads to better performing policies on the true distribution underlying the daily returns of financial assets

support vector interval regression networks for interval regression analysis in this paper the support vector interval regression networks svirns are proposed for the interval regression analysis the svirns consist of two radial basis function networks one network identifies the upper side of data interval and the other network identifies the lower side of data intervals because the support vector regression svr approach is equivalent to solving a linear constrained quadratic programming problem the number of hidden nodes and the initial values of adjustable parameters can be easily obtained since the selection of a parameter e in the svr approach may seriously affect the modeling performance a two step approach is proposed to properly select the e value after the svr approach with the selected e an initial structure of svirns can be obtained besides outliers will not significantly affect the upper and lower bound interval obtained through the proposed two step approach consequently a traditional back propagation bp learning algorithm can be used to adjust the initial structure networks of svirns under training data sets without or with outliers due to the better initial structure of svirns are obtained by the svr approach the convergence rate of svirns is faster than the conventional networks with bp learning algorithms or with robust bp learning algorithms for interval regression analysis four examples are provided to show the validity and applicability of the proposed svirns

second order conic approximation for disassembly line design with joint probabilistic constraints a problem of profit oriented disassembly line design and balancing with possible partial disassembly and presence of hazardous parts is studied the objective is to design a production line providing a maximal revenue with balanced workload task times are assumed to be random variables with known normal probability distributions the cycle time constraints are to be jointly satisfied with at least a predetermined probability level an and or graph is used to model the precedence relationships among tasks several lower and upper bounding schemes are developed using second order cone programming and convex piecewise linear approximation to show the relevance and applicability of the proposed approach a set of instances from the literature are solved to optimality

robust novelty detection via worst case cvar minimization novelty detection models aim to find the minimum volume set covering a given probability mass this paper proposes a robust single class support vector machine ssvm for novelty detection which is mainly based on the worst case conditional value at risk minimization by assuming that every input is subject to an uncertainty with a specified symmetric support this robust formulation results in a maximization term that is similar to the regularization term in the classical ssvm when the uncertainty set is ell 1 norm ell infty norm or box its training can be reformulated to a linear program while the uncertainty set is ell 2 norm or ellipsoidal its training is a tractable second order cone program the proposed method has a nice consistent statistical property as the training size goes to infinity the estimated normal region converges to the true provided that the magnitude of the uncertainty set decreases in a systematic way the experimental results on three data sets clearly demonstrate its superiority over three benchmark models

robust risk management estimating the probabilities by which different events might occur is usually a delicate task subject to many sources of inaccuracies moreover these probabilities can change over time leading to a very difficult evaluation of the risk induced by any particular decision given a set of probability measures and a set of nominal risk measures we define in this paper the concept of robust risk measure as the worst possible of our risks when each of our probability measures is likely to occur we study how some properties of this new object can be related with those of our nominal risk measures such as convexity or coherence we introduce a robust version of the conditional value at risk cvar and of entropy based risk measures we show how to compute and optimize the robust cvar using convex duality methods and illustrate its behavior using data from the new york stock exchange and from the nasdaq between 2005 and 2010

support vector fuzzy regression machines support vector machine svm has been very successful in pattern recognition and function estimation problems in this paper we introduce the use of svm for multivariate fuzzy linear and nonlinear regression models using the basic idea underlying svm for multivariate fuzzy regressions gives computational efficiency of getting solutions

the price of robustness a robust approach to solving linear optimization problems with uncertain data was proposed in the early 1970s and has recently been extensively studied and extended under this approach we are willing to accept a suboptimal solution for the nominal values of the data in order to ensure that the solution remains feasible and near optimal when the data changes a concern with such an approach is that it might be too conservative in this paper we propose an approach that attempts to make this trade off more attractive that is we investigate ways to decrease what we call the price of robustness in particular we flexibly adjust the level of conservatism of the robust solutions in terms of probabilistic bounds of constraint violations an attractive aspect of our method is that the new robust formulation is also a linear optimization problem thus we naturally extend our methods to discrete optimization problems in a tractable way we report numerical results for a portfolio optimization problem a knapsack problem and a problem from the net lib library

ambiguous joint chance constraints under mean and dispersion information we study joint chance constraints where the distribution of the uncertain parameters is only known to belong to an ambiguity set characterized by the mean and support of the uncertainties and by an upper bound on their dispersion this setting gives rise to pessimistic optimistic ambiguous chance constraints which require the corresponding classical chance constraints to be satisfied for every for at least one distribution in the ambiguity set we demonstrate that the pessimistic joint chance constraints are conic representable if i the constraint coefficients of the decisions are deterministic ii the support set of the uncertain parameters is a cone and iii the dispersion function is of first order that is it is positively homogeneous we also show that pessimistic joint chance constrained programs become intractable as soon as any of the conditions i ii or iii is relaxed in the mildest possible way we further prove that the optimistic joint chance constraints are conic representab

mixing patterns in networks we study assortative mixing in networks the tendency for vertices in networks to be connected to other vertices that are like or unlike them in some way we consider mixing according to discrete characteristics such as language or race in social networks and scalar characteristics such as age as a special example of the latter we consider mixing according to vertex degree i e according to the number of connections vertices have to other vertices do gregarious people tend to associate with other gregarious people we propose a number of measures of assortative mixing appropriate to the various mixing types and apply them to a variety of real world networks showing that assortative mixing is a pervasive phenomenon found in many networks we also propose several models of assortatively mixed networks both analytic ones based on generating function methods and numerical ones based on monte carlo graph generation techniques we use these models to probe the properties of networks as their level of assortativity is varied in the particular case of mixing by degree we find strong variation with assortativity in the connectivity of the network and in the resilience of the network to the removal of vertices

optimization of convex risk functions we consider optimization problems involving convex risk functions by employing techniques of convex analysis and optimization theory in vector spaces of measurable functions we develop new representation theorems for risk models and optimality and duality theory for problems with convex risk functions

clifford support vector machines for classification regression and recurrence this paper introduces the clifford support vector machines csvm as a generalization of the real and complex valued support vector machines using the clifford geometric algebra in this framework we handle the design of kernels involving the clifford or geometric product in this approach one redefines the optimization variables as multivectors this allows us to have a multivector as output therefore we can represent multiple classes according to the dimension of the geometric algebra in which we work we show that one can apply csvm for classification and regression and also to build a recurrent csvm the csvm is an attractive approach for the multiple input multiple output processing of high dimensional geometric entities we carried out comparisons between csvm and the current approaches to solve multiclass classification and regression we also study the performance of the recurrent csvm with experiments involving time series the authors believe that this paper can be of great use for researchers and practitioners interested in multiclass hypercomplex computing particularly for applications in complex and quaternion signal and image processing satellite control neurocomputation pattern recognition computer vision augmented virtual reality robotics and humanoids

uncertain convex programs randomized solutions and confidence levels many engineering problems can be cast as optimization problems subject to convex constraints that are parameterized by an uncertainty or instance parameter two main approaches are generally available to tackle constrained optimization problems in presence of uncertainty robust optimization and chance constrained optimization robust optimization is a deterministic paradigm where one seeks a solution which simultaneously satisfies all possible constraint instances in chance constrained optimization a probability distribution is instead assumed on the uncertain parameters and the constraints are enforced up to a pre specified level of probability unfortunately however both approaches lead to computationally intractable problem formulations

power law distributions in empirical data power law distributions occur in many situations of scientific interest and have significant consequences for our understanding of natural and man made phenomena unfortunately the detection and characterization of power laws is complicated by the large fluctuations that occur in the tail of the distribution the part of the distribution representing large but rare events and by the difficulty of identifying the range over which power law behavior holds commonly used methods for analyzing power law data such as least squares fitting can produce substantially inaccurate estimates of parameters for power law distributions and even in cases where such methods return accurate answers they are still unsatisfactory because they give no indication of whether the data obey a power law at all here we present a principled statistical framework for discerning and quantifying power law behavior in empirical data our approach combines maximum likelihood fitting methods with goodness of fit tests based on the kolmogorov smirnov ks statistic and likelihood ratios we evaluate the effectiveness of the approach with tests on synthetic data and give critical comparisons to previous approaches we also apply the proposed methods to twenty four real world data sets from a range of different disciplines each of which has been conjectured to follow a power law distribution in some cases we find these conjectures to be consistent with the data while in others the power law is ruled out

on the rate of convergence of optimal solutions of monte carlo approximations of stochastic programs in this paper we discuss monte carlo simulation based approximations of a stochastic programming problem we show that if the corresponding random functions are convex piecewise linear and the distribution is discrete then an optimal solution of the approximating problem provides an exact optimal solution of the true problem with probability one for sufficiently large sample size moreover by using the theory of large deviations we show that the probability of such an event approaches one exponentially fast with increase of the sample size in particular this happens in the case of linear two or multi stage stochastic programming with recourse if the corresponding distributions are discrete the obtained results suggest that in such cases monte carlo simulation based methods could be very efficient we present some numerical examples to illustrate the ideas involved

robustness and regularization of support vector machines we consider regularized support vector machines svms and show that they are precisely equivalent to a new robust optimization formulation we show that this equivalence of robust optimization and regularization has implications for both algorithms and analysis in terms of algorithms the equivalence suggests more general svm like algorithms for classification that explicitly build in protection to noise and at the same time control overfitting on the analysis front the equivalence of robustness and regularization provides a robust optimization interpretation for the success of regularized svms we use this new robustness interpretation of svms to give a new proof of consistency of kernelized svms thus establishing robustness as the reason regularized svms generalize well

robust classification and regression using support vector machines abstract in this paper we investigate the theoretical aspects of robust classification and robust regression using support vector machines given training data x 1 y 1 x l y l where l represents the number of samples x i r n and y i 1 1 for classification or y i r for regression we investigate the training of a support vector machine in the case where bounded perturbation is added to the value of the input x i r n we consider both cases where our training data are either linearly separable and nonlinearly separable respectively we show that we can perform robust classification or regression by using linear or second order cone programming

surname complex network for brazil and portugal abstract we present a study of social networks based on the analysis of brazilian and portuguese family names surnames we construct networks whose nodes are names of families and whose edges represent parental relations between two families from these networks we extract the connectivity distribution clustering coefficient shortest path and centrality we find that the connectivity distribution follows an approximate power law we associate the number of hubs centrality and entropy to the degree of miscegenation in the societies in both countries our results show that portuguese society has a higher miscegenation degree than brazilian society all networks analyzed lead to approximate inverse square power laws in the degree distribution we conclude that the thermodynamic limit is reached for small networks 3 or 4 thousand nodes the assortative mixing of all networks is negative showing that the more connected vertices are connected to vertices with lower connectivity finally the network of surnames presents some small world characteristics

feature selection using probabilistic prediction of support vector regression this paper presents a new wrapper based feature selection method for support vector regression svr using its probabilistic predictions the method computes the importance of a feature by aggregating the difference over the feature space of the conditional density functions of the svr prediction with and without the feature as the exact computation of this importance measure is expensive two approximations are proposed the effectiveness of the measure using these approximations in comparison to several other existing feature selection methods for svr is evaluated on both artificial and real world problems the result of the experiments show that the proposed method generally performs better than or at least as well as the existing methods with notable advantage when the dataset is sparse

concavity and efficient points of discrete distributions in probabilistic programming we consider stochastic programming problems with probabilistic constraints involving integer valued random variables the concept of a p efficient point of a probability distribution is used to derive various equivalent problem formulations next we introduce the concept of r concave discrete probability distributions and analyse its relevance for problems under consideration these notions are used to derive lower and upper bounds for the optimal value of probabilistically constrained stochastic programming problems with discrete random variables the results are illustrated with numerical examples

on constraint sampling in the linear programming approach to approximate dynamic programming in the linear programming approach to approximate dynamic programming one tries to solve a certain linear program the alp that has a relatively small numberk of variables but an intractable numberm of constraints in this paper we study a scheme that samples and imposes a subset ofm

robust convex optimization we study convex optimization problems for which the data is not specified exactly and it is only known to belong to a given uncertainty set u yet the constraints must hold for all possible values of the data from u the ensuing optimization problem is called robust optimization in this paper we lay the foundation of robust convex optimization in the main part of the paper we show that if u is an ellipsoidal uncertainty set then for some of the most important generic convex optimization problems linear programming quadratically constrained programming semidefinite programming and others the corresponding robust convex program is either exactly or approximately a tractable problem which lends itself to efficientalgorithms such as polynomial time interior point methods

on safe tractable approximations of chance constraints abstract a natural way to handle optimization problem with data affected by stochastic uncertainty is to pass to a chance constrained version of the problem where candidate solutions should satisfy the randomly perturbed constraints with probability at least 1 while being attractive from modeling viewpoint chance constrained problems as they are are in general computationally intractable in this survey paper we overview several simulation based and simulation free computationally tractable approximations of chance constrained convex programs primarily those of chance constrained linear conic quadratic and semidefinite programming

nonlinear component analysis as a kernel eigenvalue problem a new method for performing a nonlinear form of principal component analysis is proposed by the use of integral operator kernel functions one can efficiently compute principal components in high dimensional feature spaces related to input space by some nonlinear map for instance the space of all possible five pixel products in 16 16 images we give the derivation of the method and present experimental results on polynomial feature extraction for pattern recognition

worst case distribution analysis of stochastic programs we show that for even quasi concave objective functions the worst case distribution with respect to a family of unimodal distributions of a stochastic programming problem is a uniform distribution this extends the so called uniformity principle of barmish and lagoa 1997 where the objective function is the indicator function of a convex symmetric set

robust regression and lasso lasso or l1 regularized least squares has been explored extensively for its remarkable sparsity properties in this paper it is shown that the solution to lasso in addition to its sparsity has robustness properties it is the solution to a robust optimization problem this has two important consequences first robustness provides a connection of the regularizer to a physical property namely protection from noise this allows a principled selection of the regularizer and in particular generalizations of lasso that also yield convex optimization problems are obtained by considering different uncertainty sets second robustness can itself be used as an avenue for exploring different properties of the solution in particular it is shown that robustness of the solution explains why the solution is sparse the analysis as well as the specific results obtained differ from standard sparsity results providing different geometric intuition furthermore it is shown that the robust optimization formulation is related to kernel density estimation and based on this approach a proof that lasso is consistent is given using robustness directly finally a theorem is proved which states that sparsity and algorithmic stability contradict each other and hence lasso is not stable

nonparametric density estimation the l 1 view differentiation of integrals consistency lower bounds for rates of convergence rates of convergence in l1 the automatic kernel estimate l1 and pointwise convergence estimates related to the kernel estimate and the histogram estimate simulation inequalities and random variate generation the transformed kernel estimate applications in discrimination operations on density estimates estimators based on orthogonal series index

power law distribution of family names in japanese societies abstract we study the frequency distribution of family names from a common data base we count the number of people who share the same family name this is the size of the family we find that i the total number of different family names in a society scales as a power law of the population ii the total number of family names of the same size decreases as the size increases with a power law and iii the relation between size and rank of a family name also shows a power law these scaling properties are found to be consistent for five different regional communities in japan

a class of single class minimax probability machines for novelty detection single class minimax probability machines mpms offer robust novelty detection with distribution free worst case bounds on the probability that a pattern will fall inside the normal region however in practice they are too cautious in labeling patterns as outlying and so have a high false negative rate fnr in this paper we propose a more aggressive version of the single class mpm that bounds the best case probability that a pattern will fall inside the normal region these two mpms can then be used together to delimit the solution space by using the hyperplane lying in the middle of this pair of mpms a better compromise between false positives fps and false negatives fns and between recall and precision can be obtained experiments on the real world data sets show encouraging results

a robust approach based on conditional value at risk measure to statistical learning problems in statistical learning problems measurement errors in the observed data degrade the reliability of estimation there exist several approaches to handle those uncertainties in observations in this paper we propose to use the conditional value at risk cvar measure in order to depress influence of measurement errors and investigate the relation between the resulting cvar minimization problems and some existing approaches in the same framework for the cvar minimization problems which include the computation of integration we apply monte carlo sampling method and obtain their approximate solutions the approximation error bound and convergence property of the solution are proved by vapnik and chervonenkis theory numerical experiments show that the cvar minimization problem can achieve fairly good estimation results compared with several support vector machines in the presence of measurement errors

applications of second order cone programming in a second order cone program socp a linear function is minimized over the intersection of an affine set and the product of second order quadratic cones socps are nonlinear convex problems that include linear and convex quadratic programs as special cases but are less general than semidefinite programs sdps several efficient primaldual interior point methods for socp have been developed in the last few years after reviewing the basic theory of socps we describe general families of problems that tan be recast as socps these include robust linear programming and robust leastsquares problems problems involving sums or maxima of norms or with convex hyperbolic constraints we discuss a variety of engineering applications such as filter design antenna array weight design truss design and grasping forte optimization in robotics we describe an efficient primaldual interior point method for solving socps which shares many of the features of primaldual interior point methods for linear program

ambiguous chance constrained problems and robust optimization in this paper we study ambiguous chance constrained problems where the distributions of the random parameters in the problem are themselves uncertain we focus primarily on the special case where the uncertainty set of the distributions is of the form where p denotes the prohorov metric the ambiguous chance constrained problem is approximated by a robust sampled problem where each constraint is a robust constraint centered at a sample drawn according to the central measure the main contribution of this paper is to show that the robust sampled problem is a good approximation for the ambiguous chance constrained problem with a high probability this result is established using the strassen dudley representation theorem that states that when the distributions of two random variables are close in the prohorov metric one can construct a coupling of the random variables such that the samples are close with a high probability we also show that the robust sampled problem can be solved efficiently both in theory and in practice

the exact feasibility of randomized solutions of uncertain convex programs many optimization problems are naturally delivered in an uncertain framework and one would like to exercise prudence against the uncertainty elements present in the problem in previous contributions it has been shown that solutions to uncertain convex programs that bear a high probability to satisfy uncertain constraints can be obtained at low computational cost through constraint randomization in this paper we establish new feasibility results for randomized algorithms specifically the exact feasibility for the class of the so called fully supported problems is obtained it turns out that all fully supported problems share the same feasibility properties revealing a deep kinship among problems of this class it is further proven that the feasibility of the randomized solutions for all other convex programs can be bounded based on the feasibility for the prototype class of fully supported problems the feasibility result of this paper outperforms previous bounds and is not improvable because it is exact for fully supported problems

consistent identification of wiener systems a machine learning viewpoint wiener system identification has been recently performed by adopting a bayesian semiparametric approach in this framework the linear system entering the first block is given a finite dimensional parametrization while nonparametric gaussian regression is used to estimate the static nonlinearity in the second block in this paper we study the asymptotic behavior of this estimator when the number of noisy output samples tends to infinity without assuming the correctness of the bayesian prior models for this purpose we interpret wiener identification under a machine learning perspective this allows us to extend recent results on function estimation in reproducing kernel hilbert spaces to derive a condition guaranteeing the statistical consistency of the identification procedure we also discuss how the violation of such a condition can lead to useless estimates of the wiener structure

constructing uncertainty sets for robust linear optimization in this paper we propose a methodology for constructing uncertainty sets within the framework of robust optimization for linear optimization problems with uncertain parameters our approach relies on decision maker risk preferences specifically we utilize the theory of coherent risk measures initiated by artzner et al 1999 artzner p f delbaen j eber d heath 1999 coherent measures of risk math finance9 203 228 and show that such risk measures in conjunction with the support of the uncertain parameters are equivalent to explicit uncertainty sets for robust optimization we explore the structure of these sets in detail in particular we study a class of coherent risk measures called distortion risk measures which give rise to polyhedral uncertainty sets of a special structure that is tractable in the context of robust optimization in the case of discrete distributions with rational probabilities which is useful in practical settings when we are sampling from data we show that the class of all distortion risk measures and their corresponding polyhedral sets are generated by a finite number of conditional value at risk cvar measures a subclass of the distortion risk measures corresponds to polyhedral uncertainty sets symmetric through the sample mean we show that this subclass is also finitely generated and can be used to find inner approximations to arbitrary polyhedral uncertainty sets

distributionally robust joint chance constraints with second order moment information we develop tractable semidefinite programming based approximations for distributionally robust individual and joint chance constraints assuming that only the first and second order moments as well as the support of the uncertain parameters are given it is known that robust chance constraints can be conservatively approximated by worst case conditional value at risk cvar constraints we first prove that this approximation is exact for robust individual chance constraints with concave or not necessarily concave quadratic constraint functions and we demonstrate that the worst case cvar can be computed efficiently for these classes of constraint functions next we study the worst case cvar approximation for joint chance constraints this approximation affords intuitive dual interpretations and is provably tighter than two popular benchmark approximations the tightness depends on a set of scaling parameters which can be tuned via a sequential convex optimization algorithm we show that the approximation becomes essentially exact when the scaling parameters are chosen optimally and that the worst case cvar can be evaluated efficiently if the scaling parameters are kept constant we evaluate our joint chance constraint approximation in the context of a dynamic water reservoir control problem and numerically demonstrate its superiority over the two benchmark approximations

robust discrete optimization and network flows we propose an approach to address data uncertainty for discrete optimization and network flow problems that allows controlling the degree of conservatism of the solution and is computationally tractable both practically and theoretically in particular when both the cost coefficients and the data in the constraints of an integer programming problem are subject to uncertainty we propose a robust integer programming problem of moderately larger size that allows controlling the degree of conservatism of the solution in terms of probabilistic bounds on constraint violation when only the cost coefficients are subject to uncertainty and the problem is a 0 1 discrete optimization problem on n variables then we solve the robust counterpart by solving at most n 1 instances of the original problem thus the robust counterpart of a polynomially solvable 0 1 discrete optimization problem remains polynomially solvable in particular robust matching spanning tree shortest path matroid intersection etc are polynomially solvable we also show that the robust counterpart of an np hard approximable 0 1 discrete optimization problem remains approximable finally we propose an algorithm for robust network flows that solves the robust counterpart by solving a polynomial number of nominal minimum cost flow problems in a modified network

new support vector algorithms we propose a new class of support vector algorithms for regression and classification in these algorithms a parameter lets one effectively control the number of support vectors while this can be useful in its own right the parameterization has the additional benefit of enabling us to eliminate one of the other free parameters of the algorithm the accuracy parameter epsilon in the regression case and the regularization constant c in the classification case we describe the algorithms give some theoretical results concerning the meaning and the choice of and report experimental results

simple proof of convergence of the smo algorithm for different svm variants in this brief we give a new proof of the asymptotic convergence of the sequential minimum optimization smo algorithm for both the most violating pair and second order rules to select the pair of coefficients to be updated the proof is more self contained shorter and simpler than previous ones and has a different flavor partially building upon gilbert s original convergence proof of its algorithm to solve the minimum norm problem for convex hulls it is valid for both support vector classification svc and support vector regression which are formulated under a general problem that encompasses them moreover this general problem can be further extended to also cover other support vector machines svm related problems such as svc or one class svms while the convergence proof of the slight variant of smo needed for them remains basically unchanged

probabilistically constrained linear programs and risk adjusted controller design the focal point of this paper is the probabilistically constrained linear program pclp and how it can be applied to control system design under risk constraints the pclp is the counterpart of the classical linear program where it is assumed that there is random uncertainty in the constraints and therefore the deterministic constraints are replaced by probabilistic ones it is shown that for a wide class of probability density functions called log concave symmetric densities the pclp is a convex program an equivalent formulation of the pclp is also presented which provides insight into numerical implementation this concept is applied to control system design it is shown how the results in this paper can be applied to the design of controllers for discrete time systems to obtain a closed loop system with a well defined risk of violating the so called property of superstability furthermore we address the problem of risk adjusted pole placement

efficient methods for robust classification under uncertainty in kernel matrices in this paper we study the problem of designing svm classifiers when the kernel matrix k is affected by uncertainty specifically k is modeled as a positive affine combination of given positive semi definite kernels with the coefficients ranging in a norm bounded uncertainty set we treat the problem using the robust optimization methodology this reduces the uncertain svm problem into a deterministic conic quadratic problem which can be solved in principle by a polynomial time interior point ip algorithm however for large scale classification problems ip methods become intractable and one has to resort to first order gradient type methods the strategy we use here is to reformulate the robust counterpart of the uncertain svm problem as a saddle point problem and employ a special gradient scheme which works directly on the convex concave saddle function the algorithm is a simplified version of a general scheme due to juditski and nemirovski 2011 it achieves an o 1 t2 reduction of the initial error after t iterations a comprehensive empirical study on both synthetic data and real world protein structure data sets show that the proposed formulations achieve the desired robustness and the saddle point based algorithm outperforms the ip method significantly

a robust minimax approach to classification when constructing a classifier the probability of correct classification of future data points should be maximized we consider a binary classification problem where the mean and covariance matrix of each class are assumed to be known no further assumptions are made with respect to the class conditional distributions misclassification probabilities are then controlled in a worst case setting that is under all possible choices of class conditional densities with given mean and covariance matrix we minimize the worst case maximum probability of misclassification of future data points for a linear decision boundary this desideratum is translated in a very direct way into a convex second order cone optimization problem with complexity similar to a support vector machine problem the minimax problem can be interpreted geometrically as minimizing the maximum of the mahalanobis distances to the two classes we address the issue of robustness with respect to estimation errors in the means and covariances of the classes via a simple modification of the input data we also show how to exploit mercer kernels in this setting to obtain nonlinear decision boundaries yielding a classifier which proves to be competitive with current methods including support vector machines an important feature of this method is that a worst case bound on the probability of misclassification of future data is always obtained explicitly

consistency of support vector machines and other regularized kernel classifiers it is shown that various classifiers that are based on minimization of a regularized risk are universally consistent i e they can asymptotically learn in every classification task the role of the loss functions used in these algorithms is considered in detail as an application of our general framework several types of support vector machines svms as well as regularization networks are treated our methods combine techniques from stochastics approximation theory and functional analysis

optimal inequalities in probability theory a convex optimization approach we propose a semidefinite optimization approach to the problem of deriving tight moment inequalities for p x in s for a set s defined by polynomial inequalities and a random vector x defined on omega subseteq cal r n that has a given collection of up to kth order moments in the univariate case we provide optimal bounds on p x in s when the first k moments of x are given as the solution of a semidefinite optimization problem in k 1 dimensions in the multivariate case if the sets s and omega are given by polynomial inequalities we obtain an improving sequence of bounds by solving semidefinite optimization problems of polynomial size in n for fixed k we characterize the complexity of the problem of deriving tight moment inequalities we show that it is np hard to find tight bounds for k geq 4 and omega cal r n and for k geq 2 and omega cal r n when the data in the problem is rational for k 1 and omega cal r n we show that we can find tight upper bounds by solving n convex optimization problems when the set s is convex and we provide a polynomial time algorithm when s and omega are unions of convex sets over which linear functions can be optimized efficiently for the case k 2 and omega cal r n we present an efficient algorithm for finding tight bounds when s is a union of convex sets over which convex quadratic functions can be optimized efficiently

convex risk measures for portfolio optimization and concepts of flexibility due to their axiomatic foundation and their favorable computational properties convex risk measures are becoming a powerful tool in financial risk management in this paper we will review the fundamental structural concepts of convex risk measures within the framework of convex analysis then we will exploit it for deriving strong duality relations in a generic portfolio optimization context in particular the duality relationship can be used for designing new efficient approximation algorithms based on nesterov s smoothing techniques for non smooth convex optimization furthermore the presented concepts enable us to formalize the notion of flexibility as the marginal risk absorption capacity of a technology or available resources

restricted risk measures and robust optimization abstract in this paper we consider characterizations of the robust uncertainty sets associated with coherent and distortion risk measures in this context we show that if we are willing to enforce the coherent or distortion axioms only on random variables that are affine or linear functions of the vector of random parameters we may consider some new variants of the uncertainty sets determined by the classical characterizations we also show that in the finite probability case these variants are simple transformations of the classical sets finally we present results of computational experiments that suggest that the risk measures associated with these new uncertainty sets can help mitigate estimation errors of the conditional value at risk

convex approximations of chance constrained programs we consider a chance constrained problem where one seeks to minimize a convex objective over solutions satisfying with a given close to one probability a system of randomly perturbed convex constraints this problem may happen to be computationally intractable our goal is to build its computationally tractable approximation i e an efficiently solvable deterministic optimization program with the feasible set contained in the chance constrained problem we construct a general class of such convex conservative approximations of the corresponding chance constrained problem moreover under the assumptions that the constraints are affine in the perturbations and the entries in the perturbation vector are independent of each other random variables we build a large deviation type approximation referred to as bernstein approximation of the chance constrained problem this approximation is convex and efficiently solvable we propose a simulation based scheme for bounding the optimal value in the chance constrained problem and report numerical experiments aimed at comparing the bernstein and well known scenario approximation approaches finally we extend our construction to the case of ambiguous chance constrained problems where the random perturbations are independent with the collection of distributions known to belong to a given convex compact set rather than to be known exactly while the chance constraint should be satisfied for every distribution given by this set

support vector interval regression machine for crisp input and output data support vector regression svr has been very successful in function estimation problems for crisp data in this paper we propose a robust method to evaluate interval regression models for crisp input and output data combining the possibility estimation formulation integrating the property of central tendency with the principle of standard svr the proposed method is robust in the sense that outliers do not affect the resulting interval regression furthermore the proposed method is model free method since we do not have to assume the underlying model function for interval nonlinear regression model with crisp input and output in particular this method performs better and is conceptually simpler than support vector interval regression networks svirns which utilize two radial basis function networks to identify the upper and lower sides of data interval five examples are provided to show the validity and applicability of the proposed method

efficient estimation of nonparametric genetic risk function with censored data with an increasing number of causal genes discovered for complex human disorders it is crucial to assess the genetic risk of disease onset for individuals who are carriers of these causal mutations and compare the distribution of age at onset with that in non carriers in many genetic epidemiological studies aiming at estimating causal gene effect on disease the age at onset of disease is subject to censoring in addition some individuals mutation carrier or non carrier status can be unknown due to the high cost of in person ascertainment to collect dna samples or death in older individuals instead the probability of these individuals mutation status can be obtained from various sources when mutation status is missing the available data take the form of censored mixture data recently various methods have been proposed for risk estimation from such data but none is efficient for estimating a nonparametric distribution we propose a fully efficient sieve maximum likelihood estimation method in which we estimate the logarithm of the hazard ratio between genetic mutation groups using b splines while applying nonparametric maximum likelihood estimation for the reference baseline hazard function our estimator can be calculated via an expectation maximization algorithm which is much faster than existing methods we show that our estimator is consistent and semiparametrically efficient and establish its asymptotic distribution simulation studies demonstrate superior performance of the proposed method which is applied to the estimation of the distribution of the age at onset of parkinson s disease for carriers of mutations in the leucine rich repeat kinase 2 gene

risk and complexity in scenario optimization scenario optimization is a broad methodology to perform optimization based on empirical knowledge one collects previous cases called scenarios for the set up in which optimization is being performed and makes a decision that is optimal for the cases that have been collected for convex optimization a solid theory has been developed that provides guarantees of performance and constraint satisfaction of the scenario solution in this paper we open a new direction of investigation the risk that a performance is not achieved or that constraints are violated is studied jointly with the complexity as precisely defined in the paper of the solution it is shown that the joint probability distribution of risk and complexity is concentrated in such a way that the complexity carries fundamental information to tightly judge the risk this result is obtained without requiring extra knowledge on the underlying optimization problem than that carried by the scenarios in particular no extra knowledge on the distribution by which scenarios are generated is assumed so that the result is broadly applicable this deep seated result unveils a fundamental and general structure of data driven optimization and suggests practical approaches for risk assessment

random effects models for longitudinal data models for the analysis of longitudinal data must recognize the relationship between serial observations on the same unit multivariate models with general covariance structure are often difficult to apply to highly unbalanced data whereas two stage random effects models can be used easily in two stage models the probability distributions for the response vectors of different individuals belong to a single family but some random effects parameters vary across individuals with a distribution specified at the second stage a general family of models is discussed which includes both growth models and repeated measures models as special cases a unified approach to fitting these models based on a combination of empirical bayes and maximum likelihood estimation of model parameters and using the em algorithm is discussed two examples are taken from a current epidemiological study of the health effects of air pollution

robust support vector regression for uncertain input and output data in this paper a robust support vector regression rsvr method with uncertain input and output data is studied first the data uncertainties are investigated under a stochastic framework and two linear robust formulations are derived linear formulations robust to ellipsoidal uncertainties are also considered from a geometric perspective second kernelized rsvr formulations are established for nonlinear regression problems both linear and nonlinear formulations are converted to second order cone programming problems which can be solved efficiently by the interior point method simulation demonstrates that the proposed method outperforms existing rsvrs in the presence of both input and output data uncertainties

fast support vector data descriptions for novelty detection support vector data description svdd has become a very attractive kernel method due to its good results in many novelty detection problems however the decision function of svdd is expressed in terms of the kernel expansion which results in a run time complexity linear in the number of support vectors for applications where fast real time response is needed how to speed up the decision function is crucial this paper aims at dealing with the issue of reducing the testing time complexity of svdd a method called fast svdd f svdd is proposed unlike the traditional methods which all try to compress a kernel expansion into one with fewer terms the proposed f svdd directly finds the preimage of a feature vector and then uses a simple relationship between this feature vector and the svdd sphere center to re express the center with a single vector the decision function of f svdd contains only one kernel term and thus the decision boundary of f svdd is only spherical in the original space hence the run time complexity of the f svdd decision function is no longer linear in the support vectors but is a constant no matter how large the training set size is in this paper we also propose a novel direct preimage finding method which is noniterative and involves no free parameters the unique preimage can be obtained in real time by the proposed direct method without taking trial and error for demonstration several real world data sets and a large scale data set the extended mit face data set are used in experiments in addition a practical industry example regarding liquid crystal display micro defect inspection is also used to compare the applicability of svdd and our proposed f svdd when faced with mass data input the results are very encouraging

nonparametric estimation of component distributions in a multivariate mixture suppose k variate data are drawn from a mixture of two distributions each having independent components it is desired to estimate the univariate marginal distributions in each of the products as well as the mixing proportion this is the setting of two class fully parametrized latent models that has been proposed for estimating the distributions of medical test results when disease status is unavailable the problem is one of inference in a mixture of distributions without training data and until now it has been tackled only in a fully parametric setting we investigate the possibility of using nonparametric methods of course when k 1 the problem is not identifiable from a nonparametric viewpoint we show that the problem is almost identifiable when k 2 there the set of all possible representations can be expressed in terms of any one of those representations as a two parameter family furthermore it is proved that when k geq3 the problem is nonparametrically identifiable under particularly mild regularity conditions in this case we introduce root n consistent nonparametric estimators of the 2k univariate marginal distributions and the mixing proportion finite sample and asymptotic properties of the estimators are described

worst case conditional value at risk with application to robust portfolio management this paper considers the worst case conditional value at risk cvar in the situation where only partial information on the underlying probability distribution is available the minimization of the worst case cvar under mixture distribution uncertainty box uncertainty and ellipsoidal uncertainty are investigated the application of the worst case cvar to robust portfolio optimization is proposed and the corresponding problems are cast as linear programs and second order cone programs that can be solved efficiently market data simulation and monte carlo simulation examples are presented to illustrate the proposed approach

estimating the support of a high dimensional distribution suppose you are given some data set drawn from an underlying probability distribution p and you want to estimate a simple subset s of input space such that the probability that a test point drawn from p lies outside of s equals some a priori specified value between 0 and 1 we propose a method to approach this problem by trying to estimate a function f that is positive on s and negative on the complement the functional form of f is given by a kernel expansion in terms of a potentially small subset of the training data it is regularized by controlling the length of the weight vector in an associated feature space the expansion coefficients are found by solving a quadratic programming problem which we do by carrying out sequential optimization over pairs of input patterns we also provide a theoretical analysis of the statistical performance of our algorithm the algorithm is a natural extension of the support vector algorithm to the case of unlabeled data

second order cone programming approaches for handling missing and uncertain data we propose a novel second order cone programming formulation for designing robust classifiers which can handle uncertainty in observations similar formulations are also derived for designing regression functions which are robust to uncertainties in the regression setting the proposed formulations are independent of the underlying distribution requiring only the existence of second order moments these formulations are then specialized to the case of missing values in observations for both classification and regression problems experiments show that the proposed formulations outperform imputation

robust twin support vector machine for pattern classification in this paper we proposed a new robust twin support vector machine called r twsvm via second order cone programming formulations for classification which can deal with data with measurement noise efficiently preliminary experiments confirm the robustness of the proposed method and its superiority to the traditional robust svm in both computation time and classification accuracy remarkably since there are only inner products about inputs in our dual problems this makes us apply kernel trick directly for nonlinear cases simultaneously we does not need to solve the extra inverse of matrices which is totally different with existing twsvms in addition we also show that the twsvms are the special case of our robust model and simultaneously give a new dual form of twsvm by degenerating r twsvm which successfully overcomes the existing shortcomings of twsvm

extended fuzzy regression models using regularization method regularization approach to regression can be easily found in statistics and information science literatures the technique of regularization was introduced as a way of controlling the smoothness properties of regression function the main purpose of this paper is to extend the models of diamond inform sci 46 1988 141 to fuzzy multiple linear regression models and to extend the model f3 with numerical input of diamond inform sci 46 1988 141 to fuzzy nonlinear model here all extensions are based on regularization technique

robust mean covariance solutions for stochastic optimization we provide a method for deriving robust solutions to certain stochastic optimization problems based on mean covariance information about the distributions underlying the uncertain vector of returns we prove that for a general class of objective functions the robust solutions amount to solving a certain deterministic parametric quadratic program we first prove a general projection property for multivariate distributions with given means and covariances which reduces our problem to optimizing a univariate mean variance robust objective this allows us to use known univariate results in the multidimensional setting and to add new results in this direction in particular we characterize a general class of objective functions the so called one or two point support functions for which the robust objective is reduced to a deterministic optimization problem in one variable finally we adapt a result from geoffrion 1967a to reduce the main problem to a parametric quadratic program in particular our results are true for increasing concave utilities with convex or concave convex derivatives closed form solutions are obtained for special discontinuous criteria motivated by bonus and commission based incentive schemes for portfolio management we also investigate a multiproduct pricing application which motivates extensions of our results for the case of nonnegative and decision dependent returns

a computable plug in estimator of minimum volume sets for novelty detection a minimum volume set of a probability density is a region of minimum size among the regions covering a given probability mass of the density effective methods for finding the minimum volume sets are very useful for detecting failures or anomalies in commercial and security applications a problem known as novelty detection one theoretical approach of estimating the minimum volume set is to use a density level set where a kernel density estimator is plugged into the optimization problem that yields the appropriate level such a plug in estimator is not of practical use because solving the corresponding minimization problem is usually intractable a modified plug in estimator was proposed by hyndman in 1996 to overcome the computation difficulty of the theoretical approach but is not well studied in the literature in this paper we provide theoretical support to this estimator by showing its asymptotic consistency we also show that this estimator is very competitive to other existing novelty detection methods through an extensive empirical study

ambiguity in risk preferences in robust stochastic optimization abstract we consider robust stochastic optimization problems for risk averse decision makers where there is ambiguity about both the decision maker s risk preferences and the underlying probability distribution we propose and analyze a robust optimization problem that accounts for both types of ambiguity first we derive a duality theory for this problem class and identify random utility functions as the lagrange multipliers second we turn to the computational aspects of this problem we show how to evaluate our robust optimization problem exactly in some special cases and then we consider some tractable relaxations for the general case finally we apply our model to both the newsvendor and portfolio optimization problems and discuss its implications

fuzzy least squares abstract several models for simple least squares fitting of fuzzy valued data are developed criteria are given for when fuzzy data sets can be fitted to the models and analogues of the normal equations are derived

chance constrained uncertain classification via robust optimization this paper studies the problem of constructing robust classifiers when the training is plagued with uncertainty the problem is posed as a chance constrained program ccp which ensures that the uncertain data points are classified correctly with high probability unfortunately such a ccp turns out to be intractable the key novelty is in employing bernstein bounding schemes to relax the ccp as a convex second order cone program whose solution is guaranteed to satisfy the probabilistic constraint prior to this work only the chebyshev based relaxations were exploited in learning algorithms bernstein bounds employ richer partial information and hence can be far less conservative than chebyshev bounds due to this efficient modeling of uncertainty the resulting classifiers achieve higher classification margins and hence better generalization methodologies for classifying uncertain test data points and error measures for evaluating classifiers robust to uncertain data are discussed experimental results on synthetic and real world datasets show that the proposed classifiers are better equipped to handle data uncertainty and outperform state of the art in many cases

a distributional interpretation of robust optimization motivated by data driven decision making and sampling problems we investigate probabilistic interpretations of robust optimization ro we establish a connection between ro and distributionally robust stochastic programming drsp showing that the solution to any ro problem is also a solution to a drsp problem specifically we consider the case where multiple uncertain parameters belong to the same fixed dimensional space and find the set of distributions of the equivalent drsp problem the equivalence we derive enables us to construct ro formulations for sampled problems as in stochastic programming and machine learning that are statistically consistent even when the original sampled problem is not in the process this provides a systematic approach for tuning the uncertainty set the equivalence further provides a probabilistic explanation for the common shrinkage heuristic where the uncertainty set used in an ro problem is a shrunken version of the original uncertainty set

robust convex quadratically constrained programs in this paper we study robust convex quadratically constrained programs a subset of the class of robust convex programs introduced by ben tal and nemirovski 4 in contrast to 4 where it is shown that such robust problems can be formulated as semidefinite programs our focus in this paper is to identify uncertainty sets that allow this class of problems to be formulated as second order cone programs socp we propose three classes of uncertainty sets for which the robust problem can be reformulated as an explicit socp and present examples where these classes of uncertainty sets are natural

constructing risk measures from uncertainty sets we illustrate the correspondence between uncertainty sets in robust optimization and some popular risk measures in finance and show how robust optimization can be used to generalize the concepts of these risk measures we also show that by using properly defined uncertainty sets in robust optimization models one can construct coherent risk measures and address the issue of the computational tractability of the resulting formulations our results have implications for efficient portfolio optimization under different measures of risk