large scale multiple kernel learning while classical kernel based learning algorithms are based on a single kernel in practice it is often desirable to use multiple kernels lanckriet et al 2004 considered conic combinations of kernel matrices for classification leading to a convex quadratically constrained quadratic program we show that it can be rewritten as a semi infinite linear program that can be efficiently solved by recycling the standard svm implementations moreover we generalize the formulation and our method to a larger class of problems including regression and one class classification experimental results show that the proposed algorithm works for hundred thousands of examples or hundreds of kernels to be combined and helps for automatic model selection improving the interpretability of the learning result in a second part we discuss general speed up mechanism for svms especially when used with sparse feature maps as appear for string kernels allowing us to train a string kernel svm on a 10 million real world splice data set from computational biology we integrated multiple kernel learning in our machine learning toolbox shogun for which the source code is publicly available at http www fml tuebingen mpg de raetsch projects shogun

ell_ p ell_ q penalty for sparse linear and sparse multiple kernel multitask learning recently there has been much interest around multitask learning mtl problem with the constraints that tasks should share a common sparsity profile such a problem can be addressed through a regularization framework where the regularizer induces a joint sparsity pattern between task decision functions we follow this principled framework and focus on lp lq with 0 p 1 and 1 q 2 mixed norms as sparsity inducing penalties our motivation for addressing such a larger class of penalty is to adapt the penalty to a problem at hand leading thus to better performances and better sparsity pattern for solving the problem in the general multiple kernel case we first derive a variational formulation of the l1 lq penalty which helps us in proposing an alternate optimization algorithm although very simple the latter algorithm provably converges to the global minimum of the l1 lq penalized problem for the linear case we extend existing works considering accelerated proximal gradient to this penalty our contribution in this context is to provide an efficient scheme for computing the l1 lq proximal operator then for the more general case when 0 p 1 we solve the resulting nonconvex problem through a majorization minimization approach the resulting algorithm is an iterative scheme which at each iteration solves a weighted l1 lq sparse mtl problem empirical evidences from toy dataset and real word datasets dealing with brain computer interface single trial electroencephalogram classification and protein subcellular localization show the benefit of the proposed approaches and algorithms

learning the kernel with hyperkernels this paper addresses the problem of choosing a kernel suitable for estimation with a support vector machine hence further automating machine learning this goal is achieved by defining a reproducing kernel hilbert space on the space of kernels itself such a formulation leads to a statistical estimation problem similar to the problem of minimizing a regularized risk functional we state the equivalent representer theorem for the choice of kernels and present a semidefinite programming formulation of the resulting optimization problem several recipes for constructing hyperkernels are provided as well as the details of common machine learning problems experimental results for classification regression and novelty detection on uci data show the feasibility of our approach

a distance spectrum interpretation of turbo codes the performance of turbo codes is addressed by examining the code s distance spectrum the error floor that occurs at moderate signal to noise ratios is shown to be a consequence of the relatively low free distance of the code it is also shown that the error floor can be lowered by increasing the size of the interleaver without changing the free distance of the code alternatively the free distance of the code may be increased by using primitive feedback polynomials the excellent performance of turbo codes at low signal to noise ratios is explained in terms of the distance spectrum the interleaver in the turbo encoder is shown to reduce the number of low weight codewords through a process called spectral thinning this thinned distance spectrum results in the free distance asymptote being the dominant performance parameter for low and moderate signal to noise ratios

on the convergence rate of l p norm multiple kernel learning we derive an upper bound on the local rademacher complexity of lp norm multiple kernel learning which yields a tighter excess risk bound than global approaches previous local approaches analyzed the case p 1 only while our analysis covers all cases 1 p assuming the different feature mappings corresponding to the different kernels to be uncorrelated we also show a lower bound that shows that the bound is tight and derive consequences regarding excess loss namely fast convergence rates of the order o n 1 a where is the minimum eigenvalue decay rate of the individual kernels

weighted a search unifying view and application the a algorithm is a well known heuristic best first search method several performance accelerated extensions of the exact a approach are known interesting examples are approximate algorithms where the heuristic function used is inflated by a weight often referred to as weighted a these methods guarantee a bounded suboptimality as a technical contribution this paper presents the previous results related to weighted a from authors like pohl pearl kim likhachev and others in a more condensed and unifying form with this unified view a novel general bound on suboptimality of the result is derived in the case of avoiding any reopening of expanded states for e 0 this bound is 1 e n 2 where n is an upper bound on an optimal solution length binary decision diagrams bdds are well known to ai e g from set based exploration of sparse memory and symbolic manipulation of state spaces the problem of exact or approximate bdd minimization is introduced as a possible new challenge for heuristic search like many classical ai domains this problem is motivated by real world applications several variants of weighted a search are applied to problems of bdd minimization and the more classical domains like blocksworld and sliding tile puzzles for bdd minimization the comparison of the evaluated methods also includes previous heuristic and simulation based methods such as rudell s hill climbing based sifting algorithm simulated annealing and evolutionary algorithms a discussion of the results obtained in the different problem domains gives our experiences with weighted a which is of value for the ai practitioner

l p norm multiple kernel learning learning linear combinations of multiple kernels is an appealing strategy when the right choice of features is unknown previous approaches to multiple kernel learning mkl promote sparse kernel combinations to support interpretability and scalability unfortunately this l1 norm mkl is rarely observed to outperform trivial baselines in practical applications to allow for robust kernel mixtures that generalize well we extend mkl to arbitrary norms we devise new insights on the connection between several existing mkl formulations and develop two efficient interleaved optimization strategies for arbitrary norms that is lp norms with p 1 this interleaved optimization is much faster than the commonly used wrapper approaches as demonstrated on several data sets a theoretical analysis and an experiment on controlled artificial data shed light on the appropriateness of sparse non sparse and l norm mkl in various scenarios importantly empirical applications of lp norm mkl to three real world problems from computational biology show that non sparse mkl achieves accuracies that surpass the state of the art data sets source code to reproduce the experiments implementations of the algorithms and further information are available at http doc ml tu berlin de nonsparse mkl

prox method with rate of convergence o 1 t for variational inequalities with lipschitz continuous monotone operators and smooth convex concave saddle point problems we propose a prox type method with efficiency estimate o epsilon 1 for approximating saddle points of convex concave c 1 1 functions and solutions of variational inequalities with monotone lipschitz continuous operators application examples include matrix games eigenvalue minimization and computing the lovasz capacity number of a graph and these are illustrated by numerical experiments with large scale matrix games and lovasz capacity problems

probabilistic low rank multitask learning in this paper we consider the problem of learning multiple related tasks simultaneously with the goal of improving the generalization performance of individual tasks the key challenge is to effectively exploit the shared information across multiple tasks as well as preserve the discriminative information for each individual task to address this we propose a novel probabilistic model for multitask learning mtl that can automatically balance between low rank and sparsity constraints the former assumes a low rank structure of the underlying predictive hypothesis space to explicitly capture the relationship of different tasks and the latter learns the incoherent sparse patterns private to each task we derive and perform inference via variational bayesian methods experimental results on both regression and classification tasks on real world applications demonstrate the effectiveness of the proposed method in dealing with the mtl problems

design of interleavers for turbo codes iterative interleaver growth algorithms of polynomial complexity this paper addresses the problem of designing interleavers for parallel concatenated convolutional codes pcccs tailored to specific constituent codes we start by establishing the role of the interleaver in the pccc and the various parameters that influence the performance of the pccc with a given interleaver subsequently we define a canonical form of the interleaving engine denoted as the finite state permuter fsp and demonstrate the minimal delay property of this canonical form for any given permutation we present a procedure for deriving the canonical fsp engine we address the issue of implementation of the fsp and propose a very simple structure for the fsp next using the structural property of the fsp engine we develop a systematic iterative technique for construction of interleavers with a complexity that is polynomial in the interleaver size subsequently we develop a cost function that coupled with the iterative interleaver growth procedure can be used to design optimized interleavers for pcccs we provide examples of application of the interleaver design technique and compare the designed interleavers with some of the interleavers of comparable size found in the literature

localized multiple kernel learning with dynamical clustering and matrix regularization localized multiple kernel learning lmkl is an attractive strategy for combining multiple heterogeneous features with regard to their discriminative power for each individual sample however the learning of numerous local solutions may not scale well even for a moderately sized training set and the independently learned local models may suffer from overfitting hence in existing local methods the distributed samples are typically assumed to share the same weights and various unsupervised clustering methods are applied as preprocessing in this paper to enable the learner to discover and benefit from the underlying local coherence and diversity of the samples we incorporate the clustering procedure into the canonical support vector machine based lmkl framework then to explore the relatedness among different samples which has been ignored in a vector ell p norm analysis we organize the cluster specific kernel weights into a matrix and introduce a matrix based extension of the ell p norm for constraint enforcement by casting the joint optimization problem as a problem of alternating optimization we show how the cluster structure is gradually revealed and how the matrix regularized kernel weights are obtained a theoretical analysis of such a regularizer is performed using a rademacher complexity bound and complementary empirical experiments on real world data sets demonstrate the effectiveness of our technique

improving the variable ordering of obdds is np complete ordered binary decision diagrams are a useful representation of boolean functions if a good variable ordering is known variable orderings are computed by heuristic algorithms and then improved with local search and simulated annealing algorithms this approach is based on the conjecture that the following problem is np complete given an obdd g representing f and a size bound s does there exist an obdd g respecting an arbitrary variable ordering representing f with at most s nodes this conjecture is proved

variable sparsity kernel learning this paper presents novel algorithms and applications for a particular class of mixed norm regularization based multiple kernel learning mkl formulations the formulations assume that the given kernels are grouped and employ l1 norm regularization for promoting sparsity within rkhs norms of each group and ls s 2 norm regularization for promoting non sparse combinations across groups various sparsity levels in combining the kernels can be achieved by varying the grouping of kernels hence we name the formulations as variable sparsity kernel learning vskl formulations while previous attempts have a non convex formulation here we present a convex formulation which admits efficient mirror descent md based solving techniques the proposed md based algorithm optimizes over product of simplices and has a computational complexity of o m2ntot log nmax e2 where m is no training data points nmax ntot are the maximum no kernels in any group total no kernels respectively and e is the error in approximating the objective a detailed proof of convergence of the algorithm is also presented experimental results show that the vskl formulations are well suited for multi modal learning tasks like object categorization results also show that the md based algorithm outperforms state of the art mkl solvers in terms of computational efficiency

multi task learning for classification with dirichlet process priors consider the problem of learning logistic regression models for multiple classification tasks where the training data set for each task is not drawn from the same statistical distribution in such a multi task learning mtl scenario it is necessary to identify groups of similar tasks that should be learned jointly relying on a dirichlet process dp based statistical model to learn the extent of similarity between classification tasks we develop computationally efficient algorithms for two different forms of the mtl problem first we consider a symmetric multi task learning smtl situation in which classifiers for multiple tasks are learned jointly using a variational bayesian vb algorithm second we consider an asymmetric multi task learning amtl formulation in which the posterior density function from the smtl model parameters from previous tasks is used as a prior for a new task this approach has the significant advantage of not requiring storage and use of all previous data from prior tasks the amtl formulation is solved with a simple markov chain monte carlo mcmc construction experimental results on two real life mtl problems indicate that the proposed algorithms a automatically identify subgroups of related tasks whose training data appear to be drawn from similar distributions and b are more accurate than simpler approaches such as single task learning pooling of data across all tasks and simplified approximations to dp

a note on tailbiting codes and their feedback encoders tailbiting codes encoded by feedback convolutional encoders are studied a condition for when tailbiting will work is given and it is described how the encoder starting state can be obtained for feedback encoders in both controller and observer canonical forms finally results from a search for systematic feedback encoders that encode tailbiting codes with good decoding bit error probabilities are presented

properties of complexity measures for prams and wrams abstract the computation of boolean functions by parallel computers with shared memory prams and wrams is considered in particular complexity measures for parallel computers like critical and sensitive complexity are compared with other complexity measures for boolean functions like branching program depth and length of prime implicants and clauses the relations between these complexity measures and their asymptotic behaviour are investigated for the classes of boolean functions monotone functions and symmetric functions

the nonapproximability of obdd minimization the size of ordered binary decision diagrams obdds is determined by the chosen variable ordering a poor choice may cause an obdd to be too large to fit into the available memory the decision variant of the variable ordering problem is known to be np complete we strengthen this result by showing that unless p np for each constant c 1 there is no polynomial time approximation algorithm with the performance ratio c for the variable ordering problem i e no polynomial time algorithm that guarantees the computation of a variable ordering so that the resulting obdd size is larger than the minimum size by a factor of at most c this result justifies also from a theoretical point of view the use of heuristics for the variable ordering problem

lower bounds for depth restricted branching programs abstract we present a new method for proving lower bounds on the complexity of branching programs and consider k times only branching programs while exponential and nearly exponential lower bounds on the complexity of one time only branching programs were proved for many problems there are still missing methods of proving lower bounds for k times only programs k 1 we prove exponential lower bounds for k times only branching programs which have the additional restriction that the input bits are read k times yet blockwise and in each block in the same order this is done both for the algebraic decision problem polyn d n n prime d n whether a given mapping g f n f n is a polynomial over f n of degree at most d and for the corresponding monotone problem over quadratic boolean matrices as a consequence we obtain a sharp bound of order n log n on the communication complexity of polyn n 0 1 2

finding the optimal variable ordering for binary decision diagrams the ordered binary decision diagram is a canonical representation for boolean functions presented by r e bryant 1985 as a compact representation for a broad class of interesting functions derived from circuits however the size of the diagram is very sensitive to the choice of ordering on the variables hence for some applications such as differential cascode voltage switch dcvs trees it becomes extremely important to find the ordering leading to the most compact representation an algorithm for this problem with time complexity o n sup 2 3 sup n is presented this represents an improvement over the previous best algorithm

on the complexity of vlsi implementations and graph representations of boolean functions with application to integer multiplication lower bound results on boolean function complexity under two different models are discussed the first is an abstraction of tradeoffs between chip area and speed in very large scale integrated vlsi circuits the second is the ordered binary decision diagram obdd representation used as a data structure for symbolically representing and manipulating boolean functions the lower bounds demonstrate the fundamental limitations of vlsi as an implementation medium and that of the obdd as a data structure it is shown that the same technique used to prove that any vlsi implementation of a single output boolean function has area time complexity at sup 2 omega n sup 2 also proves that any obdd representation of the function has omega c sup n vertices for some c 1 but that the converse is not true an integer multiplier for word size n with outputs numbered 0 least significant through 2n 1 most significant is described for the boolean function representing either output i 1 or output 2n i 1 where 1

pareto path multitask multiple kernel learning a traditional and intuitively appealing multitask multiple kernel learning mt mkl method is to optimize the sum thus the average of objective functions with partially shared kernel function which allows information sharing among the tasks we point out that the obtained solution corresponds to a single point on the pareto front pf of a multiobjective optimization problem which considers the concurrent optimization of all task objectives involved in the multitask learning mtl problem motivated by this last observation and arguing that the former approach is heuristic we propose a novel support vector machine mt mkl framework that considers an implicitly defined set of conic combinations of task objectives we show that solving our framework produces solutions along a path on the aforementioned pf and that it subsumes the optimization of the average of objective functions as a special case using the algorithms we derived we demonstrate through a series of experimental results that the framework is capable of achieving a better classification performance when compared with other similar mtl approaches

estimating the support of a high dimensional distribution suppose you are given some data set drawn from an underlying probability distribution p and you want to estimate a simple subset s of input space such that the probability that a test point drawn from p lies outside of s equals some a priori specified value between 0 and 1 we propose a method to approach this problem by trying to estimate a function f that is positive on s and negative on the complement the functional form of f is given by a kernel expansion in terms of a potentially small subset of the training data it is regularized by controlling the length of the weight vector in an associated feature space the expansion coefficients are found by solving a quadratic programming problem which we do by carrying out sequential optimization over pairs of input patterns we also provide a theoretical analysis of the statistical performance of our algorithm the algorithm is a natural extension of the support vector algorithm to the case of unlabeled data

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

interleavers for turbo codes using permutation polynomials over integer rings a class of deterministic interleavers for turbo codes tcs based on permutation polynomials over spl zopf sub n is introduced the main characteristic of this class of interleavers is that they can be algebraically designed to fit a given component code moreover since the interleaver can be generated by a few simple computations storage of the interleaver tables can be avoided by using the permutation polynomial based interleavers the design of the interleavers reduces to the selection of the coefficients of the polynomials it is observed that the performance of the tcs using these permutation polynomial based interleavers is usually dominated by a subset of input weight 2m error events the minimum distance and its multiplicity or the first few spectrum lines of this subset are used as design criterion to select good permutation polynomials a simple method to enumerate these error events for small m is presented searches for good interleavers are performed the decoding performance of these interleavers is close to s random interleavers for long frame sizes for short frame sizes the new interleavers outperform s random interleavers

unveiling turbo codes some results on parallel concatenated coding schemes a parallel concatenated coding scheme consists of two simple constituent systematic encoders linked by an interleaver the input bits to the first encoder are scrambled by the interleaver before entering the second encoder the codeword of the parallel concatenated code consists of the input bits to the first encoder followed by the parity check bits of both encoders this construction can be generalized to any number of constituent codes parallel concatenated schemes employing two convolutional codes as constituent codes in connection with an iterative decoding algorithm of complexity comparable to that of the constituent codes have been previously shown to yield remarkable coding gains close to theoretical limits they have been named and are known as turbo codes we propose a method to evaluate an upper bound to the bit error probability of a parallel concatenated coding scheme averaged over all interleavers of a given length the analytical bounding technique is then used to shed some light on some crucial questions which have been floating around in the communications community since the proposal of turbo codes

new deterministic interleaver designs for turbo codes it is well known that an interleaver with random properties quite often generated by pseudo random algorithms is one of the essential building blocks of turbo codes however randomly generated interleavers have two major drawbacks lack of an adequate analysis that guarantees their performance and lack of a compact representation that leads to a simple implementation we present several new classes of deterministic interleavers of length n with construction complexity o n that permute a sequence of bits with nearly the same statistical distribution as a random interleaver and perform as well as or better than the average of a set of random interleavers the new classes of deterministic interleavers have a very simple representation based on quadratic congruences and hence have a structure that allows the possibility of analysis as well as a straightforward implementation using the new interleavers a turbo code of length 16384 that is only 0 7 db away from capacy at a bit error rate ber of 10 sup 5 is constructed we also generalize the theory of previously known deterministic interleavers that are based on block interleavers and we apply this theory to the construction of a nonrandom turbo code of length 16384 with a very regular structure whose performance is only 1 1 db away from capacity at a ber of 10 sup 5

multiple kernel learning algorithms in recent years several methods have been proposed to combine multiple kernels instead of using a single one these different kernels may correspond to using different notions of similarity or may be using information coming from multiple sources different representations or different feature subsets in trying to organize and highlight the similarities and differences between them we give a taxonomy of and review several multiple kernel learning algorithms we perform experiments on real data sets for better illustration and comparison of existing algorithms we see that though there may not be large differences in terms of accuracy there is difference between them in complexity as given by the number of stored support vectors the sparsity of the solution as given by the number of used kernels and training time complexity we see that overall using multiple kernels instead of a single one is useful and believe that combining kernels in a nonlinear or data dependent way seems more promising than linear combination in fusing information provided by simple linear kernels whereas linear methods are more reasonable when combining complex gaussian kernels