approximate analytical solutions of systems of pdes by homotopy analysis method in this paper the homotopy analysis method ham is applied to obtain series solutions to linear and nonlinear systems of first and second order partial differential equations pdes the ham solutions contain an auxiliary parameter which provides a convenient way of controlling the convergence region of series solutions it is shown in particular that the solutions obtained by the variational iteration method vim are only special cases of the ham solutions

the sample average approximation method for stochastic discrete optimization in this paper we study a monte carlo simulation based approach to stochastic discrete optimization problems the basic idea of such methods is that a random sample is generated and the expected value function is approximated by the corresponding sample average function the obtained sample average optimization problem is solved and the procedure is repeated several times until a stopping criterion is satisfied we discuss convergence rates stopping rules and computational complexity of this procedure and present a numerical example for the stochastic knapsack problem

a simulation based approach to two stage stochastic programming with recourse in this paper we consider stochastic programming problems where the objective function is given as an expected value function we discuss monte carlo simulation based approaches to a numerical solution of such problems in particular we discuss in detail and present numerical results for two stage stochastic programming with recourse where the random data have a continuous multivariate normal distribution we think that the novelty of the numerical approach developed in this paper is twofold first various variance reduction techniques are applied in order to enhance the rate of convergence successful application of those techniques is what makes the whole approach numerically feasible second a statistical inference is developed and applied to estimation of the error validation of optimality of a calculated solution and statistically based stopping criteria for an iterative alogrithm 1998 the mathematical programming society inc published by elsevier science b v

the entire regularization path for the support vector machine the support vector machine svm is a widely used tool for classification many efficient implementations exist for fitting a two class svm model the user has to supply values for the tuning parameters the regularization cost parameter and the kernel parameters it seems a common practice is to use a default value for the cost parameter often leading to the least restrictive model in this paper we argue that the choice of the cost parameter can be critical we then derive an algorithm that can fit the entire path of svm solutions for every value of the cost parameter with essentially the same computational cost as fitting one svm model we illustrate our algorithm on some examples and use our representation to give further insight into the range of svm solutions

use of the zero norm with linear models and kernel methods we explore the use of the so called zero norm of the parameters of linear models in learning minimization of such a quantity has many uses in a machine learning context for variable or feature selection minimizing training error and ensuring sparsity in solutions we derive a simple but practical method for achieving these goals and discuss its relationship to existing techniques of minimizing the zero norm the method boils down to implementing a simple modification of vanilla svm namely via an iterative multiplicative rescaling of the training data applications we investigate which aid our discussion include variable and feature selection on biological microarray data and multicategory classification

estimation of functional connectivity in fmri data using stability selection based sparse partial correlation with elastic net penalty abstract characterizing interactions between multiple brain regions is important for understanding brain function functional connectivity measures based on partial correlation provide an estimate of the linear conditional dependence between brain regions after removing the linear influence of other regions estimation of partial correlations is however difficult when the number of regions is large as is now increasingly the case with a growing number of large scale brain connectivity studies to address this problem we develop novel methods for estimating sparse partial correlations between multiple regions in fmri data using elastic net penalty spc en which combines l1 and l2 norm regularization we show that l1 norm regularization in spc en provides sparse interpretable solutions while l2 norm regularization improves the sensitivity of the method when the number of possible connections between regions is larger than the number of time points and when pair wise correlations between brain regions are high an issue with regularization based methods is choosing the regularization parameters which in turn determine the selection of connections between brain regions to address this problem we deploy novel stability selection methods to infer significant connections between brain regions we also compare the performance of spc en with existing methods which use only l1 norm regularization spc l1 on simulated and experimental datasets detailed simulations show that the performance of spc en measured in terms of sensitivity and accuracy is superior to spc l1 especially at higher rates of feature prevalence application of our methods to resting state fmri data obtained from 22 healthy adults shows that spc en reveals a modular architecture characterized by strong inter hemispheric links distinct ventral and dorsal stream pathways and a major hub in the posterior medial cortex features that were missed by conventional methods taken together our findings suggest that spc en provides a powerful tool for characterizing connectivity involving a large number of correlated regions that span the entire brain

a new probabilistic approach in rank regression with optimal bayesian partitioning in this paper we consider the supervised learning task which consists in predicting the normalized rank of a numerical variable we introduce a novel probabilistic approach to estimate the posterior distribution of the target rank conditionally to the predictors we turn this learning task into a model selection problem for that we define a 2d partitioning family obtained by discretizing numerical variables and grouping categorical ones and we derive an analytical criterion to select the partition with the highest posterior probability we show how these partitions can be used to build univariate predictors and multivariate ones under a naive bayes assumption we also propose a new evaluation criterion for probabilistic rank estimators based on the logarithmic score we show that such criterion presents the advantage to be minored which is not the case of the logarithmic score computed for probabilistic value estimator a first set of experimentations on synthetic data shows the good properties of the proposed criterion and of our partitioning approach a second set of experimentations on real data shows competitive performance of the univariate and selective naive bayes rank estimators projected on the value range compared to methods submitted to a recent challenge on probabilistic metric regression tasks our approach is applicable for all regression problems with categorical or numerical predictors it is particularly interesting for those with a high number of predictors as it automatically detects the variables which contain predictive information it builds pertinent predictors of the normalized rank of the numerical target from one or several predictors as the criteria selection is regularized by the presence of a prior and a posterior term it does not suffer from overfitting

solving the inverse problem of identifying an unknown source term in a parabolic equation the inverse problem of determining an unknown source term in a parabolic equation with the leading coefficient depending on time and space variables under a final overdetermination condition is considered the series pattern solution of the problem is obtained by using the weighted homotopy analysis method wham a description of the method for solving the problem and identifying the unknown source term is derived finally several examples are presented to illustrate this method

applying tensor based morphometry to parametric surfaces can improve mri based disease diagnosis many methods have been proposed for computer assisted diagnostic classification full tensor information and machine learning with 3d maps derived from brain images may help detect subtle differences or classify subjects into different groups here we develop a new approach to apply tensor based morphometry to parametric surface models for diagnostic classification we use this approach to identify cortical surface features for use in diagnostic classifiers first with holomorphic 1 forms we compute an efficient and accurate conformal mapping from a multiply connected mesh to the so called slit domain next the surface parameterization approach provides a natural way to register anatomical surfaces across subjects using a constrained harmonic map to analyze anatomical differences we then analyze the full riemannian surface metric tensors which retain multivariate information on local surface geometry as the number of voxels in a 3d image is large sparse learning is a promising method to select a subset of imaging features and to improve classification accuracy focusing on vertices with greatest effect sizes we train a diagnostic classifier using the surface features selected by an l1 norm based sparse learning method stability selection is applied to validate the selected feature sets we tested the algorithm on mri derived cortical surfaces from 42 subjects with genetically confirmed williams syndrome and 40 age matched controls multivariate statistics on the local tensors gave greater effect sizes for detecting group differences relative to other tbm based statistics including analysis of the jacobian determinant and the largest eigenvalue of the surface metric our method also gave reasonable classification results relative to the jacobian determinant the pair of eigenvalues of the jacobian matrix and volume features this analysis pipeline may boost the power of morphometry studies and may assist with image based classification

accelerated block coordinate relaxation for regularized optimization we discuss minimization of a smooth function to which is added a separable regularization function that induces structure in the solution a block coordinate relaxation approach with proximal linearized subproblems yields convergence to critical points while identification of the optimal manifold under a nondegeneracy condition allows acceleration techniques to be applied on a reduced space the work is motivated by experience with an algorithm for regularized logistic regression and computational results for the algorithm on problems of this type are presented

multi class discriminant kernel learning via convex programming regularized kernel discriminant analysis rkda performs linear discriminant analysis in the feature space via the kernel trick its performance depends on the selection of kernels in this paper we consider the problem of multiple kernel learning mkl for rkda in which the optimal kernel matrix is obtained as a linear combination of pre specified kernel matrices we show that the kernel learning problem in rkda can be formulated as convex programs first we show that this problem can be formulated as a semidefinite program sdp based on the equivalence relationship between rkda and least square problems in the binary class case we propose a convex quadratically constrained quadratic programming qcqp formulation for kernel learning in rkda a semi infinite linear programming silp formulation is derived to further improve the efficiency we extend these formulations to the multi class case based on a key result established in this paper that is the multi class rkda kernel learning problem can be decomposed into a set of binary class kernel learning problems which are constrained to share a common kernel based on this decomposition property sdp formulations are proposed for the multi class case furthermore it leads naturally to qcqp and silp formulations as the performance of rkda depends on the regularization parameter we show that this parameter can also be optimized in a joint framework with the kernel extensive experiments have been conducted and analyzed and connections to other algorithms are discussed

homotopy perturbation method a new nonlinear analytical technique in this paper a new perturbation method is proposed in contrast to the traditional perturbation methods this technique does not require a small parameter in an equation in this method according to the homotopy technique a homotopy with an imbedding parameter email protected 0 1 is constructed and the imbedding parameter is considered as a small parameter so the method is called the homotopy perturbation method which can take the full advantages of the traditional perturbation methods and homotopy techniques to illustrate its effectiveness and its convenience a duffing equation with high order of nonlinearity is used the result reveals that its first order of approximation obtained by the proposed method is valid uniformly even for very large parameter and is more accurate than the perturbation solutions

pac bayesian generalisation error bounds for gaussian process classification approximate bayesian gaussian process gp classification techniques are powerful non parametric learning methods similar in appearance and performance to support vector machines based on simple probabilistic models they render interpretable results and can be embedded in bayesian frameworks for model selection feature selection etc in this paper by applying the pac bayesian theorem of mcallester 1999a we prove distribution free generalisation error bounds for a wide range of approximate bayesian gp classification techniques we also provide a new and much simplified proof for this powerful theorem making use of the concept of convex duality which is a backbone of many machine learning techniques we instantiate and test our bounds for two particular gpc techniques including a recent sparse method which circumvents the unfavourable scaling of standard gp algorithms as is shown in experiments on a real world task the bounds can be very tight for moderate training sample sizes to the best of our knowledge these results provide the tightest known distribution free error bounds for approximate bayesian gpc methods giving a strong learning theoretical justification for the use of these techniques

on the use of stochastic hessian information in optimization methods for machine learning this paper describes how to incorporate sampled curvature information in a newton cg method and in a limited memory quasi newton method for statistical learning the motivation for this work stems from supervised machine learning applications involving a very large number of training points we follow a batch approach also known in the stochastic optimization literature as a sample average approximation approach curvature information is incorporated in two subsampled hessian algorithms one based on a matrix free inexact newton iteration and one on a preconditioned limited memory bfgs iteration a crucial feature of our technique is that hessian vector multiplications are carried out with a significantly smaller sample size than is used for the function and gradient the efficiency of the proposed methods is illustrated using a machine learning application involving speech recognition

on the rate of convergence of optimal solutions of monte carlo approximations of stochastic programs in this paper we discuss monte carlo simulation based approximations of a stochastic programming problem we show that if the corresponding random functions are convex piecewise linear and the distribution is discrete then an optimal solution of the approximating problem provides an exact optimal solution of the true problem with probability one for sufficiently large sample size moreover by using the theory of large deviations we show that the probability of such an event approaches one exponentially fast with increase of the sample size in particular this happens in the case of linear two or multi stage stochastic programming with recourse if the corresponding distributions are discrete the obtained results suggest that in such cases monte carlo simulation based methods could be very efficient we present some numerical examples to illustrate the ideas involved

multivalued behavior for a two level system using homotopy analysis method abstract we use the homotopy analysis method ham to solve the system of equations modeling the two level system and extract results which will pinpoint to turbulent behavior we look at multivalued solutions as indicative of turbulence or turbulent like behavior we take different specific cases which result in multivalued velocities the solutions are in series form and application of ham ensures convergence in some region

a practical bayesian framework for backpropagation networks a quantitative and practical bayesian framework is described for learning of mappings in feedforward networks the framework makes possible 1 objective comparisons between solutions using alternative network architectures 2 objective stopping rules for network pruning or growing procedures 3 objective choice of magnitude and type of weight decay terms or additive regularizers for penalizing large weights etc 4 a measure of the effective number of well determined parameters in a model 5 quantified estimates of the error bars on network parameters and on network output and 6 objective comparisons with alternative learning and interpolation models such as splines and radial basis functions the bayesian evidence automatically embodies occam s razor penalizing overflexible and overcomplex models the bayesian approach helps detect poor underlying assumptions in learning models for learning models well matched to a problem a good correlation between generalization ability and the bayesian evidence is obtained

the effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity features individualized behavioral cognitive prediction using machine learning ml regression approaches is becoming increasingly applied the specific ml regression algorithm and sample size are two key factors that non trivially influence prediction accuracies however the effects of the ml regression algorithm and sample size on individualized behavioral cognitive prediction performance have not been comprehensively assessed to address this issue the present study included six commonly used ml regression algorithms ordinary least squares ols regression least absolute shrinkage and selection operator lasso regression ridge regression elastic net regression linear support vector regression lsvr and relevance vector regression rvr to perform specific behavioral cognitive predictions based on different sample sizes specifically the publicly available resting state functional mri rs fmri dataset from the human connectome project hcp was used and whole brain resting state functional connectivity rsfc or rsfc strength rsfcs were extracted as prediction features twenty five sample sizes ranged from 20 to 700 were studied by sub sampling from the entire hcp cohort the analyses showed that rsfc based lasso regression performed remarkably worse than the other algorithms and rsfcs based ols regression performed markedly worse than the other algorithms regardless of the algorithm and feature type both the prediction accuracy and its stability exponentially increased with increasing sample size the specific patterns of the observed algorithm and sample size effects were well replicated in the prediction using re testing fmri data data processed by different imaging preprocessing schemes and different behavioral cognitive scores thus indicating excellent robustness generalization of the effects the current findings provide critical insight into how the selected ml regression algorithm and sample size influence individualized predictions of behavior cognition and offer important guidance for choosing the ml regression algorithm or sample size in relevant investigations

model selection for regression with continuous kernel functions using the modulus of continuity this paper presents a new method of model selection for regression problems using the modulus of continuity for this purpose we suggest the prediction risk bounds of regression models using the modulus of continuity which can be interpreted as the complexity of functions we also present the model selection criterion referred to as the modulus of continuity information criterion mcic which is derived from the suggested prediction risk bounds the suggested mcic provides a risk estimate using the modulus of continuity for a trained regression model or an estimation function while other model selection criteria such as the aic and bic use structural information such as the number of training parameters as a result the suggested mcic is able to discriminate the performances of trained regression models even with the same structure of training models to show the effectiveness of the proposed method the simulation for function approximation using the multilayer perceptrons mlps was conducted through the simulation for function approximation it was demonstrated that the suggested mcic provides a good selection tool for nonlinear regression models even with the limited size of data

hybrid deterministic stochastic methods for data fitting many structured data fitting applications require the solution of an optimization problem involving a sum over a potentially large number of measurements incremental gradient algorithms offer inexpensive iterations by sampling a subset of the terms in the sum these methods can make great progress initially but often slow as they approach a solution in contrast full gradient methods achieve steady convergence at the expense of evaluating the full objective and gradient on each iteration we explore hybrid methods that exhibit the benefits of both approaches rate of convergence analysis shows that by controlling the sample size in an incremental gradient algorithm it is possible to maintain the steady convergence rates of full gradient methods we detail a practical quasi newton implementation based on this approach numerical experiments illustrate its potential benefits

feature selection with ensembles artificial variables and redundancy elimination predictive models benefit from a compact non redundant subset of features that improves interpretability and generalization modern data sets are wide dirty mixed with both numerical and categorical predictors and may contain interactive effects that require complex models this is a challenge for filters wrappers and embedded feature selection methods we describe details of an algorithm using tree based ensembles to generate a compact subset of non redundant features parallel and serial ensembles of trees are combined into a mixed method that can uncover masking and detect features of secondary effect simulated and actual examples illustrate the effectiveness of the approach

a numerical solution of the coupled sine gordon equation using the modified decomposition method the modified decomposition method has been implemented for solving a coupled sine gordon equation we consider a system of coupled sine gordon equation which models one dimensional nonlinear wave processes in two component media by using an initial value system the numerical solutions of coupled sine gordon equation have been represented graphically

optimizing methods for linking cinematic features to fmri data abstract one of the challenges of naturalistic neurosciences using movie viewing experiments is how to interpret observed brain activations in relation to the multiplicity of time locked stimulus features as previous studies have shown less inter subject synchronization across viewers of random video footage than story driven films new methods need to be developed for analysis of less story driven contents to optimize the linkage between our fmri data collected during viewing of a deliberately non narrative silent film at land by maya deren 1944 and its annotated content we combined the method of elastic net regularization with the model driven linear regression and the well established data driven independent component analysis ica and inter subject correlation isc methods in the linear regression analysis both ic and region of interest roi time series were fitted with time series of a total of 36 binary valued and one real valued tactile annotation of film features the elastic net regularization and cross validation were applied in the ordinary least squares linear regression in order to avoid over fitting due to the multicollinearity of regressors the results were compared against both the partial least squares pls regression and the un regularized full model regression non parametric permutation testing scheme was applied to evaluate the statistical significance of regression we found statistically significant correlation between the annotation model and 9 ics out of 40 ics regression analysis was also repeated for a large set of cubic rois covering the grey matter both ic and roi based regression analyses revealed activations in parietal and occipital regions with additional smaller clusters in the frontal lobe furthermore we found elastic net based regression more sensitive than pls and un regularized regression since it detected a larger number of significant ics and rois along with the isc ranking methods our regression analysis proved a feasible method for ordering the ics based on their functional relevance to the annotated cinematic features the novelty of our method is in comparison to the hypothesis driven manual pre selection and observation of some individual regressors biased by choice in applying data driven approach to all content features simultaneously we found especially the combination of regularized regression and ica useful when analyzing fmri data obtained using non narrative movie stimulus with a large set of complex and correlated features

accuracy of pseudo inverse covariance learning a random matrix theory analysis for many learning problems estimates of the inverse population covariance are required and often obtained by inverting the sample covariance matrix increasingly for modern scientific data sets the number of sample points is less than the number of features and so the sample covariance is not invertible in such circumstances the moore penrose pseudo inverse sample covariance matrix constructed from the eigenvectors corresponding to nonzero sample covariance eigenvalues is often used as an approximation to the inverse population covariance matrix the reconstruction error of the pseudo inverse sample covariance matrix in estimating the true inverse covariance can be quantified via the frobenius norm of the difference between the two the reconstruction error is dominated by the smallest nonzero sample covariance eigenvalues and diverges as the sample size becomes comparable to the number of features for high dimensional data we use random matrix theory techniques and results to study the reconstruction error for a wide class of population covariance matrices we also show how bagging and random subspace methods can result in a reduction in the reconstruction error and can be combined to improve the accuracy of classifiers that utilize the pseudo inverse sample covariance matrix we test our analysis on both simulated and benchmark data sets

generalizing database relational algebra for the treatment of incomplete or uncertain information and vague queries abstract this paper deals with relational databases which are extended in the sense that fuzzily known values are allowed for attributes precise as well as partial imprecise uncertain knowledge concerning the value of the attributes are represented by means of 0 1 valued possibility distributions in zadeh s sense thus we have to manipulate ordinary relations on cartesian products of sets of fuzzy subsets rather than fuzzy relations besides vague queries whose contents are also represented by possibility distributions can be taken into account the basic operations of relational algebra union intersection cartesian product projection and selection are extended in order to deal with partial information and vague queries approximate equalities and inequalities modeled by fuzzy relations can also be taken into account in the selection operation then the main features of a query language based on the extended relational algebra are presented an illustrative example is provided this approach which enables a very general treatment of relational databases with fuzzy attribute values makes an extensive use of dual possibility and necessity measures

non negative matrix factorization with sparseness constraints non negative matrix factorization nmf is a recently developed technique for finding parts based linear representations of non negative data although it has successfully been applied in several applications it does not always result in parts based representations in this paper we show how explicitly incorporating the notion of sparseness improves the found decompositions additionally we provide complete matlab code both for standard nmf and for our extension our hope is that this will further the application of these methods to solving novel data analysis problems

six problems for causal inference from fmri neuroimaging e g fmri data are increasingly used to attempt to identify not only brain regions of interest rois that are especially active during perception cognition and action but also the qualitative causal relations among activity in these regions known as effective connectivity friston 1994 previous investigations and anatomical and physiological knowledge may somewhat constrain the possible hypotheses but there often remains a vast space of possible causal structures to find actual effective connectivity relations search methods must accommodate indirect measurements of nonlinear time series dependencies feedback multiple subjects possibly varying in identified regions of interest and unknown possible location dependent variations in bold response delays we describe combinations of procedures that under these conditions find feed forward sub structure characteristic of a group of subjects the method is illustrated with an empirical data set and confirmed with simulations of time series of non linear randomly generated effective connectivities with feedback subject to random differences of bold delays with regions of interest missing at random for some subjects measured with noise approximating the signal to noise ratio of the empirical data

on the degrees of freedom of the lasso we study the effective degrees of freedom of the lasso in the framework of stein s unbiased risk estimation sure we show that the number of nonzero coefficients is an unbiased estimate forthe degrees of freedom of the lasso a conclusion that requires no special assumption on the predictors in addition the unbiased estimator is shown to be asymptotically consistent with these results on hand various model selection criteria cp aic and bic are available which along with the lars algorithm provide a principled and efficient approach to obtaining the optimal lasso fit with the computational effort of a single ordinary least squares fit

the variational iteration method for solving linear and nonlinear systems of pdes in this work the variational iteration method vim is used for analytic treatment of the linear and nonlinear systems of partial differential equations the method reduces the calculation size and overcomes the difficulty of handling nonlinear terms numerical examples are examined to highlight the significant features of the vim method the method shows improvements over existing numerical techniques

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

sparse regularization techniques provide novel insights into outcome integration processes abstract by exploiting information that is contained in the spatial arrangement of neural activations multivariate pattern analysis mvpa can detect distributed brain activations which are not accessible by standard univariate analysis recent methodological advances in mvpa regularization techniques have made it feasible to produce sparse discriminative whole brain maps with highly specific patterns furthermore the most recent refinement the graph net explicitly takes the 3d structure of fmri data into account here these advanced classification methods were applied to a large fmri sample n 70 in order to gain novel insights into the functional localization of outcome integration processes while the beneficial effect of differential outcomes is well studied in trial and error learning outcome integration in the context of instruction based learning has remained largely unexplored in order to examine neural processes associated with outcome integration in the context of instruction based learning two groups of subjects underwent functional imaging while being presented with either differential or ambiguous outcomes following the execution of varying stimulus response instructions while no significant univariate group differences were found in the resulting fmri dataset l1 regularized sparse classifiers performed significantly above chance and also clearly outperformed the standard l2 regularized dense support vector machine on this whole brain between subject classification task moreover additional l2 regularization via the elastic net and spatial regularization by the graph net improved interpretability of discriminative weight maps but were accompanied by reduced classification accuracies most importantly classification based on sparse regularization facilitated the identification of highly specific regions differentially engaged under ambiguous and differential outcome conditions comprising several prefrontal regions previously associated with probabilistic learning rule integration and reward processing additionally a detailed post hoc analysis of these regions revealed that distinct activation dynamics underlay the processing of ambiguous relative to differential outcomes together these results show that l1 regularization can improve classification performance while simultaneously providing highly specific and interpretable discriminative activation patterns

probabilistic independent component analysis for functional magnetic resonance imaging we present an integrated approach to probabilistic independent component analysis ica for functional mri fmri data that allows for nonsquare mixing in the presence of gaussian noise in order to avoid overfitting we employ objective estimation of the amount of gaussian noise through bayesian analysis of the true dimensionality of the data i e the number of activation and non gaussian noise sources this enables us to carry out probabilistic modeling and achieves an asymptotically unique decomposition of the data it reduces problems of interpretation as each final independent component is now much more likely to be due to only one physical or physiological process we also describe other improvements to standard ica such as temporal prewhitening and variance normalization of timeseries the latter being particularly useful in the context of dimensionality reduction when weak activation is present we discuss the use of prior information about the spatiotemporal nature of the source processes and an alternative hypothesis testing approach for inference using gaussian mixture models the performance of our approach is illustrated and evaluated on real and artificial fmri data and compared to the spatio temporal accuracy of results obtained from classical ica and glm analyses

newton s method for large bound constrained optimization problems we analyze a trust region version of newton s method for bound constrained problems our approach relies on the geometry of the feasible set not on the particular representation in terms of constraints the convergence theory holds for linearly constrained problems and yields global and superlinear convergence without assuming either strict complementarity or linear independence of the active constraints we also show that the convergence theory leads to an efficient implementation for large bound constrained problems

variable number sample path optimization the sample path method is one of the most important tools in simulation based optimization the basic idea of the method is to approximate the expected simulation output by the average of sample observations with a common random number sequence in this paper we describe a new variant of powell s unconstrained optimization by quadratic approximation uobyqa method which integrates a bayesian variable number sample path vnsp scheme to choose appropriate number of samples at each iteration the statistically accurate scheme determines the number of simulation runs and guarantees the global convergence of the algorithm the vnsp scheme saves a significant amount of simulation operations compared to general purpose fixed number sample path methods we present numerical results based on the new algorithm

preventing over fitting during model selection via bayesian regularisation of the hyper parameters while the model parameters of a kernel machine are typically given by the solution of a convex optimisation problem with a single global optimum the selection of good values for the regularisation and kernel parameters is much less straightforward fortunately the leave one out cross validation procedure can be performed or a least approximated very efficiently in closed form for a wide variety of kernel learning methods providing a convenient means for model selection leave one out cross validation based estimates of performance however generally exhibit a relatively high variance and are therefore prone to over fitting in this paper we investigate the novel use of bayesian regularisation at the second level of inference adding a regularisation term to the model selection criterion corresponding to a prior over the hyper parameter values where the additional regularisation parameters are integrated out analytically results obtained on a suite of thirteen real world and synthetic benchmark data sets clearly demonstrate the benefit of this approach

boosting as a regularized path to a maximum margin classifier in this paper we study boosting methods from a new perspective we build on recent work by efron et al to show that boosting approximately and in some cases exactly minimizes its loss criterion with an l1 constraint on the coefficient vector this helps understand the success of boosting with early stopping as regularized fitting of the loss criterion for the two most commonly used criteria exponential and binomial log likelihood we further show that as the constraint is relaxed or equivalently as the boosting iterations proceed the solution converges in the separable case to an l1 optimal separating hyper plane we prove that this l1 optimal separating hyper plane has the property of maximizing the minimal l1 margin of the training data as defined in the boosting literature an interesting fundamental similarity between boosting and kernel support vector machines emerges as both can be described as methods for regularized optimization in high dimensional predictor space using a computational trick to make the calculation practical and converging to margin maximizing solutions while this statement describes svms exactly it applies to boosting only approximately

strong rules for discarding predictors in lasso type problems summary we consider rules for discarding predictors in lasso regression and related problems for computational efficiency el ghaoui and his colleagues have proposed safe rules based on univariate inner products between each predictor and the outcome which guarantee that a coefficient will be 0 in the solution vector this provides a reduction in the number of variables that need to be entered into the optimization we propose strong rules that are very simple and yet screen out far more predictors than the safe rules this great practical improvement comes at a price the strong rules are not foolproof and can mistakenly discard active predictors i e predictors that have non zero coefficients in the solution we therefore combine them with simple checks of the karush kuhn tucker conditions to ensure that the exact solution to the convex problem is delivered of course any approximate screening method can be combined with the karush kuhn tucker conditions to ensure the exact solution the strength of the strong rules lies in the fact that in practice they discard a very large number of the inactive predictors and almost never commit mistakes we also derive conditions under which they are foolproof strong rules provide substantial savings in computational time for a variety of statistical optimization problems

partial covariance based functional connectivity computation using ledoit wolf covariance regularization abstract functional connectivity refers to shared signals among brain regions and is typically assessed in a task free state functional connectivity commonly is quantified between signal pairs using pearson correlation however resting state fmri is a multivariate process exhibiting a complicated covariance structure partial covariance assesses the unique variance shared between two brain regions excluding any widely shared variance hence is appropriate for the analysis of multivariate fmri datasets however calculation of partial covariance requires inversion of the covariance matrix which in most functional connectivity studies is not invertible owing to rank deficiency here we apply ledoit wolf shrinkage l2 regularization to invert the high dimensional bold covariance matrix we investigate the network organization and brain state dependence of partial covariance based functional connectivity although rsns are conventionally defined in terms of shared variance removal of widely shared variance surprisingly improved the separation of rsns in a spring embedded graphical model this result suggests that pair wise unique shared variance plays a heretofore unrecognized role in rsn covariance organization in addition application of partial correlation to fmri data acquired in the eyes open vs eyes closed states revealed focal changes in uniquely shared variance between the thalamus and visual cortices this result suggests that partial correlation of resting state bold time series reflect functional processes in addition to structural connectivity

partial correlation estimation by joint sparse regression models this article features online supplementary material in this article we propose a computationally efficient approach space sparse partial correlation estimation for selecting nonzero partial correlations under the high dimension low sample size setting this method assumes the overall sparsity of the partial correlation matrix and employs sparse regression techniques for model fitting we illustrate the performance of space by extensive simulation studies it is shown that space performs well in both nonzero partial correlation selection and the identification of hub variables and also outperforms two existing methods we then apply space to a microarray breast cancer dataset and identify a set of hub genes that may provide important insights on genetic regulatory networks finally we prove that under a set of suitable assumptions the proposed procedure is asymptotically consistent in terms of model selection and parameter estimation

measuring interregional functional connectivity using coherence and partial coherence analyses of fmri data understanding functional connectivity within the brain is crucial to understanding neural function even the simplest cognitive operations are supported by highly distributed neural circuits we developed a novel method to measure task related functional interactions between neural regions by applying coherence and partial coherence analyses to functional magnetic resonance imaging fmri data coherence and partial coherence are spectral measures that estimate the linear time invariant lti relationship between time series they can be used to generate maps of task specific connectivity associated with seed regions of interest rois these maps may then be compared across tasks revealing nodes with task related changes of connectivity to the seed roi to validate the method we applied it to an event related fmri data set acquired while subjects performed two sequence tapping tasks one of which required more bimanual coordination areas showing increased functional connectivity with both tasks were the same as those showing increased activity furthermore though there were no significant differences in mean activity between the two tasks significant increases in interhemispheric coherence were found between the primary motor m1 and premotor pm regions for the task requiring more bimanual coordination this increase in interhemispheric connectivity is supported by other brain imaging techniques as well as patient studies

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

model selection beyond the bayesian frequentist divide the principle of parsimony also known as ockham s razor has inspired many theories of model selection yet such theories all making arguments in favor of parsimony are based on very different premises and have developed distinct methodologies to derive algorithms we have organized challenges and edited a special issue of jmlr and several conference proceedings around the theme of model selection in this editorial we revisit the problem of avoiding overfitting in light of the latest results we note the remarkable convergence of theories as different as bayesian theory minimum description length bias variance tradeoff structural risk minimization and regularization in some approaches we also present new and interesting examples of the complementarity of theories leading to hybrid algorithms neither frequentist nor bayesian or perhaps both frequentist and bayesian

sample size selection in optimization methods for machine learning this paper presents a methodology for using varying sample sizes in batch type optimization methods for large scale machine learning problems the first part of the paper deals with the delicate issue of dynamic sample selection in the evaluation of the function and gradient we propose a criterion for increasing the sample size based on variance estimates obtained during the computation of a batch gradient we establish an o 1 epsilon complexity bound on the total cost of a gradient method the second part of the paper describes a practical newton method that uses a smaller sample to compute hessian vector products than to evaluate the function and the gradient and that also employs a dynamic sampling technique the focus of the paper shifts in the third part of the paper to l 1 regularized problems designed to produce sparse solutions we propose a newton like method that consists of two phases a minimalistic gradient projection phase that identifies zero variables and subspace phase that applies a subsampled hessian newton iteration in the free variables numerical tests on speech recognition problems illustrate the performance of the algorithms

comparing measures of sparsity sparsity of representations of signals has been shown to be a key concept of fundamental importance in fields such as blind source separation compression sampling and signal analysis the aim of this paper is to compare several commonly used sparsity measures based on intuitive attributes intuitively a sparse representation is one in which a small number of coefficients contain a large proportion of the energy in this paper six properties are discussed robin hood scaling rising tide cloning bill gates and babies each of which a sparsity measure should have the main contributions of this paper are the proofs and the associated summary table which classify commonly used sparsity measures based on whether or not they satisfy these six propositions only two of these measures satisfy all six the pq mean with p les 1 q 1 and the gini index

on the homotopy analysis method for nonlinear problems a powerful easy to use analytic tool for nonlinear problems in general namely the homotopy analysis method is further improved and systematically described through a typical nonlinear problem i e the algebraically decaying viscous boundary layer flow due to a moving sheet two rules the rule of solution expression and the rule of coefficient ergodicity are proposed which play important roles in the frame of the homotopy analysis method and simplify its applications in science and engineering an explicit analytic solution is given for the first time with recursive formulas for coefficients this analytic solution agrees well with numerical results and can be regarded as a definition of the solution of the considered nonlinear problem

compression based averaging of selective naive bayes classifiers the naive bayes classifier has proved to be very effective on many real data applications its performance usually benefits from an accurate estimation of univariate conditional probabilities and from variable selection however although variable selection is a desirable feature it is prone to overfitting in this paper we introduce a bayesian regularization technique to select the most probable subset of variables compliant with the naive bayes assumption we also study the limits of bayesian model averaging in the case of the naive bayes assumption and introduce a new weighting scheme based on the ability of the models to conditionally compress the class labels the weighting scheme on the models reduces to a weighting scheme on the variables and finally results in a naive bayes classifier with soft variable selection extensive experiments show that the compression based averaged classifier outperforms the bayesian model averaging scheme

characterizing the hemodynamic response effects of presentation rate sampling procedure and the possibility of ordering brain activity based on relative timing rapid presentation event related functional mri er fmri allows neuroimaging methods based on hemodynamics to employ behavioral task paradigms typical of cognitive settings however the sluggishness of the hemodynamic response and its variance provide constraints on how er fmri can be applied in a series of two studies estimates of the hemodynamic response in or near the primary visual and motor cortices were compared across various paradigms and sampling procedures to determine the limits of er fmri procedures and more generally to describe the behavior of the hemodynamic response the temporal profile of the hemodynamic response was estimated across overlapping events by solving a set of linear equations within the general linear model no assumptions about the shape were made in solving the equations following estimation of the temporal profile the amplitude and timing were modeled using a gamma function results indicated that 1 within a region for a given subject estimation of the hemodynamic response is extremely stable for both amplitude r 2 0 98 and time to peak r 2 0 95 from one series of measurements to the next and slightly less stable for estimation of time to onset r 2 0 60 2 as the trial presentation rate changed from those spaced 20 s apart to temporally overlapping trials the hemodynamic response amplitude showed a small but significant decrease trial onsets spaced on average 5 s apart showed a 17 25 reduction in amplitude compared to those spaced 20 s apart power analysis indicated that the increased number of trials at fast rates outweighs this decrease in amplitude if statistically reliable response detection is the goal 3 knowledge of the amplitude and timing of the hemodynamic response in one region failed to predict those properties in another region even for within subject comparisons 4 across subjects the amplitude of the response showed no significant correlation with timing of the response for either time to onset or time to peak estimates 5 the within region stability of the response was sufficient to allow offsets in the timing of the response to be detected that were under a second placing event related fmri methods in a position to answer questions about the change in relative timing between regions

multivalued dependencies in fuzzy relational databases and lossless join decomposition abstract the fuzzy multivalued dependency is defined on the lines of classical multivalued dependency to resolve the redundancy and inconsistency in fuzzy databases the implication problem of fmvd s has been examined and a set of sound and complete inference axioms has been proposed we next examine lossless join problem of fuzzy relations in the presence of fuzzy functional depandencies and fuzzy multivalued dependencies and obtain decomposition of relation schemes

model selection in kernel based regression using the influence function recent results about the robustness of kernel methods involve the analysis of influence functions by definition the influence function is closely related to leave one out criteria in statistical learning the latter is often used to assess the generalization of a method in statistics the influence function is used in a similar way to analyze the statistical efficiency of a method links between both worlds are explored the influence function is related to the first term of a taylor expansion higher order influence functions are calculated a recursive relation between these terms is found characterizing the full taylor expansion it is shown how to evaluate influence functions at a specific sample distribution to obtain an approximation of the leave one out error a specific implementation is proposed using a l1 loss in the selection of the hyperparameters and a huber loss in the estimation procedure the parameter in the huber loss controlling the degree of robustness is optimized as well the resulting procedure gives good results even when outliers are present in the data

proximity relations in the fuzzy relational database model abstract the fuzzy relational model of buckles and petry is a rigorous scheme for incorporating non ideal or fuzzy information in a relational database in addition to providing a consistent scheme for representing fuzzy information in the relational structure the model possesses two critical properties that hold for classical relational databases these properties are that no two tuples have identical interpretations and each relational operation has a unique result the fuzzy relational model relies on similarity relations for each scalar domain in the fuzzy database these relations are reflexive symmetric and max min transitive in addition to introducing fuzziness into the relational model each similarity relation induces equivalence classes in its domain it is the existence of these equivalence classes that provides the model with the important properties possessed by classical relational databases in this paper we extend the fuzzy relational database model of buckles and petry to deal with proximity relations for scalar domains since reflexivity and symmetry are the only constraints placed on proximity relations they generalize the notion of similarity relations we show that it is possible to induce equivalence classes from proximity relations thus the nice properties of the fuzzy relational model of buckles and petry are preserved furthermore the removal of the max min transitivity restriction also provides database users with more freedom to express their value structures

deep neural network predicts emotional responses of the human brain from functional magnetic resonance imaging abstract an artificial neural network with multiple hidden layers known as a deep neural network or dnn was employed as a predictive model dnnp for the first time to predict emotional responses using whole brain functional magnetic resonance imaging fmri data from individual subjects during fmri data acquisition 10 healthy participants listened to 80 international affective digital sound stimuli and rated their own emotions generated by each sound stimulus in terms of the arousal dominance and valence dimensions the whole brain spatial patterns from a general linear model i e beta valued maps for each sound stimulus and the emotional response ratings were used as the input and output for the dnnp respectively based on a nested five fold cross validation scheme the paired input and output data were divided into training three fold validation one fold and test one fold data the dnnp was trained and optimized using the training and validation data and was tested using the test data the pearson s correlation coefficients between the rated and predicted emotional responses from our dnnp model with weight sparsity optimization mean standard error 0 52 0 02 for arousal 0 51 0 03 for dominance and 0 51 0 03 for valence with an input denoising level of 0 3 and a mini batch size of 1 were significantly greater than those of dnn models with conventional regularization schemes including elastic net regularization 0 15 0 05 0 15 0 06 and 0 21 0 04 for arousal dominance and valence respectively those of shallow models including logistic regression 0 11 0 04 0 10 0 05 and 0 17 0 04 for arousal dominance and valence respectively average of logistic regression and sparse logistic regression and those of support vector machine based predictive models svmps 0 12 0 06 0 06 0 06 and 0 10 0 06 for arousal dominance and valence respectively average of linear and non linear svmps this difference was confirmed to be significant with a bonferroni corrected p value of less than 0 001 from a one way analysis of variance anova and subsequent paired t test the weights of the trained dnnps were interpreted and input patterns that maximized or minimized the output of the dnnps i e the emotional responses were estimated based on a binary classification of each emotion category e g high arousal vs low arousal the error rates for the dnnp 31 2 1 3 for arousal 29 0 1 7 for dominance and 28 6 3 0 for valence were significantly lower than those for the linear svmp 44 7 2 0 50 7 1 7 and 47 4 1 9 for arousal dominance and valence respectively and the non linear svmp 48 8 2 3 52 2 1 9 and 46 4 1 3 for arousal dominance and valence respectively as confirmed by the bonferroni corrected p

latency in sensitive ica group independent component analysis of fmri data in the temporal frequency domain independent component analysis ica a data driven approach utilizing high order statistical moments to find maximally independent sources has found fruitful application in functional magnetic resonance imaging fmri a limitation of the standard fmri ica model is that a given component s time course is required to have the same delay at every voxel as spatially varying delays svds may be found in fmri data using an ica model with a fixed temporal delay for each source will have two implications larger svds can result in the splitting of regions with different delays into different components second smaller svds can result in a biased ica amplitude estimate due to only a slight delay difference we propose a straightforward approach for incorporating this prior temporal information and removing the limitation of a fixed source delay by performing ica on the amplitude spectrum of the original fmri data thus removing latency information a latency map is then estimated for each component using the resulting component images and the raw data we show that voxels with similar time courses but different delays are grouped into the same component additionally when using traditional ica the amplitudes of motor areas are diminished due to systematic delay differences between visual and motor areas the amplitudes are more accurately estimated when using a latency insensitive ica approach the resulting time courses the component maps and the latency maps may prove useful as an addition to the collection of methods for fmri data analysis

vc theory of large margin multi category classifiers in the context of discriminant analysis vapnik s statistical learning theory has mainly been developed in three directions the computation of dichotomies with binary valued functions the computation of dichotomies with real valued functions and the computation of polytomies with functions taking their values in finite sets typically the set of categories itself the case of classes of vector valued functions used to compute polytomies has seldom been considered independently which is unsatisfactory for three main reasons first this case encompasses the other ones second it cannot be treated appropriately through a naive extension of the results devoted to the computation of dichotomies third most of the classification problems met in practice involve multiple categories in this paper a vc theory of large margin multi category classifiers is introduced central in this theory are generalized vc dimensions called the dimensions first a uniform convergence bound on the risk of the classifiers of interest is derived the capacity measure involved in this bound is a covering number this covering number can be upper bounded in terms of the dimensions thanks to generalizations of sauer s lemma as is illustrated in the specific case of the scale sensitive natarajan dimension a bound on this latter dimension is then computed for the class of functions on which multi class svms are based this makes it possible to apply the structural risk minimization inductive principle to those machines

regularized principal manifolds many settings of unsupervised learning can be viewed as quantization problems the minimization of the expected quantization error subject to some restrictions this allows the use of tools such as regularization from the theory of supervised risk minimization for unsupervised learning this setting turns out to be closely related to principal curves the generative topographic map and robust coding we explore this connection in two ways 1 we propose an algorithm for finding principal manifolds that can be regularized in a variety of ways and 2 we derive uniform convergence bounds and hence bounds on the learning rates of the algorithm in particular we give bounds on the covering numbers which allows us to obtain nearly optimal learning rates for certain types of regularization operators experimental results demonstrate the feasibility of the approach