a consistent method for the solution to reduced fpk equation in statistical mechanics abstract the solution of reduced fokker planck kolmogorov fpk equation resulted from the problems in statistical mechanics is formulated as an exponential function of polynomials in state variables special measure is taken to satisfy the fpk equation in the weak sense of integration with the assumed function examples are given to show the application of the method to the non linear systems with additive random excitations and those with both additive and multiplicative random excitations the pdf solutions obtained with the proposed method and conventional gaussian closure method are compared with the exact solutions in special cases numerical results showed that the solutions obtained with the proposed method are much close to the exact solutions even for highly non linear system and the system with both additive and multiplicative excitations the convergence of the approximate pdf solutions obtained with the method is also shown with numerical results

scaling concepts in periodically modulated noisy systems abstract we show that scaling arguments are very useful to analyze the dynamics of periodically modulated noisy systems information about the behavior of the relevant quantities such as the signal to noise ratio upon variations of the noise level can be obtained by analyzing the symmetries and invariances of the system in this way it is possible to predict diverse physical manifestations of the cooperative behavior between noise and input signal as for instance stochastic resonance spatiotemporal stochastic resonance and stochastic multiresonance

on the sampling distribution of resubstitution and leave one out error estimators for linear classifiers error estimation is a problem of high current interest in many areas of application this paper concerns the classical problem of determining the performance of error estimators in small sample settings under a gaussianity parametric assumption we provide here for the first time the exact sampling distribution of the resubstitution and leave one out error estimators for linear discriminant analysis lda in the univariate case which is valid for any sample size and combination of parameters including unequal variances and sample sizes for each class in the multivariate case we provide a quasi binomial approximation to the distribution of both the resubstitution and leave one out error estimators for lda under a common but otherwise arbitrary class covariance matrix which is assumed to be known in the design of the lda discriminant we provide numerical examples using both synthetic and real data that indicate that these approximations are accurate provided that lda classification error is not too large

the proper formula for relative entropy and its asymptotics in quantum probability umegaki s relative entropys trd logd logd of states and with density operatorsd andd respectively is shown to be an asymptotic exponent considered from the quantum hypothesis testing viewpoint it is also proved that some other versions of the relative entropy give rise to the same asymptotics as umegaki s one as a byproduct the inequality tra logab tra loga logb is obtained for positive definite matricesa andb

exact sample conditioned mse performance of the bayesian mmse estimator for classification error part i representation in recent years biomedicine has been faced with difficult high throughput small sample classification problems in such settings classifier error estimation becomes a critical issue because training and testing must be done on the same data a recently proposed error estimator places the problem in a signal estimation framework in the presence of uncertainty permitting a rigorous solution optimal in a minimum mean square error sense the uncertainty in this model is relative to the parameters of the feature label distributions resulting in a bayesian approach to error estimation closed form solutions are available for two important problems discrete classification with dirichlet priors and linear classification of gaussian distributions with normal inverse wishart priors in this work part i of a two part study we introduce the theoretical mean square error mse conditioned on the observed sample of any estimate of the classifier error including the bayesian error estimator for both bayesian models thus bayesian error estimation has a unique advantage in that its mathematical framework naturally gives rise to a practical expected measure of performance given an observed sample in part ii of the study we examine consistency of the error estimator demonstrate various mse properties and apply the conditional mse to censored sampling

a pruning method of refining recursive reduced least squares support vector regression in this paper a pruning method is proposed to refine the recursive reduced least squares support vector regression rrlssvr and its improved version irrlssvr and thus two novel algorithms prurrlssvr and pruirrlssvr are yielded this pruning method ranks support vectors by defining a contribution function to the objective function and then the support vector with the least contribution is pruned unless it is the most recently selected support vector consequently prurrlssvr and pruirrlssvr outperform rrlssvr and irrlssvr respectively in terms of the number of support vectors while not impairing the generalization performance in addition a speedup scheme is presented that reduces the computational burden of computing the contribution function to show the effectiveness and feasibility of the proposed prurrlssvr and pruirrlssvr experiments are performed on ten benchmark data sets and a gas furnace instance

bayesian minimum mean square error estimation for classification error part i definition and the bayesian mmse error estimator for discrete classification with the advent of high throughput genomic and proteomic technologies in conjunction with the difficulty in obtaining even moderately sized samples small sample classifier design has become a major issue in the biological and medical communities with small samples training data error estimation becomes mandatory yet none of the popular error estimation techniques have been rigorously designed based on statistical inference and optimization in this investigation we place classifier error estimation into the framework of optimal mean square error mse signal estimation in the presence of uncertainty which results in a bayesian approach to error estimation based on a parameterized family of feature label distributions with the prior distribution of the parameters governing the choice of feature label distribution these bayesian error estimators are optimal when averaged over a given family of distributions unbiased when averaged over a given family and all samples and analytically address a trade off between robustness modeling assumptions and accuracy minimum mean square error in this paper part i of a two part study we define the minimum mean square error mmse error estimator discuss its basic properties provide closed form analytic estimator representation for discrete classifiers with both non informative and informative prior distributions and examine the performance and robustness of the mmse error estimator via simulations in part ii of this paper in this same issue of ieee transactions on signal processing we address all of these issues in particular closed form representation for linear classification in the gaussian model with known and unknown covariance matrices for both the discrete and gaussian cases the mmse error estimator has especially good performance for distributions having moderate true errors

an instrumental least squares support vector machine for nonlinear system identification least squares support vector machines ls svms originating from statistical learning and reproducing kernel hilbert space rkhs theories represent a promising approach to identify nonlinear systems via nonparametric estimation of the involved nonlinearities in a computationally and stochastically attractive way however application of ls svms and other rkhs variants in the identification context is formulated as a regularized linear regression aiming at the minimization of the 2 loss of the prediction error this formulation corresponds to the assumption of an auto regressive noise structure which is often found to be too restrictive in practical applications in this paper instrumental variable iv based estimation is integrated into the ls svm approach providing under minor conditions consistent identification of nonlinear systems regarding the noise modeling error it is shown how the cost function of the ls svm is modified to achieve an iv based solution although a practically well applicable choice of the instrumental variable is proposed for the derived approach optimal choice of this instrument in terms of the estimates associated variance still remains to be an open problem the effectiveness of the proposed iv based ls svm scheme is also demonstrated by a monte carlo study based simulation example

heuristics of instability and stabilization in model selection in model selection usually a best predictor is chosen from a collection hat mu cdot s of predictors where hat mu cdot s is the minimum least squares predictor in a collection mathsf u s of predictors here s is a complexity parameter that is the smaller s the lower dimensional smoother the models in mathsf u s if mathsf l is the data used to derive the sequence hat mu cdot s the procedure is called unstable if a small change in mathsf l can cause large changes in hat mu cdot s with a crystal ball one could pick the predictor in hat mu cdot s having minimum prediction error without prescience one uses test sets cross validation and so forth the difference in prediction error between the crystal ball selection and the statistician s choice we call predictive loss for an unstable procedure the predictive loss is large this is shown by some analytics in a simple case and by simulation results in a more complex comparison of four different linear regression methods unstable procedures can be stabilized by perturbing the data getting a new predictor sequence hat mu cdot s and then averaging over many such predictor sequences

an inverse random source problem for the helmholtz equation this paper is concerned with an inverse random source problem for the one dimensional stochastic helmholtz equation which is to reconstruct the statistical properties of the random source function from boundary measurements of the radiating random electric field although the emphasis of the paper is on the inverse problem we adapt a computationally more efficient approach to study the solution of the direct problem in the context of the scattering model specifically the direct model problem is equivalently formulated into a two point spatially stochastic boundary value problem for which the existence and uniqueness of the pathwise solution is proved in particular an explicit formula is deduced for the solution from an integral representation by solving the two point boundary value problem based on this formula a novel and efficient strategy which is entirely done by using the fast fourier transform is proposed to reconstruct the mean and the variance of the random source function from measurements at one boundary point where the measurements are assumed to be available for many realizations of the source term numerical examples are presented to demonstrate the validity and effectiveness of the proposed method

almost sure asymptotic stability of drift implicit θ methods for bilinear ordinary stochastic differential equations in r 1 global almost sure asymptotic stability of stochastic methods with nonrandom variable step sizes when applied to bilinear nonautonomous homogeneous test systems of ordinary stochastic differential equations sdes is investigated sufficient conditions for almost sure asymptotic stability are proved for both analytical and numerical solutions in r1 the results of saito and mitsui world sci ser appl math 2 1993 333 siam j numer anal 33 1996 2254 higham siam j numer anal 38 2001 753 and schurz stochastic anal appl 14 1996 313 handbook of stochastic analysis and applications 2002 for the constant step sizes are carried over to the case with variable step sizes and nonautonomous linear test equations the investigations indicate that methods with variable step sizes or variable parameter governed by certain conditions can successfully be used to guarantee almost sure asymptotic stability while discretizing nonautonomous sdes

general formulas for capacity of classical quantum channels the capacity of a classical quantum channel or in other words the classical capacity of a quantum channel is considered in the most general setting where no structural assumptions such as the stationary memoryless property are made on a channel a capacity formula as well as a characterization of the strong converse property is given just in parallel with the corresponding classical results of verdu han 1994 which are based on the so called information spectrum method the general results are applied to the stationary memoryless case with or without cost constraint on inputs whereby a deep relation between the channel coding theory and the hypothesis testing for two quantum states is elucidated

a compression approach to support vector model selection in this paper we investigate connections between statistical learning theory and data compression on the basis of support vector machine svm model selection inspired by several generalization bounds we construct compression coefficients for svms which measure the amount by which the training labels can be compressed by a code built from the separating hyperplane the main idea is to relate the coding precision to geometrical concepts such as the width of the margin or the shape of the data in the feature space the so derived compression coefficients combine well known quantities such as the radius margin term r2 2 the eigenvalues of the kernel matrix and the number of support vectors to test whether they are useful in practice we ran model selection experiments on benchmark data sets as a result we found that compression coefficients can fairly accurately predict the parameters for which the test error is minimized

marcus versus stratonovich for systems with jump noise the famous ito stratonovich dilemma arises when one examines a dynamical system with a multiplicative white noise in physics literature this dilemma is often resolved in favour of the stratonovich prescription because of its two characteristic properties valid for systems driven by brownian motion i it allows physicists to treat stochastic integrals in the same way as conventional integrals and ii it appears naturally as a result of a small correlation time limit procedure on the other hand the marcus prescription ieee trans inform theory 24 164 1978 stochastics 4 223 1981 should be used to retain i and ii for systems driven by a poisson process levy flights or more general jump processes in present communication we present an in depth comparison of the ito stratonovich and marcus equations for systems with multiplicative jump noise by the examples of a real valued linear system and a complex oscillator with noisy frequency the kubo anderson oscillator we compare solutions obtained with the three prescriptions

analysis of variance of cross validation estimators of the generalization error this paper brings together methods from two different disciplines statistics and machine learning we address the problem of estimating the variance of cross validation cv estimators of the generalization error in particular we approach the problem of variance estimation of the cv estimators of generalization error as a problem in approximating the moments of a statistic the approximation illustrates the role of training and test sets in the performance of the algorithm it provides a unifying approach to evaluation of various methods used in obtaining training and test sets and it takes into account the variability due to different training and test sets for the simple problem of predicting the sample mean and in the case of smooth loss functions we show that the variance of the cv estimator of the generalization error is a function of the moments of the random variables y card sj sj and y card sjc sj c where sj sj are two training sets and sjc sj c are the corresponding test sets we prove that the distribution of y and y is hypergeometric and we compare our estimator with the one proposed by nadeau and bengio 2003 we extend these results in the regression case and the case of absolute error loss and indicate how the methods can be extended to the classification case we illustrate the results through simulation

strong converse for the classical capacity of optical quantum communication channels we establish the classical capacity of optical quantum channels as a sharp transition between two regimes one which is an error free regime for communication rates below the capacity and the other in which the probability of correctly decoding a classical message converges exponentially fast to zero if the communication rate exceeds the classical capacity this result is obtained by proving a strong converse theorem for the classical capacity of all phase insensitive bosonic gaussian channels a well established model of optical quantum communication channels such as lossy optical fibers amplifier and free space communication the theorem holds under a particular photon number occupation constraint which we describe in detail in this paper our result bolsters the understanding of the classical capacity of these channels and opens the path to applications such as proving the security of noisy quantum storage models of cryptography with optical links

generic emergence of objectivity of observables in infinite dimensions quantum darwinism posits that information becomes objective whenever multiple observers indirectly probe a quantum system by each measuring a fraction of the environment it was recently shown that objectivity of observables emerges generically from the mathematical structure of quantum mechanics whenever the system of interest has finite dimensions and the number of environment fragments is large f g s l brandao m piani and p horodecki nat commun 6 7908 2015 ncaobw2041 172310 1038 ncomms8908 despite the importance of this result it necessarily excludes many practical systems of interest that are infinite dimensional including harmonic oscillators extending the study of quantum darwinism to infinite dimensions is a nontrivial task we tackle it here by using a modified diamond norm suitable to quantify the distinguishability of channels in infinite dimensions we prove two theorems that bound the emergence of objectivity first for finite mean energy systems and then for systems that can only be prepared in states with an exponential energy cutoff we show that the latter class includes any bounded energy subset of single mode gaussian states

combined 5 2 cv f test for comparing supervised classification learning algorithms dietterich 1998 reviews five statistical tests and proposes the 5 2 cv t test for determining whether there is a significant difference between the error rates of two classifiers in our experiments we noticed that the 5 2 cv t test result may vary depending on factors that should not affect the test and we propose a variant the combined 5 2 cv f test that combines multiple statistics to get a more robust test simulation results show that this combined version of the test has lower type i error and higher power than 5 2 cv proper

markov cross validation for time series model evaluations cross validation cv is a simple and universal tool to estimate generalization ability however existing cvs do not work well for periodicity overlapping or correlation of series the corresponding three criteria aimed at describing these properties are presented based on them we put forward a novel markov cross validation m cv whose data partition can be seen as a markov process the partition ensures that samples in each subset are neither too close nor too far in doing so overfitting model or information loss of series which may result in underestimation or overestimation of the error can be avoided furthermore subsets from m cv partition could well represent the original series and it may be extended to time series or stream data sampling theoretical analysis shows that m cv is the unique one which meets all of above criteria among current cvs in addition the error estimation on subsets is proved to have less variance than that on original series therefore it ensures the stability of m cv experimental results demonstrate that the proposed m cv has lower bias variance and time consumption than other cvs

analytical study of performance of linear discriminant analysis in stochastic settings this paper provides exact analytical expressions for the first and second moments of the true error for linear discriminant analysis lda when the data are univariate and taken from two stochastic gaussian processes the key point is that we assume a general setting in which the sample data from each class do not need to be identically distributed or independent within or between classes we compare the true errors of designed classifiers under the typical i i d model and when the data are correlated providing exact expressions and demonstrating that depending on the covariance structure correlated data can result in classifiers with either greater error or less error than when training with uncorrelated data the general theory is applied to autoregressive and moving average models of the first order and it is demonstrated using real genomic data

optimal mean square error calibration of classifier error estimators under bayesian models a recently proposed bayesian modeling framework for classification facilitates both the analysis and optimization of error estimation performance the bayesian error estimator is then defined to have optimal mean square error performance but in many situations closed form representations are unavailable and approximations may not be feasible to address this we present a method to optimally calibrate arbitrary error estimators for minimum mean square error performance within a supposed bayesian framework assuming a fixed sample size classification rule and error estimation rule as well as a fixed bayesian model the calibration is done by first computing a calibration function that maps error estimates to their optimally calibrated values off line once found this calibration function may be easily applied to error estimates on the fly whenever the assumptions apply we demonstrate that calibrated error estimators offer significant improvement in performance relative to classical error estimators under bayesian models with both linear and non linear classification rules

generalized relative entropies and the capacity of classical quantum channels we provide lower and upper bounds on the information transmission capacity of one single use of a classical quantum channel the lower bound is expressed in terms of the hoeffding capacity which we define similarly to the holevo capacity but replacing the relative entropy with the hoeffding distance similarly our upper bound is in terms of a quantity obtained by replacing the relative entropy with the recently introduced max relative entropy in the definition of the divergence radius of a channel

strong converse exponents for a quantum channel discrimination problem and quantum feedback assisted communication this paper studies the difficulty of discriminating between an arbitrary quantum channel and a replacer channel that discards its input and replaces it with a fixed state the results obtained here generalize those known in the theory of quantum hypothesis testing for binary state discrimination we show that in this particular setting the most general adaptive discrimination strategies provide no asymptotic advantage over non adaptive tensor power strategies this conclusion follows by proving a quantum stein s lemma for this channel discrimination setting showing that a constant bound on the type i error leads to the type ii error decreasing to zero exponentially quickly at a rate determined by the maximum relative entropy registered between the channels the strong converse part of the lemma states that any attempt to make the type ii error decay to zero at a rate faster than the channel relative entropy implies that the type i error necessarily converges to one we then refine this latter result by identifying the optimal strong converse exponent for this task as a consequence of these results we can establish a strong converse theorem for the quantum feedback assisted capacity of a channel sharpening a result due to bowen furthermore our channel discrimination result demonstrates the asymptotic optimality of a non adaptive tensor power strategy in the setting of quantum illumination as was used in prior work on the topic the sandwiched renyi relative entropy is a key tool in our analysis finally by combining our results with recent results of hayashi and tomamichel we find a novel operational interpretation of the mutual information of a quantum channel mathcal n as the optimal type ii error exponent when discriminating between a large number of independent instances of mathcal n and an arbitrary worst case replacer channel chosen from the set of all replacer channels

granularity selection for cross validation of svm abstract granularity selection is fundamental to granular computing cross validation cv is widely adopted for model selection where each fold of data set of cv can be considered as an information granule and the larger the number of the folds is the smaller the granularity of each fold is therefore for cv granularity selection is equal to the selection of the number of folds in this paper we explore the granularity selection for cv of support vector machine svm we first use the huber loss to smooth the hinge loss used in svm and to approximate cv of svm then we derive a tight upper bound of the discrepancy between the original and the approximate cv with a high convergence rate finally based on this derived tight bound we present a granularity selection criterion for trading off the accuracy and time cost experimental results demonstrate that the approximate cv with the granularity selection criterion gives the similar accuracies as the traditional cv and meanwhile significantly improves the efficiency

making good codes for classical quantum channel coding via quantum hypothesis testing in this correspondence we give an alternative proof of the direct part of the classical quantum channel coding theorem the holevo schumacher westmoreland hsw theorem using ideas of quantum hypothesis testing in order to show the existence of good codes we invoke a limit theorem relevant to the quantum stein s lemma in quantum hypothesis testing as the law of large numbers used in the classical case we also apply a greedy construction of good codes using a packing procedure of noncommutative operators consequently we derive an upper bound on the coding error probability which is used to give an alternative proof of the hsw theorem this approach elucidates how the holevo information applies to the classical quantum channel coding problems

optimal sequence of quantum measurements in the sense of stein s lemma in quantum hypothesis testing we derive a necessary and sufficient condition for a sequence of quantum measurements to achieve the optimal performance in quantum hypothesis testing we discuss what quantum measurement we should perform in order to attain the optimal exponent of the second error probability under the condition that the first error probability goes to 0 as an asymptotically optimal measurement we propose a projection measurement characterized by the irreducible representation theory of the special linear group sl especially in the spin 1 2 system it is realized by the simultaneous measurement of the total momentum and a momentum of a specified direction as a by product we obtain another proof of quantum stein s lemma in addition an asymptotically optimal measurement is constructed in the quantum gaussian case and it is physically meaningful

leave one out procedures for nonparametric error estimates the use of nonparametric error estimates may lead to biased results if the kernel covariances are estimated from the same data as are used to form the error estimate if additional design samples are available one may eliminate this bias by estimating the class covariances using an independent set of data if however additional samples are not available one may resort to leave one out type estimates of the kernel for parzen estimates or metric for nearest neighbor estimates for every sample being tested the authors present an efficient algorithm for computation of these leave one out type estimates that requires little additional computational burden over procedures currently in use the presentation is applicable to both parzen and k nearest neighbor k nn type estimates experimental results demonstrating the efficiency of the algorithm are provided

a cross validatory method for dependent data summary in this paper we extend the technique of cross validation to the case where observations form a general stationary sequence we call it h block cross validation because the idea is to reduce the training set by removing the h observations preceding and following the observation in the test set we propose taking h to be a fixed fraction of the sample size and we add a term to our h block cross validated estimate to compensate for the underuse of the sample the advantages of the proposed modification over the cross validation technique are demonstrated via simulation

strong converse and stein s lemma in quantum hypothesis testing the hypothesis testing problem for two quantum states is treated we show a new inequality between the errors of the first kind and the second kind which complements the result of hiai and petz 1991 to establish the quantum version of stein s lemma the inequality is also used to show a bound on the probability of errors of the first kind when the power exponent for the probability of errors of the second kind exceeds the quantum relative entropy which yields the strong converse in quantum hypothesis testing finally we discuss the relation between the bound and the power exponent derived by han and kobayashi 1989 in classical hypothesis testing

a general framework for the statistical analysis of the sources of variance for classification error estimators estimating the prediction error of classifiers induced by supervised learning algorithms is important not only to predict its future error but also to choose a classifier from a given set model selection if the goal is to estimate the prediction error of a particular classifier the desired estimator should have low bias and low variance however if the goal is the model selection in order to make fair comparisons the chosen estimator should have low variance assuming that the bias term is independent from the considered classifier this paper follows the analysis proposed in 1 about the statistical properties of k fold cross validation estimators and extends it to the most popular error estimators resubstitution holdout repeated holdout simple bootstrap and 0 632 bootstrap estimators without and with stratification we present a general framework to analyze the decomposition of the variance of different error estimators considering the nature of the variance irreducible reducible variance and the different sources of sensitivity internal external sensitivity an extensive empirical study has been performed for the previously mentioned estimators with naive bayes and c4 5 classifiers over training sets obtained from assorted probability distributions the empirical analysis consists of decomposing the variances following the proposed framework and checking the independence assumption between the bias and the considered classifier based on the obtained results we propose the most appropriate error estimations for model selection under different experimental conditions

universal coding for classical quantum channel we construct a universal code for a stationary and memoryless classical quantum channel as a quantum version of the universal coding by csiszar and korner our code is constructed utilizing a combination of irreducible representations a decoder introduced through the quantum information spectrum and the packing lemma

classical communication over a quantum interference channel calculating the capacity of interference channels is a notorious open problem in classical information theory such channels have two senders and two receivers and each sender would like to communicate with a partner receiver the capacity of such channels is known exactly in the settings of very strong and strong interference while the han kobayashi coding strategy gives the best known achievable rate region in the general case here we introduce and study the quantum interference channel a natural generalization of the interference channel to the setting of quantum information theory we restrict ourselves for the most part to channels with two classical inputs and two quantum outputs in order to simplify the presentation of our results though generalizations of our results to channels with quantum inputs are straightforward we are able to determine the exact classical capacity of this channel in the settings of very strong and strong interference by exploiting winter s successive decoding strategy and a novel two sender quantum simultaneous decoder respectively we provide a proof that a han kobayashi strategy is achievable with holevo information rates up to a conjecture regarding the existence of a three sender quantum simultaneous decoder this conjecture holds for a special class of quantum multiple access channels with average output states that commute and we discuss some other variations of the conjecture that hold finally we detail a connection between the quantum interference channel and prior work on the capacity of bipartite unitary gates

the estimation of prediction error having constructed a data based estimation rule perhaps a logistic regression or a classification tree the statistician would like to know its performance as a predictor of future cases there are two main theories concerning prediction error 1 penalty methods such as cp akaike s information criterion and stein s unbiased risk estimate that depend on the covariance between data points and their corresponding predictions and 2 cross validation and related nonparametric bootstrap techniques this article concerns the connection between the two theories a rao blackwell type of relation is derived in which nonparametric methods such as cross validation are seen to be randomized versions of their covariance penalty counterparts the model based penalty methods offer substantially better accuracy assuming that the model is believable

bayesian minimum mean square error estimation for classification error part ii linear classification of gaussian models in this paper part ii of a two part study we derive a closed form analytic representation of the bayesian minimum mean square error mmse error estimator for linear classification assuming gaussian models this is presented in a general framework permitting a structure on the covariance matrices and a very flexible class of prior parameter distributions with four free parameters closed form solutions are provided for known scaled identity and arbitrary covariance matrices we examine performance in small sample settings via simulations on both synthetic and real genomic data and demonstrate the robustness of these error estimators to false gaussian modeling assumptions by applying them to johnson distributions

comment on numerical methods for stochastic differential equations wilkie phys rev e 70 017701 2004 used a heuristic approach to derive runge kutta based numerical methods for stochastic differential equations based on methods used for solving ordinary differential equations the aim was to follow solution paths with high order we point out that this approach is invalid in the general case and does not lead to high order methods we warn readers against the inappropriate use of deterministic calculus in a stochastic setting

classifier performance as a function of distributional complexity abstract when choosing a classification rule it is important to take into account the amount of sample data available this paper examines the performances of classifiers of differing complexities in relation to the complexity of feature label distributions in the case of small samples we define the distributional complexity of a feature label distribution to be the minimal number of hyperplanes necessary to achieve the bayes classifier if the bayes classifier is achievable by a finite number of hyperplanes and infinity otherwise our approach is to choose a model and compare classifier efficiencies for various sample sizes and distributional complexities simulation results are obtained by generating data based on the model and the distributional complexities a linear support vector machine svm is considered along with several nonlinear classifiers for the most part we see that there is little improvement when one uses a complex classifier instead of a linear svm for higher levels of distributional complexity the linear classifier degrades but so do the more complex classifiers owing to insufficient training data hence if one were to obtain a good result with a more complex classifier it is most likely that the distributional complexity is low and there is no gain over using a linear classifier hence under the model it is generally impossible to claim that use of the nonlinear classifier is beneficial in essence the sample sizes are too small to take advantage of the added complexity an exception to this observation is the behavior of the three nearest neighbor 3nn classifier in the case of two variables but not three when there is very little overlap between the label distributions and the sample size is not too small with a sample size of 60 the 3nn classifier performs close to the bayes classifier even for high levels of distributional complexity consequently if one uses the 3nn classifier with two variables and obtains a low error then the distributional complexity might be large and if such is the case there is a significant gain over using a linear classifier

comparative analysis of multiscale gaussian random field simulation algorithms abstract we analyze and compare the efficiency and accuracy of two simulation methods for homogeneous random fields with multiscale resolution we consider in particular the fourier wavelet method and three variants of the randomization method a without any stratified sampling of wavenumber space b with stratified sampling of wavenumbers with equal energy subdivision c with stratified sampling with a logarithmically uniform subdivision we focus primarily on fractal gaussian random fields with kolmogorov type spectra previous work has shown that variants a and b of the randomization method are only able to generate a self similar structure function over three to four decades with reasonable computational effort by contrast variant c along with the fourier wavelet method is able to reproduce accurate self similar scaling of the structure function over a number of decades increasing linearly with computational effort for our examples we will show that nine decades can be reproduced we provide some conceptual and numerical comparison of the various cost contributions to each random field simulation method we find that when evaluating ensemble averaged quantities like the correlation and structure functions as well as some multi point statistical characteristics the randomization method can provide good accuracy with less cost than the fourier wavelet method the cost of the randomization method relative to the fourier wavelet method however appears to increase with the complexity of the random field statistics which are to be calculated accurately moreover the fourier wavelet method has better ergodic properties and hence becomes more efficient for the computation of spatial rather than ensemble averages which may be important in simulating the solutions to partial differential equations with random field coefficients

the capacity of the quantum channel with general signal states it is shown that the capacity of a classical quantum channel with arbitrary possibly mixed states equals the maximum of the entropy bound with respect to all a priori distributions this completes the recent result of hausladen jozsa schumacher westmoreland and wootters 1996 who proved the equality for the pure state channel

exact sample conditioned mse performance of the bayesian mmse estimator for classification error part ii consistency and performance analysis in part i of a two part study on the mse performance of bayesian error estimation we have derived analytical expressions for mse conditioned on the sample for bayesian error estimators and arbitrary error estimators in two bayesian models discrete classification with dirichlet priors and linear classification of gaussian distributions with normal inverse wishart priors here in part ii we examine the consistency of bayesian error estimation and provide several simulation studies that illustrate the concept of conditional mse and how it may be used in practice a salient application is censored sampling where sample points are collected one at a time until the conditional mse reaches a stopping criterion

bounds on error expectation for support vector machines we introduce the concept of span of support vectors sv and show that the generalization ability of support vector machines svm depends on this new geometrical concept we prove that the value of the span is always smaller and can be much smaller than the diameter of the smallest sphere containing the support vectors used in previous bounds vapnik 1998 we also demonstate experimentally that the prediction of the test error given by the span is very accurate and has direct application in model selection choice of the optimal parameters of the svm

parametric methods for comparing the performance of two classification algorithms evaluated by k fold cross validation on multiple data sets a popular procedure for identifying which one of two classification algorithms has a better performance is to test them on multiple data sets and the accuracies resulting from k fold cross validation are aggregated to draw a conclusion several nonparametric methods have been proposed for this purpose while parametric methods will be a better choice to determine the superior algorithm when the assumptions for deriving sampling distributions can be satisfied in this paper we consider every accuracy estimate resulting from the instances in a fold or a data set as a point estimator instead of a fixed value to derive the sampling distribution of the point estimator for comparing the performance of two classification algorithms the test statistics for both data set and fold averaging levels are proposed and the ways to calculate their degrees of freedom are also presented twelve data sets are chosen to demonstrate that our parametric methods can be used to effectively compare the performance of two classification algorithms on multiple data sets several critical issues in using our parametric methods and the nonparametric ones proposed in a previous study are then discussed parametric methods for performance comparison of two classification algorithms are proposed both independent and matched samples are considered the experimental results on twelve data sets demonstrate the effectiveness of our parametric methods

a smooth entropy approach to quantum hypothesis testing and the classical capacity of quantum channels we use the smooth entropy approach to treat the problems of binary quantum hypothesis testing and the transmission of classical information through a quantum channel we provide lower and upper bounds on the optimal type ii error of quantum hypothesis testing in terms of the smooth max relative entropy of the two states representing the two hypotheses then using a relative entropy version of the quantum asymptotic equipartition property qaep we can recover the strong converse rate of the i i d hypothesis testing problem in the asymptotics on the other hand combining stein s lemma with our bounds we obtain a stronger e independent version of the relative entropy qaep similarly we provide bounds on the one shot e error classical capacity of a quantum channel in terms of a smooth max relative entropy variant of its holevo capacity using these bounds and the e independent version of the relative entropy qaep we can recover both the holevo schumacher westmoreland theorem about the optimal direct rate of a memoryless quantum channel with product state encoding as well as its strong converse counterpart

weak decoupling duality and quantum identification if a quantum system is subject to noise it is possible to perform quantum error correction reversing the action of the noise if and only if no information about the system s quantum state leaks to the environment in this paper we develop an analogous duality in the case that the environment approximately forgets the identity of the quantum state a weaker condition satisfied by randomizing maps and approximate unitary designs specifically we show that the environment approximately forgets quantum states if and only if the original channel approximately preserves pairwise fidelities of pure inputs an observation we call weak decoupling duality using this tool we then go on to study the task of using the output of a channel to simulate restricted classes of measurements on a space of input states the case of simulating measurements that test whether the input state is an arbitrary pure state is known as equality testing or quantum identification an immediate consequence of weak decoupling duality is that the ability to perform quantum identification cannot be cloned we furthermore establish that the optimal amortized rate at which quantum states can be identified through a noisy quantum channel is equal to the entanglement assisted classical capacity of the channel despite the fact that the task is quantum not classical and entanglement assistance is not allowed in particular this rate is strictly positive for every nonconstant quantum channel including classical channels

fast cross validation algorithms for least squares support vector machine and kernel ridge regression given n training examples the training of a least squares support vector machine ls svm or kernel ridge regression krr corresponds to solving a linear system of dimension n in cross validating ls svm or krr the training examples are split into two distinct subsets for a number of times l wherein a subset of m examples are used for validation and the other subset of n m examples are used for training the classifier in this case l linear systems of dimension n m need to be solved we propose a novel method for cross validation cv of ls svm or krr in which instead of solving l linear systems of dimension n m we compute the inverse of an n dimensional square matrix and solve l linear systems of dimension m thereby reducing the complexity when l is large and or m is small typical multi fold leave one out cross validation loo cv and leave many out cross validations are considered for five fold cv used in practice with five repetitions over randomly drawn slices the proposed algorithm is approximately four times as efficient as the naive implementation for large data sets we propose to evaluate the cv approximately by applying the well known incomplete cholesky decomposition technique and the complexity of these approximate algorithms will scale linearly on the data size if the rank of the associated kernel matrix is much smaller than n simulations are provided to demonstrate the performance of ls svm and the efficiency of the proposed algorithm with comparisons to the naive and some existent implementations of multi fold and loo cv

one shot lossy quantum data compression we provide a framework for one shot quantum rate distortion coding in which the goal is to determine the minimum number of qubits required to compress quantum information as a function of the probability that the distortion incurred upon decompression exceeds some specified level we obtain a one shot characterization of the minimum qubit compression size for an entanglement assisted quantum rate distortion code in terms of the smooth max information a quantity previously employed in the one shot quantum reverse shannon theorem next we show how this characterization converges to the known expression for the entanglement assisted quantum rate distortion function for asymptotically many copies of a memoryless quantum information source finally we give a tight finite blocklength characterization for the entanglement assisted minimum qubit compression size of a memoryless isotropic qubit source subject to an average symbolwise distortion constraint

recursive reduced least squares support vector regression combining reduced technique with iterative strategy we propose a recursive reduced least squares support vector regression the proposed algorithm chooses the data which make more contribution to target function as support vectors and it considers all the constraints generated by the whole training set thus it acquires less support vectors the number of which can be arbitrarily predefined to construct the model with the similar generalization performance in comparison with other methods our algorithm also gains excellent parsimoniousness numerical experiments on benchmark data sets confirm the validity and feasibility of the presented algorithm in addition this algorithm can be extended to classification

limiting subhessians limiting subjets and their calculus we study calculus rules for limiting subjets of order two these subjets are obtained as limits of sequences of subjets a subjet of a function f at some point x being the taylor expansion of a twice differentiable function which minorizes f and coincides with f at x these calculus rules are deduced from approximate or fuzzy calculus rules for subjets of order two in turn these rules are consequences of delicate results of crandall ishii lions we point out the similarities and the differences with the case of first order limiting subdifferentials

a comparative study of ordinary cross validation v fold cross validation and the repeated learning testing methods summary concepts of v fold cross validation and repeated learning testing methods have been introduced here in many problems these methods are computationally much less expensive than ordinary cross validation and can be used in its place a comparative study of these three methods has been carried out in detail

kernel independent component analysis we present a class of algorithms for independent component analysis ica which use contrast functions based on canonical correlations in a reproducing kernel hilbert space on the one hand we show that our contrast functions are related to mutual information and have desirable mathematical properties as measures of statistical dependence on the other hand building on recent developments in kernel methods we show that these criteria and their derivatives can be computed efficiently minimizing these criteria leads to flexible and robust algorithms for ica we illustrate with simulations involving a wide variety of source distributions showing that our algorithms outperform many of the presently known algorithms

model selection for linear classifiers using bayesian error estimation regularized linear models are important classification methods for high dimensional problems where regularized linear classifiers are often preferred due to their ability to avoid overfitting the degree of freedom of the model dis determined by a regularization parameter which is typically selected using counting based approaches such as k fold cross validation for large data this can be very time consuming and for small sample sizes the accuracy of the model selection is limited by the large variance of cv error estimates in this paper we study the applicability of a recently proposed bayesian error estimator for the selection of the best model along the regularization path we also propose an extension of the estimator that allows model selection in multiclass cases and study its efficiency with l1 regularized logistic regression and l2 regularized linear support vector machine the model selection by the new bayesian error estimator is experimentally shown to improve the classification accuracy especially in small sample size situations and is able to avoid the excess variability inherent to traditional cross validation approaches moreover the method has significantly smaller computational complexity than cross validation highlightswe propose to use the bayesian error estimator bee for classifier model selection we show that the bee rule speeds up model selection by an order of magnitude we show that the bee rule selects a better model than cross validation we propose an approximation rule of the bee for multi label classification problems

moments and root mean square error of the bayesian mmse estimator of classification error in the gaussian model the most important aspect of any classifier is its error rate because this quantifies its predictive capacity thus the accuracy of error estimation is critical error estimation is problematic in small sample classifier design because the error must be estimated using the same data from which the classifier has been designed use of prior knowledge in the form of a prior distribution on an uncertainty class of feature label distributions to which the true but unknown feature distribution belongs can facilitate accurate error estimation in the mean square sense in circumstances where accurate completely model free error estimation is impossible this paper provides analytic asymptotically exact finite sample approximations for various performance metrics of the resulting bayesian minimum mean square error mmse error estimator in the case of linear discriminant analysis lda in the multivariate gaussian model these performance metrics include the first second and cross moments of the bayesian mmse error estimator with the true error of lda and therefore the root mean square rms error of the estimator we lay down the theoretical groundwork for kolmogorov double asymptotics in a bayesian setting which enables us to derive asymptotic expressions of the desired performance metrics from these we produce analytic finite sample approximations and demonstrate their accuracy via numerical examples various examples illustrate the behavior of these approximations and their use in determining the necessary sample size to achieve a desired rms the supplementary material contains derivations for some equations and added figures highlightsthe bayesian minimum mean square error bmmse error estimator of lda is studied the first second and cross moments of the bmmse estimator with true error are considered both conditional and unconditional performance metrics are analyzed for the first time the kolmogorov double asymptotic is used in a bayesian setting asymptotically exact finite sample approximations of performance metrics are derived

block regularized m 2 cross validated estimator of the generalization error a cross validation method based on replications of two fold cross validation is called an cross validation an cross validation is used in estimating the generalization error and comparing of algorithms performance in machine learning however the variance of the estimator of the generalization error in cross validation is easily affected by random partitions poor data partitioning may cause a large fluctuation in the number of overlapping samples between any two training test sets in cross validation this fluctuation results in a large variance in the cross validated estimator the influence of the random partitions on variance becomes serious as increases thus in this study the partitions with a restricted number of overlapping samples between any two training test sets are defined as a block regularized partition set the corresponding cross validation is called block regularized cross validation bcv it can effectively reduce the influence of random partitions we prove that the variance of the bcv estimator of the generalization error is smaller than the variance of cross validated estimator and reaches the minimum in a special situation an analytical expression of the variance can also be derived in this special situation this conclusion is validated through simulation experiments furthermore a practical construction method of bcv by a two level orthogonal array is provided finally a conservative estimator is proposed for the variance of estimator of the generalization error

the reliability of estimated confidence intervals for classification error rates when only a single sample is available error estimation accuracy is the salient issue regarding the validity of a classifier model when samples are small training data based error estimates tend to suffer from inaccuracy and quantification of error estimation accuracy is difficult numerous methods have been proposed for estimating confidence intervals for the true error based on the estimated error this paper surveys proposed methods and quantifies their performance monte carlo methods are used to obtain accurate estimates of the true confidence intervals and compare these to the intervals estimated from samples we consider different error estimators and several proposed confidence bound estimators both synthetic and real genomic data are employed our simulations show the majority of the confidence intervals methods have poor performance because of the difference of shape between true and estimated intervals according to our results the best estimation strategy is to use the 10 time 10 fold cross validation with a confidence interval based on the standard deviation

efficient leave one out cross validation of kernel fisher discriminant classifiers abstract mika et al in neural network for signal processing vol ix ieee press new york 1999 pp 41 48 apply the kernel trick to obtain a non linear variant of fisher s linear discriminant analysis method demonstrating state of the art performance on a range of benchmark data sets we show that leave one out cross validation of kernel fisher discriminant classifiers can be implemented with a computational complexity of only o l 3 operations rather than the o l 4 of a naive implementation where l is the number of training patterns leave one out cross validation then becomes an attractive means of model selection in large scale applications of kernel fisher discriminant analysis being significantly faster than conventional k fold cross validation procedures commonly used

estimating the error rate of a prediction rule improvement on cross validation abstract we construct a prediction rule on the basis of some data and then wish to estimate the error rate of this rule in classifying future observations cross validation provides a nearly unbiased estimate using only the original data cross validation turns out to be related closely to the bootstrap estimate of the error rate this article has two purposes to understand better the theoretical basis of the prediction problem and to investigate some related estimators which seem to offer considerably improved estimation in small samples

coding theorem and strong converse for quantum channels we present a new proof of holevo s 1973 1977 coding theorem for transmitting classical information through quantum channels and its strong converse the technique is largely inspired by wolfwitz s 1964 combinatorial approach using types of sequences as a byproduct of our approach which is independent of previous ones both in the coding theorem and the converse we can give a new proof of holevo s information bound

fast cross validation for kernel based algorithms cross validation cv is a widely adopted approach for selecting the optimal model however the computation of empirical cross validation error cve has high complexity due to multiple times of learner training in this paper we develop a novel approximation theory of cve and present an approximate approach to cv based on the bouligand influence function bif for kernel based algorithms we first represent the bif and higher order bifs in taylor expansions and approximate cv via the taylor expansions we then derive an upper bound of the discrepancy between the original and approximate cv furthermore we provide a novel computing method to calculate the bif for general distribution and evaluate bif criterion for sample distribution to approximate cv the proposed approximate cv requires training on the full data set only once and is suitable for a wide variety of kernel based algorithms experimental results demonstrate that the proposed approximate cv is sound and effective

no unbiased estimator of the variance of k fold cross validation most machine learning researchers perform quantitative experiments to estimate generalization error and compare the performance of different algorithms in particular their proposed algorithm in order to be able to draw statistically convincing conclusions it is important to estimate the uncertainty of such estimates this paper studies the very commonly used k fold cross validation estimator of generalization performance the main theorem shows that there exists no universal valid under all distributions unbiased estimator of the variance of k fold cross validation the analysis that accompanies this result is based on the eigen decomposition of the covariance matrix of errors which has only three different eigenvalues corresponding to three degrees of freedom of the matrix and three components of the total variance this analysis helps to better understand the nature of the problem and how it can make naive estimators that don t take into account the error correlations due to the overlap between training and test sets grossly underestimate variance this is confirmed by numerical experiments in which the three components of the variance are compared when the difficulty of the learning problem and the number of folds are varied

error estimation based on variance analysis of k fold cross validation when the numbers of samples and folds are both large enough we proved that cv variance and its accuracy have the quadratic relationship the relationships between cv variance and its factors have been derived allowing to predict which variance is less before applying k fold cv theoretical explanations have been given for some empirical evidences of rodriguez and kohavi from the respect of variance analysis the proposed normalized variance has significant correlation with the error and is unrelated to k so that it can serve as a stable error measurement cross validation cv is often used to estimate the generalization capability of a learning model the variance of cv error has a considerable impact on the accuracy of cv estimator and the adequacy of the learning model so it is very important to analyze cv variance the aim of this paper is to investigate how to improve the accuracy of the error estimation based on variance analysis we first describe the quantitative relationship between cv variance and its accuracy which can provide guidance for improving the accuracy by reducing the variance we then study the relationships between variance and relevant variables including the sample size the number of folds and the number of repetitions these form the basis of theoretical strategies of regulating cv variance our classification results can theoretically explain the empirical results of rodrguez and kohavi finally we propose a uniform normalized variance which not only measures model accuracy but also is irrelative to fold number therefore it simplifies the selection of fold number in k fold cv and normalized variance can serve as a stable error measurement for model comparison and selection we report the results of experiments using 5 supervised learning models and 20 datasets the results indicate that it is reliable to determine which variance is less before k fold cv by the proposed theorems and thus the accuracy of error estimation can be promoted by reducing variance in so doing we are more likely to select the best parameter or model

algorithmic stability and sanity check bounds for leave one out cross validation in this article we prove sanity check bounds for the error of the leave one out cross validation estimate of the generalization error that is bounds showing that the worst case error of this estimate is not much worse than that of the training error estimate the name sanity check refers to the fact that although we often expect the leave one out estimate to perform considerably better than the training error estimate we are here only seeking assurance that its performance will not be considerably worse perhaps surprisingly such assurance has been given only for limited cases in the prior literature on cross validation any nontrivial bound on the error of leave one out must rely on some notion of algorithmic stability previous bounds relied on the rather strong notion of hypothesis stability whose application was primarily limited to nearest neighbor and other local algorithms here we introduce the new and weaker notion of error stability and apply it to obtain sanity check bounds for leave one out for other classes of learning algorithms including training error minimization procedures and bayesian algorithms we also provide lower bounds demonstrating the necessity of some form of error stability for proving bounds on the error of the leave one out estimate and the fact that for training error minimization algorithms in the worst case such bounds must still depend on the vapnik chervonenkis dimension of the hypothesis class

a general formula for channel capacity a formula for the capacity of arbitrary single user channels without feedback not necessarily information stable stationary etc is proved capacity is shown to equal the supremum over all input processes of the input output inf information rate defined as the liminf in probability of the normalized information density the key to this result is a new converse approach based on a simple new lower bound on the error probability of m ary hypothesis tests among equiprobable hypotheses a necessary and sufficient condition for the validity of the strong converse is given as well as general expressions for spl epsiv capacity

efficient svm training using low rank kernel representations svm training is a convex optimization problem which scales with the training set size rather than the feature space dimension while this is usually considered to be a desired quality in large scale problems it may cause training to be impractical the common techniques to handle this difficulty basically build a solution by solving a sequence of small scale subproblems our current effort is concentrated on the rank of the kernel matrix as a source for further enhancement of the training procedure we first show that for a low rank kernel matrix it is possible to design a better interior point method ipm in terms of storage requirements as well as computational complexity we then suggest an efficient use of a known factorization technique to approximate a given kernel matrix by a low rank matrix which in turn will be used to feed the optimizer finally we derive an upper bound on the change in the objective function value based on the approximation error and the number of active constraints support vectors this bound is general in the sense that it holds regardless of the approximation method

operational capacity and pseudoclassicality of a quantum channel we explore some basic properties of coding theory of a general quantum communication channel and its operational capacity including 1 adaptive measurement with feedback code 2 reconsideration of single letterized capacity formula and 3 pseudoclassicality of a channel

bounds on the bayes classification error based on pairwise risk functions upper and lower bounds on the bayes risk for multiple composite hypothesis classification are obtained bounds on the bayes risk for m simple classes are derived in terms of the risk functions for m 1 classes and so on until the desired result depends only on the pairwise m 2 bayes risks a method of computing upper and lower bounds on the pairwise bayes risk for composite classes is developed algorithms for computing the upper and lower bounds for the general m class case and for composite hypothesis classes are presented numerical examples of the application of the bounding techniques to a problem involving the classification of aircraft are discussed results for the bounds and other performance measures are compared for the most interesting cases

estimating the confidence interval for prediction errors of support vector machine classifiers support vector machine svm is one of the most popular and promising classification algorithms after a classification rule is constructed via the svm it is essential to evaluate its prediction accuracy in this paper we develop procedures for obtaining both point and interval estimators for the prediction error under mild regularity conditions we derive the consistency and asymptotic normality of the prediction error estimators for svm with finite dimensional kernels a perturbation resampling procedure is proposed to obtain interval estimates for the prediction error in practice with numerical studies on simulated data and a benchmark repository we recommend the use of interval estimates centered at the cross validated point estimates for the prediction error further applications of the proposed procedure in model evaluation and feature selection are illustrated with two examples

polar codes for arbitrary classical quantum channels and arbitrary cq macs we prove polarization theorems for arbitrary classical quantum cq channels the input alphabet is endowed with an arbitrary abelian group operation and an arikan style transformation is applied using this operation it is shown that as the number of polarization steps becomes large the synthetic cq channels polarize to deterministic homomorphism channels that project their input to a quotient group of the input alphabet this result is used to construct polar codes for arbitrary cq channels and arbitrary cq multiple access channels the encoder can be implemented in o n log n operations where n is the blocklength of the code a quantum successive cancellation decoder for the constructed codes is proposed it is shown that the probability of error of this decoder decays faster than 2 n beta for any beta

an extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons in a recently published paper in jmlr dem sar 2006 recommends a set of non parametric statistical tests and procedures which can be safely used for comparing the performance of classifiers over multiple data sets after studying the paper we realize that the paper correctly introduces the basic procedures and some of the most advanced ones when comparing a control method however it does not deal with some advanced topics in depth regarding these topics we focus on more powerful proposals of statistical procedures for comparing n n classifiers moreover we illustrate an easy way of obtaining adjusted and comparable p values in multiple comparison procedures

joint sampling distribution between actual and estimated classification errors for linear discriminant analysis error estimation must be used to find the accuracy of a designed classifier an issue that is critical in biomarker discovery for disease diagnosis and prognosis in genomics and proteomics this paper presents for what is believed to be the first time the analytical formulation for the joint sampling distribution of the actual and estimated errors of a classification rule the analysis presented here concerns the linear discriminant analysis lda classification rule and the resubstitution and leave one out error estimators under a general parametric gaussian assumption exact results are provided in the univariate case and a simple method is suggested to obtain an accurate approximation in the multivariate case it is also shown how these results can be applied in the computation of condition bounds and the regression of the actual error given the observed error estimate in contrast to asymptotic results the analysis presented here is applicable to finite training data in particular it applies in the small sample settings commonly found in genomics and proteomics applications numerical examples which include parameters estimated from actual microarray data illustrate the analysis throughout

exact representation of the second order moments for resubstitution and leave one out error estimation for linear discriminant analysis in the univariate heteroskedastic gaussian model this paper provides exact analytical expressions for the bias variance and rms for the resubstitution and leave one out error estimators in the case of linear discriminant analysis lda in the univariate heteroskedastic gaussian model neither the variances nor the sample sizes for the two classes need be the same the generality of heteroskedasticity unequal variances is a fundamental feature of the work presented in this paper which distinguishes it from past work the expected resubstitution and leave one out errors are represented by probabilities involving bivariate gaussian distributions their second moments and cross moments with the actual error are represented by 3 and 4 variate gaussian distributions from these the bias deviation variance and rms for resubstitution and leave one out as estimators of the actual error can be computed the rms expressions are applied to the determination of sample size and illustrated in biomarker classification

block regularized repeated learning testing for estimating generalization error abstract repeated learning testing rlt is a popular cross validation method that is usually adopted in estimation and comparison of algorithm performance in machine learning however the variance of the estimator of the generalization error in the rlt method is easily affected by random partitions poor data partitioning may cause a large overlap between any two training sets of rlt and enlarge the variance in the rlt estimator of the generalization error thus in this study we constrain numbers of overlapping samples between any two training sets and construct a novel data partitioning schema in which rlt is called the block regularized rlt brlt we theoretically prove that the brlt estimator has a smaller variance than does the rlt estimator specifically the variance of the rlt estimator reaches its minimum when samples in a data set equally occur in all training sets and all the numbers of overlapping samples are equal furthermore we provide easy to use construction algorithms of brlt s partition set for several training set size and partition count settings by adopting two level orthogonal arrays we also illustrate brlt s optimal properties with several simulated and real world examples

extended kernel recursive least squares algorithm this paper presents a kernelized version of the extended recursive least squares ex krls algorithm which implements for the first time a general linear state model in reproducing kernel hilbert spaces rkhs or equivalently a general nonlinear state model in the input space the center piece of this development is a reformulation of the well known extended recursive least squares ex rls algorithm in rkhs which only requires inner product operations between input vectors thus enabling the application of the kernel property commonly known as the kernel trick the first part of the paper presents a set of theorems that shows the generality of the approach the ex krls is preferable to 1 a standard kernel recursive least squares krls in applications that require tracking the state vector of general linear state space models in the kernel space or 2 an ex rls when the application requires a nonlinear observation and state models the second part of the paper compares the ex krls in nonlinear rayleigh multipath channel tracking and in lorenz system modeling problem we show that the proposed algorithm is able to outperform the standard krls and ex rls in both simulations

from rough path estimates to multilevel monte carlo new classes of stochastic differential equations can now be studied using rough path theory see e g t j lyons m caruana and t levy differential equations driven by rough paths springer berlin 2007 or p k friz and m hairer a course on rough paths universitext springer cham 2014 in this paper we investigate from a numerical analysis point of view stochastic differential equations driven by gaussian noise in the aforementioned sense our focus lies on numerical implementations and more specifically on the savings possible via multilevel methods our analysis relies on a subtle combination of pathwise estimates gaussian concentration and multilevel ideas numerical examples are given which both illustrate and confirm our findings

a two dimensional accuracy based measure for classification performance abstract accuracy has been used traditionally to evaluate the performance of classifiers however it is well known that accuracy is not able to capture all the different factors that characterize the performance of a multiclass classifier in this manuscript accuracy is studied and analyzed as a weighted average of the classification rate of each class this perspective allows us to propose the dispersion of the classification rate of each class as its complementary measure in this sense a graphical performance metric which is defined in a two dimensional space composed by accuracy and dispersion is proposed to evaluate the performance of classifiers we show that the combined values of accuracy and dispersion must fall within a clearly bounded two dimensional region different for each problem the nature of this region depends only on the a priori probability of each class and not on the classifier used thus the performance of multiclassifiers is represented in a two dimensional space where the models can be compared in a more fair manner providing greater awareness of the strategies that are more accurate when trying to improve the performance of a classifier furthermore we experimentally analyze the behavior of seven different performance metrics based on the computation of the confusion matrix values in several scenarios identifying clusters and relationships between measures as shown in the experimentation the graphical metric proposed is specially suitable in challenging highly imbalanced and with a high number of classes datasets the approach proposed is a novel point of view to address the evaluation of multiclassifiers and it is an alternative to other evaluation measures used in machine learning

fuzzy rough regression with application to wind speed prediction accurate wind speed prediction is a prerequisite of large scale wind power generation there are several uncertain factors which degrade the performance of the current wind speed prediction systems fuzzy rough sets are considered as a powerful tool to deal with uncertainty and have been widely discussed and applied in classification learning in this work we describe a regression algorithm based on fuzzy rough sets consisting of fuzzy partition fuzzy approximation and estimation of regression values in this algorithm the training set is divided into k fuzzy classes with fuzzy partition and then the predicted values of test samples are determined in the finite intervals with fuzzy rough approximation finally they are estimated with lower and upper limits of the intervals numerical experiments on uci data sets and wind speed prediction show the effectiveness of the proposed algorithm

credible intervals for precision and recall based on a k fold cross validated beta distribution in typical machine learning applications such as information retrieval precision and recall are two commonly used measures for assessing an algorithm s performance symmetrical confidence intervals based on k fold cross validated t distributions are widely used for the inference of precision and recall measures as we confirmed through simulated experiments however these confidence intervals often exhibit lower degrees of confidence which may easily lead to liberal inference results thus it is crucial to construct faithful confidence credible intervals for precision and recall with a high degree of confidence and a short interval length in this study we propose two posterior credible intervals for precision and recall based on k fold cross validated beta distributions the first credible interval for precision or recall is constructed based on the beta posterior distribution inferred by all k data sets corresponding to k confusion matrices from a k fold cross validation second considering that each data set corresponding to a confusion matrix from a k fold cross validation can be used to infer a beta posterior distribution of precision or recall the second proposed credible interval for precision or recall is constructed based on the average of k beta posterior distributions experimental results on simulated and real data sets demonstrate that the first credible interval proposed in this study almost always resulted in degrees of confidence greater than 95 with an acceptable degree of confidence both of our two proposed credible intervals have shorter interval lengths than those based on a corrected k fold cross validated t distribution meanwhile the average ranks of these two credible intervals are superior to that of the confidence interval based on a k fold cross validated t distribution for the degree of confidence and are superior to that of the confidence interval based on a corrected k fold cross validated t distribution for the interval length in all 27 cases of simulated and real data experiments however the confidence intervals based on the k fold and corrected k fold cross validated t distributions are in the two extremes thus when focusing on the reliability of the inference for precision and recall the proposed methods are preferable especially for the first credible interval

linear model selection by cross validation abstract we consider the problem of selecting a model having the best predictive ability among a class of linear models the popular leave one out cross validation method which is asymptotically equivalent to many other model selection methods such as the akaike information criterion aic the c p and the bootstrap is asymptotically inconsistent in the sense that the probability of selecting the model with the best predictive ability does not converge to 1 as the total number of observations n we show that the inconsistency of the leave one out cross validation can be rectified by using a leave n v out cross validation with n v the number of observations reserved for validation satisfying n v n 1 as n this is a somewhat shocking discovery because nv n 1 is totally opposite to the popular leave one out recipe in cross validation motivations justifications and discussions of some practical aspects of the use of the leave n v out cross validation method are provided and results