max norm optimization for robust matrix recovery this paper studies the matrix completion problem under arbitrary sampling schemes we propose a new estimator incorporating both max norm and nuclear norm regularization based on which we can conduct efficient low rank matrix recovery using a random subset of entries observed with additive noise under general non uniform and unknown sampling distributions this method significantly relaxes the uniform sampling assumption imposed for the widely used nuclear norm penalized approach and makes low rank matrix recovery feasible in more practical settings theoretically we prove that the proposed estimator achieves fast rates of convergence under different settings computationally we propose an alternating direction method of multipliers algorithm to efficiently compute the estimator which bridges a gap between theory and practice of machine learning methods with max norm regularization further we provide thorough numerical studies to evaluate the proposed method using both simulated and real datasets

the mathematics of eigenvalue optimization optimization problems involving the eigenvalues of symmetric and nonsymmetric matrices present a fascinating mathematical challenge such problems arise often in theory and practice particularly in engineering design and are amenable to a rich blend of classical mathematical techniques and contemporary optimization theory this essay presents a personal choice of some central mathematical ideas outlined for the broad optimization community i discuss the convex analysis of spectral functions and invariant matrix norms touching briefly on semidefinite representability and then outlining two broader algebraic viewpoints based on hyperbolic polynomials and lie algebra analogous nonconvex notions lead into eigenvalue perturbation theory the last third of the article concerns stability for polynomials matrices and associated dynamical systems ending with a section on robustness the powerful and elegant language of nonsmooth analysis appears throughout as a unifying narrative thread

information geometry on hierarchy of probability distributions an exponential family or mixture family of probability distributions has a natural hierarchical structure this paper gives an orthogonal decomposition of such a system based on information geometry a typical example is the decomposition of stochastic dependency among a number of random variables in general they have a complex structure of dependencies pairwise dependency is easily represented by correlation but it is more difficult to measure effects of pure triplewise or higher order interactions dependencies among these variables stochastic dependency is decomposed quantitatively into an orthogonal sum of pairwise triplewise and further higher order dependencies this gives a new invariant decomposition of joint entropy this problem is important for extracting intrinsic interactions in firing patterns of an ensemble of neurons and for estimating its functional connections the orthogonal decomposition is given in a wide class of hierarchical structures including both exponential and mixture families as an example we decompose the dependency in a higher order markov chain into a sum of those in various lower order markov chains

the arithmetic of the digital computer a new approach a new approach to the arithmetic of the digital computer is surveyed the methodology for defining and implementing floating point arithmetic is described shortcomings of elementary floating point arithmetic are revealed through sample problems the development of automatic computation with emphasis on the user control of errors is reviewed the limitations of conventional rule of thumb procedures for error control in scientific computation are demonstrated by means of examples computer arithmetic is extended so that the arithmetic operations in the linear spaces and their interval correspondents which are most commonly used in computation can be performed with maximum accuracy on digital computers a new fundamental computer operation the scalar product is introduced to develop this advanced computer arithmetic a process of automatic error control called validation which delivers high accuracy with guarantees for scientific computations is described validation of computations for a large class of nu

two stage orthogonal least squares methods for neural network construction a number of neural networks can be formulated as the linear in the parameters models training such networks can be transformed to a model selection problem where a compact model is selected from all the candidates using subset selection algorithms forward selection methods are popular fast subset selection approaches however they may only produce suboptimal models and can be trapped into a local minimum more recently a two stage fast recursive algorithm tsfra combining forward selection and backward model refinement has been proposed to improve the compactness and generalization performance of the model this paper proposes unified two stage orthogonal least squares methods instead of the fast recursive based methods in contrast to the tsfra this paper derives a new simplified relationship between the forward and the backward stages to avoid repetitive computations using the inherent orthogonal properties of the least squares methods furthermore a new term exchanging scheme for backward model refinement is introduced to reduce computational demand finally given the error reduction ratio criterion effective and efficient forward and backward subset selection procedures are proposed extensive examples are presented to demonstrate the improved model compactness constructed by the proposed technique in comparison with some popular methods

a comparison of descriptive models of a single spike train by information geometric measure in examining spike trains different models are used to describe their structure the different models often seem quite similar but because they are cast in different formalisms it is often difficult to compare their predictions here we use the information geometric measure an orthogonal coordinate representation of point processes to express different models of stochastic point processes in a common coordinate system within such a framework it becomes straightforward to visualize higher order correlations of different models and thereby assess the differences between models we apply the information geometric measure to compare two similar but not identical models of neuronal spike trains the inhomogeneous markov and the mixture of poisson models it is shown that they differ in the second and higher order interaction terms in the mixture of poisson model the second and higher order interactions are of comparable magnitude within each order whereas in the inhomogeneous markov model they have alternating signs over different orders this provides guidance about what measurements would effectively separate the two models as newer models are proposed they also can be compared to these models using information geometry

deterministic and probabilistic conditions for finite completability of low tucker rank tensor we investigate the fundamental conditions on the sampling pattern i e locations of the sampled entries for finite completability of a low rank tensor given some components of its tucker rank in order to find the deterministic necessary and sufficient conditions we propose an algebraic geometric analysis on the tucker manifold which allows us to incorporate multiple rank components in the proposed analysis in contrast with the conventional geometric approaches on the grassmannian manifold this analysis characterizes the algebraic independence of a set of polynomials defined based on the sampling pattern which is closely related to finite completability of the sampled tensor where finite completability simply means that the number of possible completions of the sampled tensor is finite probabilistic conditions are then studied and a lower bound on the sampling probability is given which guarantees that the proposed deterministic conditions on the sampling patterns for finite completability hold with high probability furthermore using the proposed geometric approach for finite completability we propose a sufficient condition on the sampling pattern that ensures there exists exactly one completion of the sampled tensor

decoupling inequalities for the tail probabilities of multivariate u statistics in this paper we present a decoupling inequality that shows that multivariate u statistics can be studied as sums of conditionally independent random variables this result has important implications in several areas of probability and statistics including the study of random graphs and multiple stochastic integration more precisely we get the following result let x j be a sequence of independent random variables on a measurable space mathscr j s and let x j i j 1 ldots k be k independent copies of x i let f i 1i 2 ldots i k be families of functions of k variables taking s times cdots times s into a banach space b cdots then for all n geq k geq 2 t 0 there exist numerical constants c k depending on k only so that p bigg big sum 1 leq i 1 neq i 2 neq cdots neq i k leq n f i 1 cdots i k x 1 i 1 x 1 i 2 ldots x 1 i k big geq t bigg leq c kp bigg c k big sum 1 leq i 1 neq i 2 neq cdots neq i k leq n f i 1 cdots i k x 1 i 1 x 2 i 2 ldots x k i k big geq t bigg the reverse bound holds if in addition the following symmetry condition holds almost surely f i 1i 2 cdots i k x i 1 x i 2 ldots x i k f i pi 1 i pi 2 cdots i pi k x i pi 1 x i pi 2 ldots x i pi k for all permutations pi of 1 ldots k

nonlinear model structure design and construction using orthogonal least squares and d optimality design a very efficient learning algorithm for model subset selection is introduced based on a new composite cost function that simultaneously optimizes the model approximation ability and model robustness and adequacy the derived model parameters are estimated via forward orthogonal least squares but the model subset selection cost function includes a d optimality design criterion that maximizes the determinant of the design matrix of the subset to ensure the model robustness adequacy and parsimony of the final model the proposed approach is based on the forward orthogonal least square ols algorithm such that new d optimality based cost function is constructed based on the orthogonalization process to gain computational advantages and hence to maintain the inherent advantage of computational efficiency associated with the conventional forward ols approach illustrative examples are included to demonstrate the effectiveness of the new approach

indirectly estimated adaptive detectors for code division multiple access signals this paper presents the development of indirectly estimated adaptive idea detectors for code division multiple access cdma signals in wireless communication systems multiuser cdma detection is formulated as an inverse problem in the presence of noise and the estimation of the detector parameters is accomplished using elements of regularization theory the development of autonomous cdma detectors that are capable of adjusting to the channel characteristics with the support of training sequences relies on data dependent procedures designed to reduce the effect of noise and interfering users on the performance of the detector the detector parameters can be computed by adaptive algorithms that require no matrix inversion the proposed idea detector is evaluated and compared with existing multiuser cdma detectors in terms of their robustness to noise resistance to the near far problem and ability to handle multipath signals

entropy numbers of embeddings of schatten classes abstract let 0 p q and denote by s p n and s q n the corresponding finite dimensional schatten classes we prove optimal bounds up to constants only depending on p and q for the entropy numbers of natural embeddings between s p n and s q n this complements the known results in the classical setting of natural embeddings between finite dimensional l p spaces due to schutt edmunds triebel triebel and guedon litvak kuhn we present a rather short proof that uses all the known techniques as well as a constructive proof of the upper bound in the range n n n 2 that allows deeper structural insight and is therefore interesting in its own right our main result can also be used to provide an alternative proof of recent lower bounds in the area of low rank matrix recovery

characterization of the subdifferential of some matrix norms abstract a characterization is given of the subdifferential of matrix norms from two classes orthogonally invariant norms and operator or subordinate norms specific results are derived for some special cases

cartoon texture image decomposition using blockwise low rank texture characterization using a novel characterization of texture we propose an image decomposition technique that can effectively decomposes an image into its cartoon and texture components the characterization rests on our observation that the texture component enjoys a blockwise low rank nature with possible overlap and shear because texture in general is globally dissimilar but locally well patterned more specifically one can observe that any local block of the texture component consists of only a few individual patterns based on this premise we first introduce a new convex prior named the block nuclear norm bnn leading to a suitable characterization of the texture component we then formulate a cartoon texture decomposition model as a convex optimization problem where the simultaneous estimation of the cartoon and texture components from a given image or degraded observation is executed by minimizing the total variation and bnn in addition patterns of texture extending in different directions are extracted separately which is a special feature of the proposed model and of benefit to texture analysis and other applications furthermore the model can handle various types of degradation occurring in image processing including blur missing pixels with several types of noise by rewriting the problem via variable splitting the so called alternating direction method of multipliers becomes applicable resulting in an efficient algorithmic solution to the problem numerical examples illustrate that the proposed model is very selective to patterns of texture which makes it produce better results than state of the art decomposition models

the ornstein uhlenbeck process does not reproduce spiking statistics of neurons in prefrontal cortex cortical neurons of behaving animals generate irregular spike sequences recently there has been a heated discussion about the origin of this irregularity softky and koch 1993 pointed out the inability of standard single neuron models to reproduce the irregularity of the observed spike sequences when the model parameters are chosen within a certain range that they consider to be plausible shadlen and newsome 1994 on the other hand demonstrated that a standard leaky integrate and fire model can reproduce the irregularity if the inhibition is balanced with the excitation motivated by this discussion we attempted to determine whether the ornstein uhlenbeck process which is naturally derived from the leaky integration assumption can in fact reproduce higher order statistics of biological data for this purpose we consider actual neuronal spike sequences recorded from the monkey prefrontal cortex to calculate the higher order statistics of the interspike intervals consistency of the data with the model is examined on the basis of the coefficient of variation and the skewness coefficient which are respectively a measure of the spiking irregularity and a measure of the asymmetry of the interval distribution it is found that the biological data are not consistent with the model if the model time constant assumes a value within a certain range believed to cover all reasonable values this fact suggests that the leaky integrate and fire model with the assumption of uncorrelated inputs is not adequate to account for the spiking in at least some cortical neurons

fundamental sampling patterns for low rank multi view data completion abstract we consider the multi view data completion problem i e to complete a matrix u u 1 u 2 where the ranks of u u1 and u2 are given in particular we investigate the fundamental conditions on the sampling pattern i e locations of the sampled entries for finite completability of such a multi view data given the corresponding rank constraints we provide a geometric analysis on the manifold structure for multi view data to incorporate more than one rank constraint we derive a probabilistic condition in terms of the number of samples per column that guarantees finite completability with high probability finally we derive the guarantees for unique completability numerical results demonstrate reduced sampling complexity when the multi view structure is taken into account as compared to when only low rank structure of individual views is taken into account then we propose an apporach using newton s method to almost achieve these information theoretic bounds for mulit view data retrieval by taking advantage of the rank decomposition and the analysis in this work

a new class of wavelet networks for nonlinear system identification a new class of wavelet networks wns is proposed for nonlinear system identification in the new networks the model structure for a high dimensional system is chosen to be a superimposition of a number of functions with fewer variables by expanding each function using truncated wavelet decompositions the multivariate nonlinear networks can be converted into linear in the parameter regressions which can be solved using least squares type methods an efficient model term selection approach based upon a forward orthogonal least squares ols algorithm and the error reduction ratio err is applied to solve the linear in the parameters problem in the present study the main advantage of the new wn is that it exploits the attractive features of multiscale wavelet decompositions and the capability of traditional neural networks by adopting the analysis of variance anova expansion wns can now handle nonlinear identification problems in high dimensions

a optimality orthogonal forward regression algorithm using branch and bound in this brief we propose an orthogonal forward regression ofr algorithm based on the principles of the branch and bound bb and a optimality experimental design at each forward regression step each candidate from a pool of candidate regressors referred to as s is evaluated in turn with three possible decisions 1 one of these is selected and included into the model 2 some of these remain in s for evaluation in the next forward regression step and 3 the rest are permanently eliminated from s based on the bb principle in combination with an a optimality composite cost function for model structure determination a simple adaptive diagnostics test is proposed to determine the decision boundary between 2 and 3 as such the proposed algorithm can significantly reduce the computational cost in the a optimality ofr algorithm numerical examples are used to demonstrate the effectiveness of the proposed algorithm

information geometric measures as robust estimators of connection strengths and external inputs information geometry has been suggested to provide a powerful tool for analyzing multineuronal spike trains among several advantages of this approach a significant property is the close link between information geometric measures and neural network architectures previous modeling studies established that the first and second order information geometric measures corresponded to the number of external inputs and the connection strengths of the network respectively this relationship was however limited to a symmetrically connected network and the number of neurons used in the parameter estimation of the log linear model needed to be known recently simulation studies of biophysical model neurons have suggested that information geometry can estimate the relative change of connection strengths and external inputs even with asymmetric connections inspired by these studies we analytically investigated the link between the information geometric measures and the neural network structure with asymmetrically connected networks of n neurons we focused on the information geometric measures of orders one and two which can be derived from the two neuron log linear model because unlike higher order measures they can be easily estimated experimentally considering the equilibrium state of a network of binary model neurons that obey stochastic dynamics we analytically showed that the corrected first and second order information geometric measures provided robust and consistent approximation of the external inputs and connection strengths respectively these results suggest that information geometric measures provide useful insights into the neural network architecture and that they will contribute to the study of system level neuroscience

rank minimization over finite fields fundamental limits and coding theoretic interpretations this paper establishes information theoretic limits for estimating a finite field low rank matrix given random linear measurements of it these linear measurements are obtained by taking inner products of the low rank matrix with random sensing matrices necessary and sufficient conditions on the number of measurements required are provided it is shown that these conditions are sharp and the minimum rank decoder is asymptotically optimal the reliability function of this decoder is also derived by appealing to de caen s lower bound on the probability of a union the sufficient condition also holds when the sensing matrices are sparse a scenario that may be amenable to efficient decoding more precisely it is shown that if the n n sensing matrices contain on average nlog n entries the number of measurements required is the same as that when the sensing matrices are dense and contain entries drawn uniformly at random from the field analogies are drawn between the aforementioned results and rank metric codes in the coding theory literature in fact we are also strongly motivated by understanding when minimum rank distance decoding of random rank metric codes succeeds to this end we derive minimum distance properties of equiprobable and sparse rank metric codes these distance properties provide a precise geometric interpretation of the fact that the sparse ensemble requires as few measurements as the dense one

convergence rates with inexact non expansive operators in this paper we present a convergence rate analysis for the inexact krasnosel ski mann iteration built from non expansive operators the presented results include two main parts we first establish the global pointwise and ergodic iteration complexity bounds then under a metric sub regularity assumption we establish a local linear convergence for the distance of the iterates to the set of fixed points the obtained results can be applied to analyze the convergence rate of various monotone operator splitting methods in the literature including the forward backward splitting the generalized forward backward the douglas rachford splitting alternating direction method of multipliers and primal dual splitting methods for these methods we also develop easily verifiable termination criteria for finding an approximate solution which can be seen as a generalization of the termination criterion for the classical gradient descent method we finally develop a parallel analysis for the non stationary krasnosel ski mann iteration

rigorous perturbation bounds for eigenvalues and eigenvectors of a matrix tight perturbation bounds are given for the shifts in the eigenvalues and eigenvectors of a matrix the case of simple as well as multiple eigenvalues is treated with the results tested numerically

synchronous firing and higher order interactions in neuron pool the stochastic mechanism of synchronous firing in a population of neurons is studied from the point of view of information geometry higher order interactions of neurons which cannot be reduced to pairwise correlations are proved to exist in synchronous firing in a neuron pool where each neuron fires stochastically the probability distribution q r of the activity r which is the fraction of firing neurons in the pool is studied when q r has a widespread distribution in particular when q r has two peaks the neurons fire synchronously at one time and are quiescent at other times the mechanism of generating such a probability distribution is interesting because the activity r is concentrated on its mean value when each neuron fires independently because of the law of large numbers even when pairwise interactions or third order interactions exist the concentration is not resolved this shows that higher order interactions are necessary to generate widespread activity distributions we analyze a simple model in which neurons receive common overlapping inputs and prove that such a model can have a widespread distribution of activity generating higher order stochastic interactions

matrix completion by least square low rank and sparse self representations abstract conventional matrix completion methods are generally based on rank minimization these methods assume that the given matrix is of low rank and the data points are drawn from a single subspace of low dimensionality therefore they are not effective in completing matrices where the data are drawn from multiple subspaces in this paper we establish a novel matrix completion framework that is based on self representation specifically least square low rank and sparse self representations based matrix completion algorithms are provided the underlying idea is that one data point can be efficiently reconstructed by other data points belonging to a common subspace where the missing entries are recovered so as to fit the common subspace the proposed algorithms actually maximize the weighted correlations among the columns of a given matrix we prove that the proposed algorithms are approximations for rank minimization of the incomplete matrix in addition they are able to complete high rank or even full rank matrices when the data are drawn from multiple subspaces comparative studies are conducted on synthetic datasets natural image inpainting tasks temperature prediction task and collaborative filtering tasks the results show that the proposed algorithms often outperform other state of the art algorithms in various tasks

group based image decomposition using 3 d cartoon and texture priors we propose a novel image decomposition method to decompose an image into its cartoon and texture components to exploit the nonlocal self similarity of cartoon plus texture images we construct groups by stacking together similar image patches into 3 d arrays and consider group as the basic unit of decomposition we decompose each group via a convex optimization model consisting of 3 d cartoon and texture priors these priors characterize the local properties of the cartoon and texture components and the nonlocal similarity within each component in a unified and natural manner we develop the alternating direction method of multipliers admm to efficiently solve the proposed model for further improvement we investigate an adaptive rule for the estimation of the regularization parameter the proposed method is also extended to tackle noisy images numerical experiments confirm that the performance of the proposed method is competitive with some of the state of the art schemes

matrix completion from noisy entries given a matrix m of low rank we consider the problem of reconstructing it from noisy observations of a small random subset of its entries the problem arises in a variety of applications from collaborative filtering the netflix problem to structure from motion and positioning we study a low complexity algorithm introduced by keshavan montanari and oh 2010 based on a combination of spectral techniques and manifold optimization that we call here optspace we prove performance guarantees that are order optimal in a number of circumstances

consistency and mse performance of music based doa of a single source in white noise with randomly missing data music is a popular algorithm for estimating the direction of arrival doa in array signal processing applications in this paper we analyze the performance of the music algorithm for a single source system in the presence of noisy and missing data when only a random subset of the entries in the data matrix are observed we prove consistency of the doa estimate when signal from a single source is impinging on low coherence arrays and derive an analytic expression for the mean squared error mse performance of music for the case of uniform linear arrays in the large array and relatively large sample setting our analysis is mathematically justified in both the sample rich and deficient regimes the expression for the mse is a simple function of array geometry signal to noise ratio snr the fraction of entries observed and the ratio of the number of sensors to number of snapshots we derive a phase transition threshold which separates a regime where music is consistent from a regime where music is inconsistent this threshold depends upon the snr the probability of observing entries in the data matrix and number of sensors and snapshots in a simple manner which we make explicit

regularization in the selection of radial basis function centers subset selection and regularization are two well known techniques that can improve the generalization performance of nonparametric linear regression estimators such as radial basis function networks this paper examines regularized forward selection rfs a combination of forward subset selection and zero order regularization an efficient implementation of rfs into which either delete 1 or generalized cross validation can be incorporated and a reestimation formula for the regularization parameter are also discussed simulation studies are presented that demonstrate improved generalization performance due to regularization in the forward selection of radial basis function centers

generalized neurofuzzy network modeling algorithms using bezier bernstein polynomial functions and additive decomposition this paper introduces a new neurofuzzy model construction algorithm for nonlinear dynamic systems based upon basis functions that are bezier bernstein polynomial functions this paper is generalized in that it copes with n dimensional inputs by utilising an additive decomposition construction to overcome the curse of dimensionality associated with high n this new construction algorithm also introduces univariate bezier bernstein polynomial functions for the completeness of the generalized procedure like the b spline expansion based neurofuzzy systems bezier bernstein polynomial function based neurofuzzy networks hold desirable properties such as nonnegativity of the basis functions unity of support and interpretability of basis function as fuzzy membership functions moreover with the additional advantages of structural parsimony and delaunay input space partition essentially overcoming the curse of dimensionality associated with conventional fuzzy and rbf networks this new modeling network is based on additive decomposition approach together with two separate basis function formation approaches for both univariate and bivariate bezier bernstein polynomial functions used in model construction the overall network weights are then learnt using conventional least squares methods numerical examples are included to demonstrate the effectiveness of this new data based modeling approach

matrix completion from a few entries let m be an n n matrix of rank r and assume that a uniformly random subset e of its entries is observed we describe an efficient algorithm which we call optspace that reconstructs m from e o rn observed entries with relative root mean square error 1 2 rmse c nr e 1 2 with probability larger than 1 1 n3 further if r o 1 and m is sufficiently unstructured then optspace reconstructs it exactly from e o n log n entries with probability larger than 1 1 n3 this settles in the case of bounded rank a question left open by candes and recht and improves over the guarantees for their reconstruction algorithm the complexity of our algorithm is o e r log n which opens the way to its use for massive data sets in the process of proving these statements we obtain a generalization of a celebrated result by friedman kahn szemeredi and feige ofek on the spectrum of sparse random matrices

perturbation bounds for polynomials using two different elementary approaches we derive a global and a local perturbation theorem on polynomial zeros that significantly improve the results of ostrowski acta math 72 99 257 1940 elsner et al linear algebra appl 142 195 209 1990 a comparison of different perturbation bounds shows that our results are better in many cases than the similar local result of beauzamy can math bull 42 1 3 12 1999 using the matrix theoretical approach we also improve the backward stability result of edelman and murakami proceedings of the fifth siam conference on applied linear algebra siam philapdelphia 1994 math comput 64 210 763 1995

recovering the missing components in a large noisy low rank matrix application to sfm in computer vision it is common to require operations on matrices with missing data for example because of occlusion or tracking failures in the structure from motion sfm problem such a problem can be tackled allowing the recovery of the missing values if the matrix should be of low rank when noise free the filling in of missing values is known as imputation imputation can also be applied in the various subspace techniques for face and shape classification online recommender systems and a wide variety of other applications however iterative imputation can lead to the recovery of data that is seriously in error in this paper we provide a method to recover the most reliable imputation in terms of deciding when the inclusion of extra rows or columns containing significant numbers of missing entries is likely to lead to poor recovery of the missing parts although the proposed approach can be equally applied to a wide range of imputation methods this paper addresses only the sfm problem the performance of the proposed method is compared with jacobs and shum s methods for sfm

generalization of the bauer fike theorem the bauer fike theorem for diagonalizable matrices is generalized to cases where i non diagonalizable matrices or ii only part of the spectrum is considered

alternating direction method with gaussian back substitution for separable convex programming we consider the linearly constrained separable convex minimization problem whose objective function is separable into m individual convex functions with nonoverlapping variables a douglas rachford alternating direction method of multipliers adm has been well studied in the literature for the special case of m 2 but the convergence of extending adm to the general case of m ge 3 is still open in this paper we show that the straightforward extension of adm is valid for the general case of m ge 3 if it is combined with a gaussian back substitution procedure the resulting adm with gaussian back substitution is a novel approach towards the extension of adm from m 2 to m ge 3 and its algorithmic framework is new in the literature for the adm with gaussian back substitution we prove its convergence via the analytic framework of contractive type methods and we show its numerical efficiency by some application problems

noisy matrix completion understanding statistical guarantees for convex relaxation via nonconvex optimization this paper studies noisy low rank matrix completion given partial and noisy entries of a large low rank matrix the goal is to estimate the underlying matrix faithfully and efficiently arguably o

accelerating permutation testing in voxel wise analysis through subspace tracking a new plugin for snpm permutation testing is a non parametric method for obtaining the max null distribution used to compute corrected p values that provide strong control of false positives in neuroimaging however the computational burden of running such an algorithm can be significant we find that by viewing the permutation testing procedure as the construction of a very large permutation testing matrix t one can exploit structural properties derived from the data and the test statistics to reduce the runtime under certain conditions in particular we see that t is low rank plus a low variance residual this makes t a good candidate for low rank matrix completion where only a very small number of entries of t 0 35 of all entries in our experiments have to be computed to obtain a good estimate based on this observation we present rapidpt an algorithm that efficiently recovers the max null distribution commonly obtained through regular permutation testing in voxel wise analysis we present an extensive validation on a synthetic dataset and four varying sized datasets against two baselines statistical nonparametric mapping snpm13 and a standard permutation testing implementation referred as naivept we find that rapidpt achieves its best runtime performance on medium sized datasets 50 n 200 with speedups of 1 5 38 vs snpm13 and 20x 1000 vs naivept for larger datasets n 200 rapidpt outperforms naivept 6 200 on all datasets and provides large speedups over snpm13 when more than 10000 permutations 2 15 are needed the implementation is a standalone toolbox and also integrated within snpm13 able to leverage multi core architectures when available

recovering low rank and sparse components of matrices from incomplete and noisy observations many problems can be characterized by the task of recovering the low rank and sparse components of a given matrix recently it was discovered that this nondeterministic polynomial time hard np hard task can be well accomplished both theoretically and numerically via heuristically solving a convex relaxation problem where the widely acknowledged nuclear norm and l 1 norm are utilized to induce low rank and sparsity this paper studies the recovery task in the general settings that only a fraction of entries of the matrix can be observed and the observation is corrupted by both impulsive and gaussian noise we show that the resulting model falls into the applicable scope of the classical augmented lagrangian method moreover the separable structure of the new model enables us to solve the involved subproblems more efficiently by splitting the augmented lagrangian function hence some splitting numerical algorithms are developed for solving the new recovery model some preliminary numerical experiments verify that these augmented lagrangian based splitting algorithms are easily implementable and surprisingly efficient for tackling the new recovery model

on l_q optimization and matrix completion rank minimization problems which consist of finding a matrix of minimum rank subject to linear constraints have been proposed in many areas of engineering and science a specific problem is the matrix completion problem in which a low rank data matrix can be recovered from incomplete samples of its entries by solving a rank penalized least squares problem the rank penalty is in fact the l0 norm of the matrix singular values a recent convex relaxation of this penalty is the commonly used l1 norm of the matrix singular values in this paper we bridge the gap between these two penalties and propose the lq 0 q 1 penalized least squares problem for matrix completion an iterative algorithm is developed by solving a non standard optimization problem and a non trivial convergence result is proved we illustrate with simulations comparing the reconstruction quality of the three matrix singular value penalty functions l0 l1 and lq 0 q 1

directly estimated adaptive detectors for code division multiple access signals this paper presents adaptive multiuser detectors for code division multiple access cdma signals in wireless communication systems directly estimated adaptive dea detectors are developed by formulating cdma detection as an inverse problem in the presence of channel induced interference and noise the detector parameters are computed by a fully sequential adaptive algorithm that requires no matrix inversion and can be implemented online as the data arrive at the receiver the proposed dea detector is experimentally evaluated in terms of its robustness to noise resistance to the near far problem and ability to handle multipath fading signals this experimental study indicates that the proposed dea detector requires shorter training sequences of bits to achieve the performance levels attained by existing adaptive implementations of the minimum mean square error detector

term and variable selection for non linear system identification the purpose of variable selection is to pre select a subset consisting of the significant variables or to eliminate the redundant variables from all the candidate variables of a system under study prior to model term detection it is required that the selected significant variables alone should sufficiently represent the system generally not all the model terms which are produced by combining different variables make an equal contribution to the system output and terms which make little contribution can be omitted a parsimonious representation which contains only the significant terms can often be obtained without the loss of representational accuracy by eliminating the redundant terms based on these observations a new variable and term selection algorithm is proposed in this paper the term detection algorithm can be applied to the general class of non linear modelling problems which can be expressed as a linear in the parameters form the variable selection procedure is based on locally linea

restricted strong convexity and weighted matrix completion optimal bounds with noise we consider the matrix completion problem under a form of row column weighted entrywise sampling including the case of uniformentrywise sampling as a special case we analyze the associated random observation operator and prove that with high probability it satisfies a form of restricted strong convexity with respect to weighted frobenius norm using this property we obtain as corollaries a number of error bounds on matrix completion in the weighted frobenius norm under noisy sampling and for both exact and near low rank matrices our results are based on measures of the spikiness and low rankness of matrices that are less restrictive than the incoherence conditions imposed in previous work our technique involves an m estimator that includes controls on both the rank and spikiness of the solution and we establish non asymptotic error bounds in weighted frobenius norm for recovering matrices lying with lq balls of bounded spikiness using information theoretic methods we show that no algorithm can achieve better estimates up to a logarithmic factor over these same sets showing that our conditions on matrices and associated rates are essentially optimal

orthogonal least squares methods and their application to non linear system identification abstract identification algorithms based on the well known linear least squares methods of gaussian elimination cholesky decomposition classical gram schmidt modified gram schmidt householder transformation givens method and singular value decomposition are reviewed the classical gram schmidt modified gram schmidt and householder transformation algorithms are then extended to combine structure determination or which terms to include in the model and parameter estimation in a very simple and efficient manner for a class of multivariate discrete time non linear stochastic systems which are linear in the parameters

correlations without synchrony peaks in spike train correlograms are usually taken as indicative of spike timing synchronization between neurons strictly speaking however a peak merely indicates that the two spike trains were not independent two biologically plausible ways of departing from independence that are capable of generating peaks very similar to spike timing peaks are described here covariations over trials in response latency and covariations over trials in neuronal excitability since peaks due to these interactions can be similar to spike timing peaks interpreting a correlogram may be a problem with ambiguous solutions what peak shapes do latency or excitability interactions generate when are they similar to spike timing peaks when can they be ruled out from having caused an observed correlogram peak these are the questions addressed here the previous article in this issue proposes quantitative methods to tell cases apart when latency or excitability covariations cannot be ruled out

a spike train probability model poisson processes usually provide adequate descriptions of the irregularity in neuron spike times after pooling the data across large numbers of trials as is done in constructing the peristimulus time histogram when probabilities are needed to describe the behavior of neurons within individual trials however poisson process models are often inadequate in principle an explicit formula gives the probability density of a single spike train in great generality but without additional assumptions the firing rate intensity function appearing in that formula cannot be estimated we propose a simple solution to this problem which is to assume that the time at which a neuron fires is determined probabilistically by and only by two quantities the experimental clock time and the elapsed time since the previous spike we show that this model can be fitted with standard methods and software and that it may used successfully to fit neuronal data

practical matrix completion and corruption recovery using proximal alternating robust subspace minimization low rank matrix completion is a problem of immense practical importance recent works on the subject often use nuclear norm as a convex surrogate of the rank function despite its solid theoretical foundation the convex version of the problem often fails to work satisfactorily in real life applications real data often suffer from very few observations with support not meeting the randomness requirements ubiquitous presence of noise and potentially gross corruptions sometimes with these simultaneously occurring this paper proposes a proximal alternating robust subspace minimization method to tackle the three problems the proximal alternating scheme explicitly exploits the rank constraint on the completed matrix and uses the ell 0 l 0 pseudo norm directly in the corruption recovery step we show that the proposed method for the non convex and non smooth model converges to a stationary point although it is not guaranteed to find the global optimal solution in practice we find that our algorithm can typically arrive at a good local minimizer when it is supplied with a reasonably good starting point based on convex optimization extensive experiments with challenging synthetic and real data demonstrate that our algorithm succeeds in a much larger range of practical problems where convex optimization fails and it also outperforms various state of the art algorithms

on maximal entries in the principal eigenvector of graphs abstract for an undirected connected graph it is well known that an eigenvector belonging to the principal eigenvalue of g can be given such that all entries are positive we ask whether this vector carries information on the structure of the graph and approach this question by investigating the values that can occur in its maximal entry

fuzzy basis functions universal approximation and orthogonal least squares learning fuzzy systems are represented as series expansions of fuzzy basis functions which are algebraic superpositions of fuzzy membership functions using the stone weierstrass theorem it is proved that linear combinations of the fuzzy basis functions are capable of uniformly approximating any real continuous function on a compact set to arbitrary accuracy based on the fuzzy basis function representations an orthogonal least squares ols learning algorithm is developed for designing fuzzy systems based on given input output pairs then the ols algorithm is used to select significant fuzzy basis functions which are used to construct the final fuzzy system the fuzzy basis function expansion is used to approximate a controller for the nonlinear ball and beam system and the simulation results show that the control performance is improved by incorporating some common sense fuzzy control rules

combined genetic algorithm optimization and regularized orthogonal least squares learning for radial basis function networks presents a two level learning method for radial basis function rbf networks a regularized orthogonal least squares rols algorithm is employed at the lower level to construct rbf networks while the two key learning parameters the regularization parameter and the rbf width are optimized using a genetic algorithm ga at the upper level nonlinear time series modeling and prediction is used as an example to demonstrate the effectiveness of this hierarchical learning approach

low rank tensor recovery via iterative hard thresholding abstract we study extensions of compressive sensing and low rank matrix recovery matrix completion to the recovery of low rank tensors of higher order from a small number of linear measurements while the theoretical understanding of low rank matrix recovery is already well developed only few contributions on the low rank tensor recovery problem are available so far in this paper we introduce versions of the iterative hard thresholding algorithm for several tensor decompositions namely the higher order singular value decomposition hosvd the tensor train format tt and the general hierarchical tucker decomposition ht we provide a partial convergence result for these algorithms which is based on a variant of the restricted isometry property of the measurement operator adapted to the tensor decomposition at hand that induces a corresponding notion of tensor rank we show that subgaussian measurement ensembles satisfy the tensor restricted isometry property with high probability under a certain almost optimal bound on the number of measurements which depends on the corresponding tensor format these bounds are extended to partial fourier maps combined with random sign flips of the tensor entries finally we illustrate the performance of iterative hard thresholding methods for tensor recovery via numerical experiments where we consider recovery from gaussian random measurements tensor completion recovery of missing entries and fourier measurements for third order tensors

a neurofuzzy network knowledge extraction and extended gram schmidt algorithm for model subspace decomposition this paper introduces a new neurofuzzy model construction and parameter estimation algorithm from observed finite data sets based on a takagi sugeno t s inference mechanism and a new extended gram schmidt orthogonal decomposition algorithm for the modeling of a priori unknown dynamical systems in the form of a set of fuzzy rules the paper introduces a one to one mapping between a fuzzy rule base and a model matrix feature subspace hence rule based knowledge can be extracted to enhance model transparency model transparency is explored by the derivation of an equivalence between an a optimality experimental design criterion of the weighting matrix and the average model output sensitivity to the fuzzy rule the a optimality experimental design criterion of the weighting matrices of fuzzy rules is used to construct an initial model rule base an extended gram schmidt algorithm is then developed to estimate the parameter vector for each rule this new algorithm decomposes the model rule bases via an orthogonal subspace decomposition approach so as to enhance model transparency with the capability of interpreting the derived rule base energy level

image deblurring with an inaccurate blur kernel using a group based low rank image prior abstract we address the problem of restoring an original image from its blurry and noisy observation together with inaccurate information of the blurring process for this purpose we propose an enhanced regularized structured total least squares rstls model that can estimate the latent image and blur kernel simultaneously in the proposed model both the image and the blur kernel are characterized by a group based low rank prior which assumes that a group of vectorized similar data patches can be well approximated by a low rank matrix we develop an alternating minimization algorithm to solve the proposed model efficiently numerical experiments demonstrate the effectiveness of our method in terms of both quantitative measures and visual quality

the power of convex relaxation near optimal matrix completion this paper is concerned with the problem of recovering an unknown matrix from a small fraction of its entries this is known as the matrix completion problem and comes up in a great number of applications including the famous netflix prize and other similar questions in collaborative filtering in general accurate recovery of a matrix from a small number of entries is impossible but the knowledge that the unknown matrix has low rank radically changes this premise making the search for solutions meaningful this paper presents optimality results quantifying the minimum number of entries needed to recover a matrix of rank r exactly by any method whatsoever the information theoretic limit more importantly the paper shows that under certain incoherence assumptions on the singular vectors of the matrix recovery is possible by solving a convenient convex program as soon as the number of entries is on the order of the information theoretic limit up to logarithmic factors this convex program simply finds among all matrices consistent with the observed entries that with minimum nuclear norm as an example we show that on the order of nr log n samples are needed to recover a random n x n matrix of rank r by any method and to be sure nuclear norm minimization succeeds as soon as the number of entries is of the form nr polylog n

on the properties of variational approximations of gibbs posteriors the pac bayesian approach is a powerful set of techniques to derive nonasymptotic risk bounds for random estimators the corresponding optimal distribution of estimators usually called the gibbs posterior is unfortunately often intractable one may sample from it using markov chain monte carlo but this is usually too slow for big datasets we consider instead variational approximations of the gibbs posterior which are fast to compute we undertake a general study of the properties of such approximations our main finding is that such a variational approximation has often the same rate of convergence as the original pac bayesian procedure it approximates in addition we show that when the risk function is convex a variational approximation can be obtained in polynomial time using a convex solver we give finite sample oracle inequalities for the corresponding estimator we specialize our results to several learning tasks classification ranking matrix completion discuss how to implement a variational approximation in each case and illustrate the good properties of said approximation on real datasets

nonlinear model structure detection using optimum experimental design and orthogonal least squares a very efficient learning algorithm for model subset selection is introduced based on a new composite cost function that simultaneously optimizes the model approximation ability and model adequacy the derived model parameters are estimated via forward orthogonal least squares but the subset selection cost function includes an a optimality design criterion to minimize the variance of the parameter estimates that ensures the adequacy and parsimony of the final model an illustrative example is included to demonstrate the effectiveness of the new approach

conditional mixture model for correlated neuronal spikes analysis of correlated spike trains is a hot topic of research in computational neuroscience a general model of probability distributions for spikes includes too many parameters to be of use in analyzing real data instead we need a simple but powerful generative model for correlated spikes we developed a class of conditional mixture models that includes a number of existing models and analyzed its capabilities and limitations we apply the model to dynamical aspects of neuron pools when hebbian cell assemblies coexist in a pool of neurons the condition is specified by these assemblies such that the probability distribution of spikes is a mixture of those of the component assemblies the probabilities of activation of the hebbian assemblies change dynamically we used this model as a basis for a competitive model governing the states of assemblies

shape and motion from image streams under orthography a factorization method inferring scene geometry and camera motion from a stream of images is possible in principle but is an ill conditioned problem when the objects are distant with respect to their size we have developed a factorization method that can overcome this difficulty by recovering shape and motion under orthography without computing depth as an intermediate step an image stream can be represented by the 2fxp measurement matrix of the image coordinates of p points tracked through f frames we show that under orthographic projection this matrix is of rank 3 based on this observation the factorization method uses the singular value decomposition technique to factor the measurement matrix into two matrices which represent object shape and camera rotation respectively two of the three translation components are computed in a preprocessing stage the method can also handle and obtain a full solution from a partially filled in measurement matrix that may result from occlusions or tracking failures the method gives accurate results and does not introduce smoothing in either shape or motion we demonstrate this with a series of experiments on laboratory and outdoor image streams with and without occlusions

matrix completion with the trace norm learning bounding and transducing trace norm regularization is a widely used and successful approach for collaborative filtering and matrix completion however previous learning guarantees require strong assumptions such as a uniform distribution over the matrix entries in this paper we bridge this gap by providing such guarantees under much milder assumptions which correspond to matrix completion as performed in practice in fact we claim that previous difficulties partially stemmed from a mismatch between the standard learning theoretic modeling of matrix completion and its practical application our results also shed some light on the issue of matrix completion with bounded models which enforce predictions to lie within a certain range in particular we provide experimental and theoretical evidence that such models lead to a modest yet significant improvement

orthogonal least squares learning algorithm for radial basis function networks the radial basis function network offers a viable alternative to the two layer neural network in many applications of signal processing a common learning algorithm for radial basis function networks is based on first choosing randomly some data points as radial basis function centers and then using singular value decomposition to solve for the weights of the network such a procedure has several drawbacks and in particular an arbitrary selection of centers is clearly unsatisfactory the authors propose an alternative learning procedure based on the orthogonal least squares method the procedure chooses radial basis function centers one by one in a rational way until an adequate network has been constructed in the algorithm each selected center maximizes the increment to the explained variance or energy of the desired output and does not suffer numerical ill conditioning problems the orthogonal least squares learning strategy provides a simple and efficient means for fitting radial basis function networks this is illustrated using examples taken from two different signal processing applications

a unified wavelet based modelling framework for non linear system identification the wanarx model structure a new unified modelling framework based on the superposition of additive submodels functional components and wavelet decompositions is proposed for non linear system identification a non linear model which is often represented using a multivariate non linear function is initially decomposed into a number of functional components via the well known analysis of variance anova expression which can be viewed as a special form of the narx non linear autoregressive with exogenous inputs model for representing dynamic input output systems by expanding each functional component using wavelet decompositions including the regular lattice frame decomposition wavelet series and multiresolution wavelet decompositions the multivariate non linear model can then be converted into a linear in the parameters problem which can be solved using least squares type methods an efficient model structure determination approach based upon a forward orthogonal least squares ols algorithm which involves a stepwise o

the time rescaling theorem and its application to neural spike train data analysis measuring agreement between a statistical model and a spike train data series that is evaluating goodness of fit is crucial for establishing the model s validity prior to using it to make inferences about a particular neural system assessing goodness of fit is a challenging problem for point process neural spike train models especially for histogram based models such as perstimulus time histograms psth and rate functions estimated by spike train smoothing the time rescaling theorem is a well known result in probability theory which states that any point process with an integrable conditional intensity function may be transformed into a poisson process with unit rate we describe how the theorem may be used to develop goodness of fit tests for both parametric and histogram based point process models of neural spike trains we apply these tests in two examples a comparison of psth inhomogeneous poisson and inhomogeneous markov interval models of neural spike trains from the supplementary eye field of a macque monkey and a comparison of temporal and spatial smoothers inhomogeneous poisson inhomogeneous gamma and inhomogeneous inverse gaussian models of rat hippocampal place cell spiking activity to help make the logic behind the time rescaling theorem more accessible to researchers in neuroscience we present a proof using only elementary probability theory arguments we also show how the theorem may be used to simulate a general point process model of a spike train our paradigm makes it possible to compare parametric and histogram based neural spike train models directly these results suggest that the time rescaling theorem can be a valuable tool for neural spike train data analysis

a sharp version of bauer fike s theorem in this paper we present a sharp version of bauer fike s theorem we replace the matrix norm with its spectral radius or sign complex spectral radius for diagonalizable matrices 1 norm and norm for non diagonalizable matrices we also give the applications to the pole placement problem and the singular system

information geometric measures for estimation of connection weight under correlated inputs the brain processes information in a highly parallel manner determination of the relationship between neural spikes and synaptic connections plays a key role in the analysis of electrophysiological data information geometry ig has been proposed as a powerful analysis tool for multiple spike data providing useful insights into the statistical interactions within a population of neurons previous work has demonstrated that ig measures can be used to infer the connection weight between two neurons in a neural network this property is useful in neuroscience because it provides a way to estimate learning induced changes in synaptic strengths from extracellular neuronal recordings a previous study has shown however that this property would hold only when inputs to neurons are not correlated since neurons in the brain often receive common inputs this would hinder the application of the ig method to real data we investigated the two neuron ig measures in higher order log linear models to overcome this limitation first we mathematically showed that the estimation of uniformly connected synaptic weight can be improved by taking into account higher order log linear models second we numerically showed that the estimation can be improved for more general asymmetrically connected networks considering the estimated number of the synaptic connections in the brain we showed that the two neuron ig measure calculated by the fourth or fifth order log linear model would provide an accurate estimation of connection strength within approximately a 10 error these studies suggest that the two neuron ig measure with higher order log linear expansion is a robust estimator of connection weight even under correlated inputs providing a useful analytical tool for real multineuronal spike data

low rank data completion with very low sampling rate using newton s method newton s method is a widely applicable and empirically efficient method for finding the solution to a set of equations the recently developed algebraic geometry analyses provide information theoretic bounds on the sampling rate to ensure the existence of a unique completion a remained open question from these works is to retrieve the sampled data when the sampling rate is very close to the mentioned information theoretic bounds this paper is concerned with proposing algorithms to retrieve the sampled data when the sampling rate is too small and close to the mentioned information theoretic bounds hence we propose a new approach for recovering a partially sampled low rank matrix or tensor when the number of samples is only slightly more than the dimension of the corresponding manifold by solving a set of polynomial equations using newton s method in particular we consider low rank matrix completion matrix sensing and tensor completion each observed entry contributes one polynomial equation in terms of the factors in the rank factorization of the data by exploiting the specific structures of the resulting set of polynomial equations we analytically characterize the convergence regions of the newton s method for matrix completion and matrix sensing through extensive numerical results we show that the proposed approach outperforms the well known methods such as nuclear norm minimization and alternating minimization in terms of the success rate of data recovery noiseless case and peak signal to noise ratio noisy case especially when the sampling rate is very low

information geometric measure for neural spikes this study introduces information geometric measures to analyze neural firing patterns by taking not only the second order but also higher order interactions among neurons into account information geometry provides useful tools and concepts for this purpose including the orthogonality of coordinate parameters and the pythagoras relation in the kullback leibler divergence based on this orthogonality we show a novel method for analyzing spike firing patterns by decomposing the interactions of neurons of various orders as a result purely pairwise triple wise and higher order interactions are singled out we also demonstrate the benefits of our proposal by using several examples

generalized reduced rank latent factor regression for high dimensional tensor fields and neuroimaging genetic applications we propose a generalized reduced rank latent factor regression model grrlf for the analysis of tensor field responses and high dimensional covariates the model is motivated by the need from imaging genetic studies to identify genetic variants that are associated with brain imaging phenotypes often in the form of high dimensional tensor fields grrlf identifies from the structure in the data the effective dimensionality of the data and then jointly performs dimension reduction of the covariates dynamic identification of latent factors and nonparametric estimation of both covariate and latent response fields after accounting for the latent and covariate effects grllf performs a nonparametric test on the remaining factor of interest grrlf provides a better factorization of the signals compared with common solutions and is less susceptible to overfitting because it exploits the effective dimensionality the generality and the flexibility of grrlf also allow various statistical models to be handled in a unified framework and solutions can be efficiently computed within the field of neuroimaging it improves the sensitivity for weak signals and is a promising alternative to existing approaches the operation of the framework is demonstrated with both synthetic datasets and a real world neuroimaging example in which the effects of a set of genes on the structure of the brain at the voxel level were measured and the results compared favorably with those from existing approaches

influence of external inputs and asymmetry of connections on information geometric measures involving up to ten neuronal interactions the investigation of neural interactions is crucial for understanding information processing in the brain recently an analysis method based on information geometry ig has gained increased attention and the property of the pairwise ig measure has been studied extensively in relation to the two neuron interaction however little is known about the property of ig measures involving more neuronal interactions in this study we systematically investigated the influence of external inputs and the asymmetry of connections on the ig measures in cases ranging from 1 neuron to 10 neuron interactions first the analytical relationship between the ig measures and external inputs was derived for a network of 10 neurons with uniform connections our results confirmed that the single and pairwise ig measures were good estimators of the mean background input and of the sum of the connection weights respectively for the ig measures involving 3 to 10 neuronal interactions we found that the influence of external inputs was highly nonlinear second by computer simulation we extended our analytical results to asymmetric connections for a network of 10 neurons the simulation showed that the behavior of the ig measures in relation to external inputs was similar to the analytical solution obtained for a uniformly connected network when the network size was increased to 1000 neurons the influence of external inputs almost disappeared this result suggests that all ig measures from 1 neuron to 10 neuron interactions are robust against the influence of external inputs in addition we investigated how the strength of asymmetry influenced the ig measures computer simulation of a 1000 neuron network showed that all the ig measures were robust against the modulation of the asymmetry of connections our results provide further support for an information geometric approach and will provide useful insights when these ig measures are applied to real experimental spike data

on the rank minimization problem over a positive semidefinite linear matrix inequality we consider the problem of minimizing the rank of a positive semidefinite matrix subject to the constraint that an affine transformation of it is also positive semidefinite our method for solving this problem employs ideas from the ordered linear complementarity theory and the notion of the least element in a vector lattice this problem is of importance in many contexts for example in feedback synthesis problems and such an example is also provided