distribution estimation consistent in total variation and in two types of information divergence the problem of the nonparametric estimation of a probability distribution is considered from three viewpoints the consistency in total variation the consistency in information divergence and consistency in reversed order information divergence these types of consistencies are relatively strong criteria of convergence and a probability distribution cannot be consistently estimated in either type of convergence without any restrictions on the class of probability distributions allowed histogram based estimators of distribution are presented which under certain conditions converge in total variation in information divergence and in reversed order information divergence to the unknown probability distribution some a priori information about the true probability distribution is assumed in each case as the concept of consistency in information divergence is stronger than that of convergence in total variation additional assumptions are imposed in the cases of informational divergences

strong data processing inequalities for input constrained additive noise channels this paper quantifies the intuitive observation that adding noise reduces available information by means of nonlinear strong data processing inequalities consider the random variables w to x to y forming a markov chain where y x z with x and z real valued independent and x bounded in l p norm it is shown that i w y le f i i w x with f i t whenever t 0 if and only if z has a density whose support is not disjoint from any translate of itself a related question is to characterize for what couplings w x the mutual information i w y is close to maximum possible to that end we show that in order to saturate the channel i e for i w y to approach capacity it is mandatory that i w x to infty under suitable conditions on the channel a key ingredient for this result is a deconvolution lemma which shows that postconvolution total variation distance bounds the preconvolution kolmogorov smirnov distance explicit bounds are provided for the special case of the additive gaussian noise channel with quadratic cost constraint these bounds are shown to be order optimal for this case simplified proofs are provided leveraging gaussian specific tools such as the connection between information and estimation i mmse and talagrand s information transportation inequality

on the support of mse optimal fixed rate scalar quantizers this paper determines how the support regions of optimal and asymptotically optimal fixed rate scalar quantizers with respect to mean squared error depend on the number of quantization points n and the probability density of the variable being quantized it shows that for asymptotic optimality it is necessary and sufficient that the support region grow fast enough that the outer or overload distortion decreases as o 1 n sup 2 formulas are derived for the minimal support of asymptotically optimal quantizers for generalized gamma densities including gaussian and laplacian interestingly these turn out to be essentially the same as for the support of optimal fixed rate uniform scalar quantizers heuristic arguments are then used to find closed form estimates for the support of truly optimal quantizers for generalized gamma densities these are found to be more accurate than the best prior estimates as computed by numerical algorithms they demonstrate that the support of an optimal quantizer is larger than the minimal asymptotically optimal support by a factor depending on the density but not n and that the outer distortion of optimal quantizers decreases as 1 n sup 3

on brlek reutenauer conjecture brlek and reutenauer conjectured that any infinite word u with language closed under reversal satisfies the equality 2d u email protected n 0 t u n in which d u denotes the defect of u and t u n denotes c u n 1 c u n 2 p u n 1 p u n where c u and p u are the factor and palindromic complexity of u respectively brlek and reutenauer verified their conjecture for periodic infinite words using their result we prove the conjecture for uniformly recurrent words moreover we summarize results and some open problems related to defects which may be useful for the proof of the brlek reutenauer conjecture in full generality

nonparametric multivariate density estimation a comparative study the paper algorithmically and empirically studies two major types of nonparametric multivariate density estimation techniques where no assumption is made about the data being drawn from any of known parametric families of distribution the first type is the popular kernel method and several of its variants which uses locally tuned radial basis e g gaussian functions to interpolate the multidimensional density the second type is based on an exploratory projection pursuit technique which interprets the multidimensional density through the construction of several 1d densities along highly interesting projections of multidimensional data performance evaluations using training data from mixture gaussian and mixture cauchy densities are presented the results show that the curse of dimensionality and the sensitivity of control parameters have a much more adverse impact on the kernel density estimators than on the projection pursuit density estimators

a survey of the theory of source coding with a fidelity criterion the purpose of this paper is threefold 1 to acquaint the reader with the types of problems that have been considered in the area of source coding with a fidelity criterion 2 to survey results that have been obtained on these problems and 3 to outline future research trends in the area

on using prototype reduction schemes to optimize kernel based nonlinear subspace methods abstract the subspace method of pattern recognition is a classification technique in which pattern classes are specified in terms of linear subspaces spanned by their respective class based basis vectors to overcome the limitations of the linear methods kernel based nonlinear subspace kns methods have been recently proposed in the literature in kns the kernel principal component analysis kpca has been employed to get principal components not in an input space but in a high dimensional space where the components of the space are nonlinearly related to the input variables the length of projections onto the basis vectors in the kpca are computed using a kernel matrix k whose dimension is equivalent to the number of sample data points clearly this is problematic especially for large data sets in this paper we suggest a computationally superior mechanism to solve the problem rather than define the matrix k with the whole data set and compute the principal components we propose that the data be reduced into a smaller representative subset using a prototype reduction scheme prs since a prs has the capability of extracting vectors that satisfactorily represent the global distribution structure we demonstrate that data points which are ineffective in the classification can be eliminated to obtain a reduced kernel matrix k without degrading the performance our experimental results demonstrate that the proposed mechanism dramatically reduces the computation time without sacrificing the classification accuracy for samples involving real life data sets as well as artificial data sets the results especially demonstrate the computational advantage for large data sets such as those involved in data mining and text categorization applications

on using prototype reduction schemes to enhance the computation of volume based inter class overlap measures in most pattern recognition pr applications it is advantageous if the accuracy or error rate of the classifier can be evaluated or bounded prior to testing it in a real life setting it is also well known that if the two class conditional distributions have a large overlapping volume almost all the available work on overlapping of classes deals with the case when there are only two classes the classification accuracy is poor this is because if we intend to use the classification accuracy as a criterion for evaluating a pr system the points within the overlapping volume tend to lead to maximal misclassification unfortunately the computation of the indices which quantify the overlapping volume is expensive in this vein we propose a strategy of using a prototype reduction scheme prs to approximately but quickly compute the latter in this paper we demonstrate first of all that this is an extremely expedient proposition indeed we show that by completely discarding we are not aware of any reported scheme which discards irrelevant sample training points and which simultaneously attains to an almost comparable accuracy the points not included by the prs we can obtain a reduced set of sample points using which in turn the measures for the overlapping volume can be computed the value of the corresponding figures is comparable to those obtained with the original training set i e the one which considers all the data points even though the computations required to obtain the prototypes and the corresponding measures are significantly less the proposed method has been rigorously tested on artificial and real life datasets and the results obtained are in our opinion quite impressive sometimes faster by two orders of magnitude

multiresolution estimates of classification complexity in this paper we study two measures of classification complexity based on feature space partitioning purity and neighborhood separability the new measures of complexity are compared with probabilistic distance measures and a number of other nonparametric estimates of classification complexity on a total of 10 databases from the university of california irvine uci repository

on pairs of f divergences and their joint range we compare two f divergences and prove that their joint range is the convex hull of the joint range for distributions supported on only two points the proofs use various methods from topology and convex analysis

refinements of pinsker s inequality let v and d denote respectively total variation and divergence we study lower bounds of d with v fixed the theoretically best i e largest lower bound determines a function l l v vajda s 1970 tight lower bound the main result is an exact parametrization of l this leads to taylor polynomials which are lower bounds for l and thereby to extensions of the classical pinsker 1960 inequality which has numerous applications cf pinsker and followers

nonparametric regression estimation using penalized least squares we present multivariate penalized least squares regression estimates we use vapnik chervonenkis see statistical learning theory 1998 theory and bounds on the covering numbers to analyze convergence of the estimates we show strong consistency of the truncated versions of the estimates without any conditions on the underlying distribution

high rate vector quantization for the neyman pearson detection of correlated processes this paper investigates the effect of quantization on the performance of the neyman pearson test it is assumed that a sensing unit observes samples of a correlated stationary ergodic multivariate process each sample is passed through an n point quantizer and transmitted to a decision device which performs a binary hypothesis test for any false alarm level it is shown that the miss probability of the neyman pearson test converges to zero exponentially as the number of samples tends to infinity assuming that the observed process satisfies certain mixing conditions the main contribution of this paper is to provide a compact closed form expression of the error exponent in the high rate regime i e when the number n of quantization levels tends to infinity generalizing previous results of gupta and hero to the case of nonindependent observations if d represents the dimension of one sample it is proved that the error exponent converges at rate n2 d to the one obtained in the absence of quantization as an application relevant high rate quantization strategies which lead to a large error exponent are determined numerical results indicate that the proposed quantization rule can yield better performance than existing ones in terms of detection error

least squares quantization in pcm it has long been realized that in pulse code modulation pcm with a given ensemble of signals to handle the quantum values should be spaced more closely in the voltage regions where the signal amplitude is more likely to fall it has been shown by panter and dite that in the limit as the number of quanta becomes infinite the asymptotic fractional density of quanta per unit voltage should vary as the one third power of the probability density per unit voltage of signal amplitudes in this paper the corresponding result for any finite number of quanta is derived that is necessary conditions are found that the quanta and associated quantization intervals of an optimum finite quantization scheme must satisfy the optimization criterion used is that the average quantization noise power be a minimum it is shown that the result obtained here goes over into the panter and dite result as the number of quanta become large the optimum quautization schemes for 2 b quanta b 1 2 cdots 7 are given numerically for gaussian and for laplacian distribution of signal amplitudes

on the asymptotic distribution of the errors in vector quantization in a recent paper lee and neuhoff see ibid vol 42 no 2 p446 60 1996 found an asymptotic formula for the distribution of the length of the errors produced by a vector quantizer with many quantization points this distribution depends on the source probability density the quantizer point density and the quantizer shape profile the latter characterizes the shapes of the quantization cells as a function of position the purpose of this paper is to give a rigorous derivation of this formula by identifying precise conditions under which it is shown that if a sequence of vector quantizers with a given dimension and an increasing number of points has specific point densities and specific shape profiles converging to a model point density and a model shape profile respectively then the distribution of the length of the quantization error suitably normalized converges to the aforementioned formula with the model point density and the model shape profile substituted

k vectors an alternating minimization algorithm for learning regression functions the k vectors algorithm for learning regression functions proposed here is akin to the well known k means algorithm both algorithms partition the feature space but unlike the k means algorithm the k vectors algorithm aims to reconstruct the response rather than the feature the partitioning rule of the algorithm is based on maximizing the correlation inner product of the feature vector with a set of k vectors and generates polyhedral cells similar to the ones generated by the nearest neighbor rule of the k means algorithm similarly to k means the learning algorithm alternates between two types of steps in the first type of steps k labels are determined via a centroid type rule in the response space which uses a surrogate hinge type loss function to the mean squared error loss function in the second type of steps the k vectors which determine the partition are updated according to a multiclass classification rule in the spirit of support vector machines it is proved that both steps of the algorithm only require solving convex optimization problems and that the algorithm is empirically consistent as the length of the training sequence increases to infinity fixed points of the empirical version of the algorithm tend to fixed points of the population version of the algorithm learnability of the predictor class posit by the algorithm is also established

on divergences and informations in statistics and information theory the paper deals with the f divergences of csiszar generalizing the discrimination information of kullback the total variation distance the hellinger divergence and the pearson divergence all basic properties of f divergences including relations to the decision errors are proved in a new manner replacing the classical jensen inequality by a new generalized taylor expansion of convex functions some new properties are proved too e g relations to the statistical sufficiency and deficiency the generalized taylor expansion also shows very easily that all f divergences are average statistical informations differences between prior and posterior bayes errors mutually differing only in the weights imposed on various prior distributions the statistical information introduced by de groot and the classical information of shannon are shown to be extremal cases corresponding to alpha 0 and alpha 1 in the class of the so called arimoto alpha informations introduced in this paper for 0 alpha 1 by means of the arimoto alpha entropies some new examples of f divergences are introduced as well namely the shannon divergences and the arimoto alpha divergences leading for alphauarr1 to the shannon divergences square roots of all these divergences are shown to be metrics satisfying the triangle inequality the last section introduces statistical tests and estimators based on the minimal f divergence with the empirical distribution achieved in the families of hypothetic distributions for the kullback divergence this leads to the classical likelihood ratio test and estimator

quantizing for minimum average misclassification risk in pattern classification a decision rule is a labeled partition of the observation space where labels represent classes a way to establish a decision rule is to attach a label to each code vector of a vector quantizer vq when a labeled vq is adopted as a classifier we have to design it in such a way that high classification performance is obtained by a given number of code vectors in this paper we propose a learning algorithm which optimizes the position of labeled code vectors in the observation space under the minimum average misclassification risk criterion

termination and continuity of greedy growing for tree structured vector quantizers tree structured vector quantizers tsvq provide a computationally efficient variable rate method of compressing vector valued data in applications the problem of designing a tsvq from empirical training data is critical greedy growing algorithms are a common and effective approach to the design problem they are recursive procedures that produce a tsvq one node at a time by optimizing a simple splitting criterion at each step while unsupervised greedy growing algorithms are well understood from an experimental point of view there has been little theory to support their use or to examine their behavior on large training sets the authors present a rigorous analysis of a greedy growing algorithm proposed by riskin 1990 riskin and gray 1991 and balakrishnan 1991 the first part of the paper is a description of the algorithm and an examination of its asymptotic behavior as it applies to a fixed absolutely continuous distribution the second part of the paper establishes the structural consistency of the algorithm with respect to a convergent sequence of distributions as an application the authors obtain results concerning the large sample empirical behavior of the algorithm when it is applied to stationary ergodic training vectors

a vq based blind image restoration algorithm learning based algorithms for image restoration and blind image restoration are proposed such algorithms deviate from the traditional approaches in this area by utilizing priors that are learned from similar images original images and their degraded versions by the known degradation operator restoration problem are utilized for designing the vq codebooks the codevectors are designed using the blurred images for each such vector the high frequency information obtained from the original images is also available during restoration the high frequency information of a given degraded image is estimated from its low frequency information based on the codebooks for the blind restoration problem a number of codebooks are designed corresponding to various versions of the blurring function given a noisy and blurred image one of the codebooks is chosen based on a similarity measure therefore providing the identification of the blur to make the restoration process computationally efficient the principal component analysis pca and vq nearest neighbor approaches are utilized simulation results are presented to demonstrate the effectiveness of the proposed algorithms

complexity and palindromic defect of infinite words in this note we state a conjecture and prove it in the periodic case which is an equality relating the number of factors and palindromic factors of infinite words this equality establishes a link between two inequalities one due to droubay justin and pirillo and the other to balazi masakova and pelantova by means of the palindromic defect

on quantizer distortion and the upper bound for exponential entropy a sharp upper bound is derived for the exponential entropy in the class of absolutely continuous distributions with specific standard deviation and an exact description of the extremal distributions this result is interpreted as determining the least favorable cases for certain methods of quantization of analog sources it is known that for a large class of quantizers both zero memory and vector the rth power distortion as well as some other distortion criteria are bounded below by a constant depending on r multiplied by a certain integral of the source s probability density it is pointed out that this bound can be rewritten in terms of the exponential entropy the exponential entropy measures the quantitative extent or range of the source distribution this fact gives a physical interpretation of the indicated limits of quantizer performance further elucidated by the main result

combining image compression and classification using vector quantization we describe a method of combining classification and compression into a single vector quantizer by incorporating a bayes risk term into the distortion measure used in the quantizer design algorithm once trained the quantizer can operate to minimize the bayes risk weighted distortion measure if there is a model providing the required posterior probabilities or it can operate in a suboptimal fashion by minimizing the squared error only comparisons are made with other vector quantizer based classifiers including the independent design of quantization and minimum bayes risk classification and kohonen s lvq a variety of examples demonstrate that the proposed method can provide classification ability close to or superior to learning vq while simultaneously providing superior compression performance

consistency of data driven histogram methods for density estimation and classification we present general sufficient conditions for the almost surel1 consistency of histogram density estimates based on data dependent partitions analogous conditions guarantee the almost sure risk consistency of histogram classification schemes based on data dependent partitions multivariate data is considered throughout in each case the desired consistency requires shrinking cells subexponential growth of a combinatorial complexity measure and sub linear growth of the number of cells it is not required that the cells of every partition be rectangles with sides paralles to the coordinate axis or that each cell contain a minimum number of points no assumptions are made concerning the common distribution of the training vectors we apply the results to establish the consistency of several known partitioning estimates including the kn spacing density estimate classifiers based on statistically equivalent blocks and classifiers based on multivariate clustering schemes

factor versus palindromic complexity of uniformly recurrent infinite words we study the relation between the palindromic and factor complexity of infinite words we show that for uniformly recurrent words one has p n p n 1 email protected n 2 for all email protected n for a large class of words this is a better estimate of the palindromic complexity in terms of the factor complexity than the one presented in j p allouche m baake j cassaigne d damanik palindrome complexity theoret comput sci 292 2003 9 31 we provide several examples of infinite words for which our estimate reaches its upper bound in particular we derive an explicit prescription for the palindromic complexity of infinite words coding r interval exchange transformations if the permutation p connected with the transformation is given by p k r 1 k for all k then there is exactly one palindrome of every even length and exactly r palindromes of every odd length

extensions of rich words a word w is rich if it has w 1 many distinct palindromic factors including the empty word this article contains several results about rich words particularly related to extending them a word w can be eventually extended richly in n ways if there exists a finite word u and n distinct letters a alph w such that wua is rich we will prove that every non unary rich word can be eventually extended richly in at least two different ways but not always in three or more ways we will also prove that every rich word can be extended to both periodic and aperiodic infinite rich words the defect of a finite word w is defined by d w w 1 pal w this concept has been studied in various papers here we will define a new concept infinite defect for a finite word w the definition is d w min d z zis an infinite word which has factorw we will show that the infinite defect of a finite word is always finite and give some upper bounds for it the difference between defect and infinite defect is also investigated we will also give an upper and a lower bound for the number of rich words a new class of words two dimensional rich words is also introduced

a note on the computation of optimal minimum mean square error quantizers this paper considers the problems associated with computing optimal minimum mean square error quantizers most computational methods in current use are iterative these iterative schemes are extremely sensitive to initial conditions various methods of obtaining good initial conditions are presented and discussed

on the renyi divergence joint range of relative entropies and a channel coding theorem this paper starts by considering the minimization of the renyi divergence subject to a constraint on the total variation distance based on the solution of this optimization problem the exact locus of the points bigl d q p 1 d q p 2 bigr is determined when p 1 p 2 and q are arbitrary probability measures which are mutually absolutely continuous and the total variation distance between p 1 and p 2 is not below a given value it is further shown that all the points of this convex region are attained by probability measures which are defined on a binary alphabet this characterization yields a geometric interpretation of the minimal chernoff information subject to a constraint on the variational distance this paper also derives an exponential upper bound on the performance of binary linear block codes or code ensembles under maximum likelihood decoding its derivation relies on the gallager bounding technique and it reproduces the shulman feder bound as a special case the bound is expressed in terms of the renyi divergence from the normalized distance spectrum of the code or the average distance spectrum of the ensemble to the binomially distributed distance spectrum of the capacity achieving ensemble of random block codes this exponential bound provides a quantitative measure of the degradation in performance of binary linear block codes or code ensembles as a function of the deviation of their distance spectra from the binomial distribution an efficient use of this bound is considered

vector quantization technique for nonparametric classifier design an effective data reduction technique based on vector quantization is introduced for nonparametric classifier design two new nonparametric classifiers are developed and their performance is evaluated using various examples the new methods maintain a classification accuracy that is competitive with that of classical methods but at the same time yields very high data reduction rates

a note on differentiable palindromes abstract we give a characterization of the palindromes in a class of infinite words over 1 2 related to the kolakoski word k this characterization based on the left palindromic closure of all prefixes of k is obtained by using a bijection between the class of right infinite words over and a class of words over the same alphabet and reveals the first link between the existence of some palindromes and the recurrence of k indeed the existence of arbitrarily long palindromes implies the recurrence of k and a stronger assumption implies the closure of the set of its factors by permutation of the letters in

the reduced parzen classifier the parzen density estimate is known to be an effective tool for estimating the bayes error given a set of training samples from the class distributions an algorithm is developed to select a given number of representative samples whose parzen density estimate closely matches that of the entire sample set using this reduced representative set a piecewise quadratic classifier which provides nearly optimal performance is designed

on the support of fixed rate minimum mean squared error scalar quantizers for a laplacian source this correspondence shows that the support growth of a fixed rate optimum minimum mean squared error scalar quantizer for a laplacian density is logarithmic with the number of quantization points specifically it is shown that for a unit variance laplacian density the ratio of the support determining threshold of an optimum quantizer to 3 spl radic 2lnn 2 converges to 1 as the number n of quantization points grows also derived is a limiting upper bound that says that the support determining threshold cannot exceed the logarithmic growth by more than a small constant e g 0 0669 these results confirm the logarithmic growth of the optimum support that has previously been derived heuristically

self consistency and principal component analysis abstract i examine the self consistency of a principal component axis that is when a distribution is centered about a principal component axis a principal component axis of a random vector x is self consistent if each point on the axis corresponds to the mean of x given that x projects orthogonally onto that point a large class of symmetric multivariate distributions are examined in terms of self consistency of principal component subspaces elliptical distributions are characterized by the preservation of self consistency of principal component axes after arbitrary linear transformations a lack of fit test is proposed that tests for self consistency of a principal axis the test is applied to two real datasets

geodesic entropic graphs for dimension and entropy estimation in manifold learning in the manifold learning problem one seeks to discover a smooth low dimensional surface i e a manifold embedded in a higher dimensional linear vector space based on a set of measured sample points on the surface in this paper we consider the closely related problem of estimating the manifold s intrinsic dimension and the intrinsic entropy of the sample points specifically we view the sample points as realizations of an unknown multivariate density supported on an unknown smooth manifold we introduce a novel geometric approach based on entropic graph methods although the theory presented applies to this general class of graphs we focus on the geodesic minimal spanning tree gmst to obtaining asymptotically consistent estimates of the manifold dimension and the re spl acute nyi spl alpha entropy of the sample density on the manifold the gmst approach is striking in its simplicity and does not require reconstruction of the manifold or estimation of the multivariate density of the samples the gmst method simply constructs a minimal spanning tree mst sequence using a geodesic edge matrix and uses the overall lengths of the msts to simultaneously estimate manifold dimension and entropy we illustrate the gmst approach on standard synthetic manifolds as well as on real data sets consisting of images of faces

vanishing distortion and shrinking cells we establish an asymptotic connection between vanishing rth power distortion and shrinking cell diameters for vector quantizers with convex cells as a consequence a number of shrinking cell conditions may be easily verified by showing that the quantizers in question have distortion tending to zero this also plays an important role in the asymptotic analysis of a common greedy growing scheme for tree structured vector quantizers

on using prototype reduction schemes to optimize locally linear reconstruction methods this paper concerns the use of prototype reduction schemes prs to optimize the computations involved in typical k nearest neighbor k nn rules these rules have been successfully used for decades in statistical pattern recognition pr 1 15 applications and are particularly effective for density estimation classification and regression because of the known error bounds that they possess for a given data point of unknown identity the k nn possesses the phenomenon that it combines the information about the samples from a priori target classes values of selected neighbors to predict the target class of the tested sample or to estimate the density function value of the given queried sample recently an implementation of the k nn named as the locally linear reconstruction llr 2 has been proposed the salient and brilliant feature of the latter is that by invoking a quadratic optimization process it is capable of systematically setting model parameters such as the number of neighbors specified by the parameter k and the weights however the llr takes more time than other conventional methods when it has to be applied to classification tasks to overcome this problem we propose a strategy of using a prs to efficiently compute the optimization problem in this paper we demonstrate first of all that by completely discarding the points not included by the prs we can obtain a reduced set of sample points using which in turn the quadratic optimization problem can be computed far more expediently the values of the corresponding indices are comparable to those obtained with the original training set i e the one which considers all the data points even though the computations required to obtain the prototypes and the corresponding classification accuracies are noticeably less the proposed method has been tested on artificial and real life data sets and the results obtained are very promising and could have potential in pr applications

bregman divergence bounds and universality properties of the logarithmic loss a loss function measures the discrepancy between the true values and their estimated fits for a given instance of data in classification problems a loss function is said to be proper if a minimizer of the expected loss is the true underlying probability we show that for binary classification the divergence associated with smooth proper and convex loss functions is upper bounded by the kullback leibler kl divergence to within a normalization constant this implies that by minimizing the logarithmic loss associated with the kl divergence we minimize an upper bound to any choice of loss from this set as such the logarithmic loss is universal in the sense of providing performance guarantees with respect to a broad class of accuracy measures importantly this notion of universality is not problem specific enabling its use in diverse applications including predictive modeling data clustering and sample complexity analysis generalizations to arbitary finite alphabets are also developed the derived inequalities extend several well known f divergence results

analysis of a complexity based pruning scheme for classification trees a complexity based pruning procedure for classification trees is described and bounds on its finite sample performance are established the procedure selects a subtree of a possibly random initial tree in order to minimize a complexity penalized measure of empirical risk the complexity assigned to a subtree is proportional to the square root of its size two cases are considered in the first the growing and pruning data sets are identical and in the second they are independent using the performance bound the bayes risk consistency of pruned trees obtained via the procedure is established when the sequence of initial trees satisfies suitable geometric and structural constraints the pruning method and its analysis are motivated by work on adaptive model selection using complexity regularization

automatic pattern recognition a study of the probability of error a test sequence is used to select the best rule from a class of discrimination rules defined in terms of the training sequence the vapnik chervonenkis and related inequalities are used to obtain distribution free bounds on the difference between the probability of error of the selected rule and the probability of error of the best rule in the given class the bounds are used to prove the consistency and asymptotic optimality for several popular classes including linear discriminators nearest neighbor rules kernel based rules histogram rules binary tree classifiers and fourier series classifiers in particular the method can be used to choose the smoothing parameter in kernel based rules to choose k in the k nearest neighbor rule and to choose between parametric and nonparametric rules

asymptotic distribution of the errors in scalar and vector quantizers high rate or asymptotic quantization theory has found formulas for the average squared length more generally the qth moment of the length of the error produced by various scalar and vector quantizers with many quantization points in contrast this paper finds an asymptotic formula for the probability density of the length of the error and in certain special cases for the probability density of the multidimensional error vector itself the latter can be used to analyze the distortion of two stage vector quantization the former permits one to learn about the point density and cell shapes of a quantizer from a histogram of quantization error lengths histograms of the error lengths in simulations agree well with the derived formulas also presented are a number of properties of the error density including the relationship between the error density the point density and the cell shapes the fact that its qth moment equals bennett s integral a formula for the average distortion of a scalar or vector quantizer and the fact that for stationary sources the marginals of the multidimensional error density of an optimal vector quantizer with large dimension are approximately i i d gaussian

episturmian words and some constructions of de luca and rauzy abstract in this paper we study infinite episturmian words which are a natural generalization of sturmian words to an arbitrary alphabet a characteristic property is they are closed under reversal and have at most one right special factor of each length they are first obtained by a construction due to de luca which utilizes the palindrome closure they can also be obtained by the way of extended rauzy rules

trimmed k means an attempt to robustify quantizers a class of procedures based on impartial trimming self determined by the data is introduced with the aim of robustifying k means hence the associated clustering analysis we include a detailed study of optimal regions showing that only nonpathological regions can arise from impartial trimming procedures the asymptotic results provided in the paper focus on strong consistency of the suggested methods under widely general conditions a section is devoted to exploring the performance of the procedure to detect anomalous data in simulated data sets

an analytic function describing the error performance of optimum quantizers corresp for optimizing or analyzing digital source coding systems a model for the mean square quantizing error as a function of the number of levels is necessary an analytic approximation function is presented which is valid for a wide range of important probability density functions

estimating random integrals from noisy observations sampling designs and their performance the problem of estimating a weighted average of a random process from noisy observations at a finite number of sampling points is considered the performance of sampling designs with optimal or suboptimal but easily computable estimator coefficients is studied several examples and special cases are studied including additive independent noise nonlinear distortion with noise and quantization noise

bennett s integral for vector quantizers this paper extends bennett s 1948 integral from scalar to vector quantizers giving a simple formula that expresses the rth power distortion of a many point vector quantizer in terms of the number of points point density function inertial profile and the distribution of the source the inertial profile specifies the normalized moment of inertia of quantization cells as a function of location the extension is formulated in terms of a sequence of quantizers whose point density and inertial profile approach known functions as the number of points increase precise conditions are given for the convergence of distortion suitably normalized to bennett s integral previous extensions did not include the inertial profile and consequently provided only bounds or applied only to quantizers with congruent cells such as lattice and optimal quantizers the new version of bennett s integral provides a framework for the analysis of suboptimal structured vector quantizers it is shown how the loss in performance of such quantizers relative to optimal unstructured ones can be decomposed into point density and cell shape losses as examples these losses are computed for product quantizers and used to gain further understanding of the performance of scalar quantizers applied to stationary memoryless sources and of transform codes applied to gaussian sources with memory it is shown that the short coming of such quantizers is that they must compromise between point density and cell shapes

sharp inequalities for f divergences f divergences are a general class of divergences between probability measures which include as special cases many commonly used divergences in probability mathematical statistics and information theory such as kullback leibler divergence chi squared divergence squared hellinger distance total variation distance and so on in this paper we study the problem of maximizing or minimizing an f divergence between two probability measures subject to a finite number of constraints on other f divergences we show that these infinite dimensional optimization problems can all be reduced to optimization problems over small finite dimensional spaces which are tractable our results lead to a comprehensive and unified treatment of the problem of obtaining sharp inequalities between f divergences we demonstrate that many of the existing results on inequalities between f divergences can be obtained as special cases of our results we also improve on some existing non sharp inequalities

bayes risk weighted vector quantization with posterior estimation for image compression and classification classification and compression play important roles in communicating digital information their combination is useful in many applications including the detection of abnormalities in compressed medical images in view of the similarities of compression and low level classification it is not surprising that there are many similar methods for their design because some of these methods are useful for designing vector quantizers it seems natural that vector quantization vq is explored for the combined goal we investigate several vq based algorithms that seek to minimize both the distortion of compressed images and errors in classifying their pixel blocks these algorithms are investigated with both full search and tree structured codes we emphasize a nonparametric technique that minimizes both error measures simultaneously by incorporating a bayes risk component into the distortion measure used for the design and encoding we introduce a tree structured posterior estimator to produce the class posterior probabilities required for the bayes risk computation in this design for two different image sources we demonstrate that this system provides superior classification while maintaining compression close or superior to that of several other vq based designs including kohonen s 1992 learning vector quantizer and a sequential quantizer classifier design

rich sturmian and trapezoidal words in this paper we explore various interconnections between rich words sturmian words and trapezoidal words rich words first introduced by the second and third authors together with j justin and s widmer constitute a new class of finite and infinite words characterized by having the maximal number of palindromic factors every finite sturmian word is rich but not conversely trapezoidal words were first introduced by the first author in studying the behavior of the subword complexity of finite sturmian words unfortunately this property does not characterize finite sturmian words in this note we show that the only trapezoidal palindromes are sturmian more generally we show that sturmian palindromes can be characterized either in terms of their subword complexity the trapezoidal property or in terms of their palindromic complexity we also obtain a similar characterization of rich palindromes in terms of a relation between palindromic complexity and subword complexity

high resolution source coding for non difference distortion measures multidimensional companding entropy coded vector quantization is studied using high resolution multidimensional companding over a class of nondifference distortion measures for distortion measures which are locally quadratic a rigorous derivation of the asymptotic distortion and entropy coded rate of multidimensional companders is given along with conditions for the optimal choice of the compressor function this optimum compressor when it exists depends on the distortion measure but not on the source distribution the rate distortion performance of the companding scheme is studied using an asymptotic expression for the rate distortion function which parallels the shannon lower bound for difference distortion measures it is proved that the high resolution performance of the scheme is arbitrarily close to the rate distortion limit for large quantizer dimensions if the compressor function and the lattice quantizer used in the companding scheme are optimal extending an analogous statement for entropy coded lattice quantization and mse distortion the companding approach is applied to obtain a high resolution quantizing scheme for noisy sources

asymptotic theory of greedy approximations to minimal k point random graphs let spl chi sub n x sub 1 x sub n be an independent and identically distributed i i d sample having multivariate distribution p we derive almost sure a s limits for the power weighted edge weight function of greedy approximations to a class of minimal graphs spanning k of the n samples the class includes minimal k point graphs constructed by the partitioning method of ravi sundaram marathe rosenkrantz and ravi see proc 5th annu acm siam symp discrete algorithms arlington va p 546 55 1994 where the edge weight function satisfies the quasi additive property of redmond and yukich see ann appl probab vol 4 no 4 p 1057 73 1994 in particular this includes greedy approximations to the k point minimal spanning tree k mst steiner tree k st and the traveling salesman problem k tsp an expression for the influence function of the minimal weight function is given which characterizes the asymptotic sensitivity of the graph weight to perturbations in the underlying distribution the influence function takes a form which indicates that the k point minimal graph in d 1 dimensions has robustness properties in r sup d which are analogous to those of rank order statistics in one dimension a direct result of our theory is that the log weight of the k point minimal graph is a consistent nonparametric estimate of the renyi entropy of the distribution p possible applications of this work include analysis of random communication network topologies estimation of the mixing coefficient in spl epsiv contaminated mixture models outlier discrimination and rejection clustering and pattern recognition robust nonparametric regression two sample matching and image registration

on finite block length quantization distortion we investigate the upper and lower bounds on the quantization distortions for independent and identically distributed continuous sources in the finite block length regime we derive a lower bound on the quantization distortion as well as an upper bound on the quantization distortion based on random quantization codebooks moreover we apply the obtained bounds to the continuous gaussian source for the gaussian source we propose a computationally tractable method to numerically compute the upper and lower bounds for both bounded and unbounded quantization codebooks numerical results show that the gap between the upper and lower bounds is small for the block length of several hundreds and the upper and lower bounds are tighter than those proposed in the existing works

tree structured nonlinear signal modeling and prediction we develop a regression tree approach to identification and prediction of signals that evolve according to an unknown nonlinear state space model in this approach a tree is recursively constructed that partitions the p dimensional state space into a collection of piecewise homogeneous regions utilizing a 2 sup p ary splitting rule with an entropy based node impurity criterion on this partition the joint density of the state is approximately piecewise constant leading to a nonlinear predictor that nearly attains minimum mean square error this process decomposition is closely related to a generalized version of the thresholded ar signal model art which we call piecewise constant ar pcar we illustrate the method for two cases where classical linear prediction is ineffective a chaotic double scroll signal measured at the output of a chua type electronic circuit and a second order art model we show that the prediction errors are comparable with the nearest neighbor approach to nonlinear prediction but with greatly reduced complexity

a simple class of asymptotically optimal quantizers a simple class of quantizers is introduced which are asymptotically optimal as the number of quantization levels increases to infinity with respect to a mean r th power absolute error distortion measure these asymptotically optimal quantizers are very easy to compute their performance is evaluated for several distributions and compares favorably with the performance of the optimal quantizers in all cases for which the latter have been computed in addition their asymptotic robustness is studied under location scale and shape mismatch for several families of distributions

randomized allocation with nonparametric estimation for a multi armed bandit problem with covariates we study a multi armed bandit problem in a setting where covariates are available we take a nonparametric approach to estimate the functional relationship between the response reward and the covariates the estimated relationships and appropriate randomization are used to select a good arm to play for a greater expected reward randomization helps balance the tendency to trust the currently most promising arm with further exploration of other arms it is shown that with some familiar nonparametric methods e g histogram the proposed strategy is strongly consistent in the sense that the accumulated reward is asymptotically equivalent to that based on the best arm which depends on the covariates almost surely 1 introduction multi armed bandit problems have been extensively studied in probability and statistics in the past few decades and still receive considerable interest in these and related fields readers are referred to berry and fristedt 1985 and gittins 1989 and the references cited therein for the history many elegant results and applications in clinical trials scheduling and other industrial problems some recent developments have been reported in for example lai and yakowitz 1995 who considered the situation in which observations are dependent berry chen zame heath and shepp 1997 who considered infinitely many arms and auer cesa bianchi freund and schapire 1995 who considered worst case performances in classic bandit problems each arm sequentially generates rewards based on a distribution with some unknown parameters and one needs to sequentially select one arm to play for the maximum expected reward in a majority of the earlier work no auxiliary information beyond the observed rewards was considered when selecting an arm to play exceptions include woodroofe 1979 sarkar 1991 and clayton 1989 who considered one armed bandit problems with covariates the first two papers studied bayesian sequential allocation in non bernoulli bandit models with parametric frameworks and showed that the myopic rule is asymptotically optimal the third studied bernoulli bandit problems

f divergence inequalities this paper develops systematic approaches to obtain f divergence inequalities dealing with pairs of probability measures defined on arbitrary alphabets functional domination is one such approach where special emphasis is placed on finding the best possible constant upper bounding a ratio of f divergences another approach used for the derivation of bounds among f divergences relies on moment inequalities and the logarithmic convexity property which results in tight bounds on the relative entropy and bhattacharyya distance in terms of chi 2 divergences a rich variety of bounds are shown to hold under boundedness assumptions on the relative information special attention is devoted to the total variation distance and its relation to the relative information and relative entropy including reverse pinsker inequalities as well as on the e gamma divergence which generalizes the total variation distance pinsker s inequality is extended for this type of f divergence a result which leads to an inequality linking the relative entropy and relative information spectrum integral expressions of the renyi divergence in terms of the relative information spectrum are derived leading to bounds on the renyi divergence in terms of either the variational distance or relative entropy

distributed functional scalar quantization simplified distributed functional scalar quantization dfsq theory provides optimality conditions and predicts performance of data acquisition systems in which a computation on acquired data is desired we address two limitations of previous works prohibitively expensive decoder design and a restriction to source distributions with bounded support we show that a much simpler decoder has equivalent asymptotic performance to the conditional expectation estimator studied previously thus reducing decoder design complexity the simpler decoder features decoupled communication and computation blocks moreover we extend the dfsq framework with the simpler decoder to source distributions with unbounded support finally through simulation results we demonstrate that performance at moderate coding rates is well predicted by the asymptotic analysis and we give new insight on the rate of convergence

on using prototype reduction schemes to optimize dissimilarity based classification the aim of this paper is to present a strategy by which a new philosophy for pattern classification namely that pertaining to dissimilarity based classifiers dbcs can be efficiently implemented this methodology proposed by duin and his co authors see refs experiments with a featureless approach to pattern recognition pattern recognition lett 18 1997 1159 1166 relational discriminant analysis pattern recognition lett 20 1999 1175 1181 dissimilarity representations allow for buillding good classifiers pattern recognition lett 23 2002 943 956 dissimilarity representations in pattern recognition concepts theory and applications ph d thesis delft university of technology delft the netherlands 2005 prototype selection for dissimilarity based classifiers pattern recognition 39 2006 189 208 is a way of defining classifiers between the classes and is not based on the feature measurements of the individual patterns but rather on a suitable dissimilarity measure between them the advantage of this methodology is that since it does not operate on the class conditional distributions the accuracy can exceed the bayes error bound the problem with this strategy is however the need to compute store and process the inter pattern dissimilarities for all the training samples and thus the accuracy of the classifier designed in the dissimilarity space is dependent on the methods used to achieve this in this paper we suggest a novel strategy to enhance the computation for all families of dbcs rather than compute store and process the dbc based on the entire data set we advocate that the training set be first reduced into a smaller representative subset also rather than determine this subset on the basis of random selection or clustering etc we advocate the use of a prototype reduction scheme prs whose output yields the points to be utilized by the dbc the rationale for this is explained in the paper apart from utilizing prss in the paper we also propose simultaneously employing the mahalanobis distance as the dissimilarity measurement criterion to increase the dbcs classification accuracy our experimental results demonstrate that the proposed mechanism increases the classification accuracy when compared with the conventional approaches for samples involving real life as well as artificial data sets even though the resulting dissimilarity criterion is not symmetric

model selection for cart regression trees the performance of the classification and regression trees cart pruning algorithm and the final discrete selection by test sample as a functional estimation procedure are considered the validation of the pruning procedure applied to gaussian and bounded regression is of primary interest on the one hand the paper shows that the complexity penalty used in the pruning algorithm is valid in both cases and on the other hand that conditionally to the construction of the maximal tree the final selection does not alter dramatically the estimation accuracy of the regression function in both cases the risk bounds that are proved obtained by using the penalized model selection validate the cart algorithm which is used in many applications such as meteorology biology medicine pollution monitoring or image coding

palindrome complexity we study the palindrome complexity of infinite sequences on finite alphabets i e the number of palindromic factors blocks of given length occurring in a given sequence we survey the known results and obtain new results for some sequences in particular for rote sequences and for fixed points of primitive morphisms of constant length belonging to class p of hof knill simon we also give an upper bound for the palindrome complexity of a sequence in terms of its block complexity

asymptotic quantization error of continuous signals and the quantization dimension extensions of the limiting qnanfizafion error formula of bennet are proved these are of the form d s k n f n beta b where n is the number of output levels d s k n f is the s th moment of the metric distance between quantizer input and output beta b 0 k s beta is the signal space dimension and f is the signal distribution if a suitably well behaved k dimensional signal density f x exists b b s k int f rho x dx 1 rho rho k s k and b s k does not depend on f for k 1 s 2 this reduces to bennett s formula if f is the cantor distribution on 0 1 0 and this k equals the fractal dimension of the cantor set 12 13 random quantization optimal quantization in the presence of an output information constraint and quantization noise in high dimensional spaces are also investigated

a bootstrap technique for nearest neighbor classifier design a bootstrap technique for nearest neighbor classifier design is proposed our primary interest in designing a classifier is in small training sample size situations conventional bootstrapping techniques sample the training samples with replacement on the other hand our technique generates bootstrap samples by locally combining original training samples the nearest neighbor classifier is designed on the bootstrap samples and is tested on the test samples independent of training samples the performance of the proposed classifier is demonstrated on three artificial data sets and one real data set experimental results show that the nearest neighbor classifier designed on the bootstrap samples outperforms the conventional k nn classifiers as well as the edited 1 nn classifiers particularly in high dimensions

palindromic richness for languages invariant under more symmetries for a given finite group g consisting of morphisms and antimorphisms of a free monoid a we study infinite words with language closed under the group g we focus on the notion of g richness which describes words rich in generalized palindromic factors i e in factors w satisfying q w w for some antimorphism email protected g we give several equivalent descriptions which are generalizations of known characterizations of rich words in the terms of classical palindromes and show two examples of g rich words

proof of the brlek reutenauer conjecture brlek and reutenauer conjectured that any infinite word u with language closed under reversal satisfies the equality 2d u email protected n 0 t u n in which d u denotes the defect of u and t u n denotes c u n 1 c u n 2 p u n 1 p u n where c u and p u are the factor and palindromic complexity of u respectively this conjecture was verified for periodic words by brlek and reutenauer themselves using their results for periodic words we have recently proved the conjecture for uniformly recurrent words in the present article we prove the conjecture in its general version by a new method without exploiting the result for periodic words

histogram regression estimation using data dependent partitions we establish general sufficient conditions for the l 2 conoistency of multivariate histogram regression estimates based on data dependent partitions these same conditions insure the consistency of partitioning regression estimates based on local polynomial fits and with an additional regularity assumption the consistency of histogram estimates for conditional medians our conditions require shrinking cells subexponential growth of a combinatorial complexity measure and sublinear growth of restricted cell counts it is not assumed that the cells of every partition be rectangles with sides parallel to the coordinate axis or that each cell contain a minimum number of points response variables are assumed to be bounded throughout our results may be applied to a variety of partitioning schemes we established the consistency of histograms regression estimates based on cubic partitions with data dependent offsets k thresholding in one dimension and empirically optimal nearest neighbor clustering schemes in addition it is shown that empirically optimal regression trees are consistent when the size of the trees grows with the number of samples at an appropriate rate

recursive partitioning to reduce distortion adaptive partitioning of a multidimensional feature space plays a fundamental role in the design of data compression schemes most partition based design methods operate in an iterative fashion seeking to reduce distortion at each stage of their operation by implementing a linear split of a selected cell the operation and eventual outcome of such methods is easily described in terms of binary tree structured vector quantizers this paper considers a class of simple growing procedures for tree structured vector quantizers of primary interest is the asymptotic distortion of quantizers produced by the unsupervised implementation of the procedures it is shown that application of the procedures to a convergent sequence of distributions with a suitable limit yields quantizers whose distortion tends to zero analogous results are established for tree structured vector quantizers produced from stationary ergodic training data the analysis is applicable to procedures employing both axis parallel and oblique splitting and a variety of distortion measures the results of the paper apply directly to unsupervised procedures that may be efficiently implemented on a digital computer

minimum kl divergence on complements of l_ 1 balls pinsker s widely used inequality upper bounds the total variation distance left vert p q right vert 1 in terms of the kullback leibler divergence d p vert q although in general a bound in the reverse direction is impossible in many applications the quantity of interest is actually d ast v q defined for an arbitrary fixed q as the infimum of d p vert q over all distributions p that are at least v far away from q in total variation we show that d ast v q leq c v 2 o v 3 where c c q 1 2 for balanced distributions thereby providing a kind of reverse pinsker inequality some of the structural results obtained in the course of the proof may be of independent interest an application to large deviations is given

the effect of quantization on the performance of sampling designs the most common form of quantization is rounding off which occurs in all digital systems a general quantizer approximates an observed value by the nearest among a finite number of representative values in estimating weighted integrals of a time series with no quadratic mean derivatives by means of samples at discrete times it is known that the rate of convergence of the mean square error is reduced from n sup 2 to n sup 1 5 when the samples are quantized for smoother time series with k 1 2 quadratic mean derivatives it is now shown that the rate of convergence is reduced from n sup 2k 2 to n sup 2 when the samples are quantized which is a very significant reduction the interplay between sampling and quantization is also studied leading to asymptotically optimal allocation between the number of samples and the number of levels of quantization

properties of a generalized divergence related to tsallis generalized divergence in this paper we investigate the partition inequality joint convexity and pinsker s inequality for a divergence that generalizes the tsallis relative entropy and kullback leibler divergence the generalized divergence is defined in terms of a deformed exponential function which replaces the tsallis q exponential we also constructed a family of probability distributions related to the generalized divergence we found necessary and sufficient conditions for the partition inequality to be satisfied a sufficient condition for the joint convexity was established we proved that the generalized divergence satisfies the partition inequality and is jointly convex if and only if it coincides with the tsallis relative entropy as an application of partition inequality a criterion for the pinsker s inequality was found