probability estimation in the rare events regime we address the problem of estimating the probability of an observed string that is drawn i i d from an unknown distribution motivated by models of natural language we consider the regime in which the length of the observed string and the size of the underlying alphabet are comparably large in this regime the maximum likelihood distribution tends to overestimate the probability of the observed letters so the good turing probability estimator is typically used instead we show that when used to estimate the sequence probability the good turing estimator is not consistent in this regime we then introduce a novel sequence probability estimator that is consistent this estimator also yields consistent estimators for other quantities of interest and a consistent universal classifier

concentration inequalities for the missing mass and for histogram rule error this paper gives distribution free concentration inequalities for the missing mass and the error rate of histogram rules negative association methods can be used to reduce these concentration problems to concentration questions about independent sums although the sums are independent they are highly heterogeneous such highly heterogeneous independent sums cannot be analyzed using standard concentration inequalities such as hoeffding s inequality the angluin valiant bound bernstein s inequality bennett s inequality or mcdiarmid s theorem the concentration inequality for histogram rule error is motivated by the desire to construct a new class of bounds on the generalization error of decision trees

calculation of entropy from data of motion we discuss the question of determining the entropy given the phase space trajectory which describes the detailed history of a many body system over a period of observation our viewpoint is that the determination of entropy as well as all other thermodynamic properties should require no concepts or information other than those given and defined by the trajectory the counting of coincidence or repetition of states along the trajectory is presented as a way to determine entropy given the trajectory an illustrative program based on the kinetic ising model is described in detail

asymptotically optimal classification for multiple tests with empirically observed statistics the decision problem of testing m hypotheses when the source is kth order markov and there are m or fewer training sequences of length n and a single test sequence of length n is considered k m n n are all given it is shown what the requirements are on m n n to achieve vanishing exponential error probabilities and how to determine or bound the exponent a likelihood ratio test that is allowed to produce a no match decision is shown to provide asymptotically optimal error probabilities and minimum no match decisions as an important serial case the binary hypotheses problem without rejection is discussed it is shown that for this configuration only one training sequence is needed to achieve an asymptotically optimal test

classification of homogeneous data with large alphabets given training sequences generated by two distinct but unknown distributions on a common alphabet we study the problem of determining whether a third sequence was generated according to the first or second distribution to model sources such as natural language for which the underlying distributions are difficult to learn from realistic amounts of data we allow the alphabet size to grow and therefore the probability distributions to change with the block length our primary focus is the situation in which the underlying probabilities are all of the same order and in this regime we show that consistent classification is possible if and only if the alphabet grows subquadratically with the block length we also show that some commonly used statistical tests are suboptimal in that they are consistent only if the alphabet grows sublinearly

entropy estimates of small data sets estimating entropies from limited data series is known to be a non trivial task naive estimations are plagued with both systematic bias and statistical errors here we present a new balanced estimator for entropy functionals shannon renyi and tsallis specially devised to provide a compromise between low bias and small statistical errors for short data series this new estimator outperforms other currently available ones when the data sets are small and the probabilities of the possible outputs of the random variable are not close to zero otherwise other well known estimators remain a better choice the potential range of applicability of this estimator is quite broad specially for biological and digital data series

a poisson model for the coverage problem with a genomic application summary suppose a population has infinitely many individuals and is partitioned into unknown n disjoint classes the sample coverage of a random sample from the population is the total proportion of the classes observed in the sample this paper uses a nonparametric poisson mixture model to give new understanding and results for inference on the sample coverage the poisson mixture model provides a simplified framework for inferring any general abundance f coverage the sum of the proportions of those classes that contribute exactly k individuals in the sample for some k in with being a set of nonnegative integers a new moment based derivation of the well known turing estimators is presented as an application a gene categorisation problem in genomic research is addressed since turing s approach is a moment based method maximum likelihood estimation and minimum distance estimation are indicated as alternatives for the coverage problem finally it will be shown that any turing estimator is asymptotically fully efficient

estimating entropy on m bins given fewer than m samples consider a sequence p sub n of discrete probability measures supported on m sub n points and assume that we observe n independent and identically distributed i i d samples from each p sub n we demonstrate the existence of an estimator of the entropy h p sub n which is consistent even if the ratio n m sub n is bounded and as a corollary even if this ratio tends to zero albeit at a sufficiently slow rate

on classification with empirically observed statistics and universal data compression classification with empirically observed statistics is studied for finite alphabet sources efficient universal discriminant functions are described and shown to be related to universal data compression it is demonstrated that if one of the probability measure of the two classes is not known it is still possible to define a universal discrimination function which performs as the optimal likelihood ratio discriminant function which can be evaluated only if the probability measures of the two classes are available if both of the probability measures are not available but training vectors from at least one of the two classes are available it is demonstrated that no discriminant function can perform efficiency of the length of the training vectors does not grow at least linearly with the length of the classified vector a universal discriminant function is introduced and shown to perform efficiently when the length of the training vectors grows linearly with the length of the classified sequence in the sense that it yields an error exponent that is arbitrarily close to that of the optimal discriminant function

a semi lagrangian approach for american asian options under jump diffusion a semi lagrangian method is presented to price continuously observed fixed strike asian options at each timestep a set of one dimensional partial integrodifferential equations pides is solved and the solution of each pide is updated using semi lagrangian timestepping crank nicolson and second order backward differencing timestepping schemes are studied monotonicity and stability results are derived with low volatility values it is observed that the nonsmoothness at the strike in the payoff affects the convergence rate a subquadratic convergence rate is observed

finite difference scheme with a moving mesh for pricing asian options in this paper we propose a stable numerical method for pricing asian call options which is based on a central difference scheme with a moving mesh in the spatial discretization and the rannacher time stepping scheme in the time discretization at each time mesh point we make a piecewise uniform mesh to discretise the space interval which ensures that the matrix associated with the discrete operator is an m matrix hence the spatial discretization scheme is maximum norm stable for arbitrary volatility and arbitrary interest rate we show that the scheme is second order convergent with respect to both time and spatial variables numerical results support the theoretical results

analytical pricing of geometric asian power options on an underlying driven by a mixed fractional brownian motion abstract in this paper we study the pricing problem of the continuously monitored fixed and floating strike geometric asian power options in a mixed fractional brownian motion environment first we derive both closed form solutions and mixed fractional partial differential equations for fixed and floating strike geometric asian power options based on delta hedging strategy and partial differential equation method second we present the lower and upper bounds of the prices of fixed and floating strike geometric asian power options under the assumption that both risk free interest rate and volatility are interval numbers finally numerical studies are performed to illustrate the performance of our proposed pricing model

minimax rates of entropy estimation on large alphabets via best polynomial approximation consider the problem of estimating the shannon entropy of a distribution over k elements from n independent samples we show that the minimax mean square error is within the universal multiplicative constant factors of big k n log k big 2 log 2 k n if n exceeds a constant factor of k log k otherwise there exists no consistent estimator this refines the recent result of valiant and valiant that the minimal sample size for consistent entropy estimation scales according to theta k log k the apparatus of the best polynomial approximation plays a key role in both the construction of optimal estimators and by a duality argument the minimax lower bound

wavelet galerkin methods for second kind integral equations we use vector valued multiwavelets on compact sets to develop a galerkin method for systems of integral equations of the second kind we propose a compression strategy for the coefficient matrix of the linear system obtained from this method and show that the compressed scheme preserves almost optimal convergence rate of the original scheme and yields a sparse matrix with a bounded condition number

the complexity of approximating the entropy we consider the problem of approximating the entropy of a discrete distribution under several different models of oracle access to the distribution in the evaluation oracle model the algorithm is given access to the explicit array of probabilities specifying the distribution in this model linear time in the size of the domain is both necessary and sufficient for approximating the entropy in the generation oracle model the algorithm has access only to independent samples from the distribution in this case we show that a gamma multiplicative approximation to the entropy can be obtained in o n 1 eta gamma 2 log n time for distributions with entropy omega gamma eta where n is the size of the domain of the distribution and eta is an arbitrarily small positive constant we show that this model does not permit a multiplicative approximation to the entropy in general for the class of distributions to which our upper bound applies we obtain a lower bound of omega n 1 2 gamma 2 we next consider a combined oracle model in which the algorithm has access to both the generation and the evaluation oracles of the distribution in this model significantly greater efficiency can be achieved we present an algorithm for gamma multiplicative approximation to the entropy that runs in o gamma 2 log 2 n h 2 gamma 1 2 time for distributions with entropy omega h for such distributions we also show a lower bound of omega log n h gamma 2 1 gamma 2 finally we consider two special families of distributions those in which the probabilities of the elements decrease monotonically with respect to a known ordering of the domain and those that are uniform over a subset of the domain in each case we give more efficient algorithms for approximating the entropy

universal entropy estimation via block sorting in this correspondence we present a new universal entropy estimator for stationary ergodic sources prove almost sure convergence and establish an upper bound on the convergence rate for finite alphabet finite memory sources the algorithm is motivated by data compression using the burrows wheeler block sorting transform bwt by exploiting the property that the bwt output sequence is close to a piecewise stationary memoryless source we can segment the output sequence and estimate probabilities in each segment experimental results show that our algorithm outperforms lempel ziv lz string matching based algorithms

large deviation asymptotics for occupancy problems in the standard formulation of the occupancy problem one considers the distribution of r balls in n cells with each ball assigned independently to a given cell with probability 1 n although closed form expressions can be given for the distribution of various interesting quantities such as the fraction of cells that contain a given number of balls these expressions are often of limited practical use approximations provide an attractive alternative and in the present paper we consider a large deviation approximation as r and n tend to infinity in order to analyze the problem we first consider a dynamical model where the balls are placed in the cells sequentially and time corresponds to the number of balls that have already been thrown a complete large deviation analysis of this process level problem is carried out and the rate function for the original problem is then obtained via the contraction principle the variational problem that characterizes this rate function is analyzed and a fairly complete and explicit solution is obtained the minimizing trajectories and minimal cost are identified up to two constants and the constants are characterized as the unique solution to an elementary fixed point problem these results are then used to solve a number of interesting problems including an overflow problem and the partial coupon collector s problem

generalized error exponents for small sample universal hypothesis testing the small sample universal hypothesis testing problem is investigated in this paper in which the number of samples n is smaller than the number of possible outcomes m the goal of this paper is to find an appropriate criterion to analyze statistical tests in this setting a suitable model for analysis is the high dimensional model in which both n and m increase to infinity and n o m a new performance criterion based on large deviations analysis is proposed and it generalizes the classical error exponent applicable for large sample problems in which m o n this generalized error exponent criterion provides insights that are not available from asymptotic consistency or central limit theorem analysis the following results are established for the uniform null distribution 1 the best achievable probability of error pe decays as pe exp n2 m j 1 o 1 for some j 0 2 a class of tests based on separable statistics including the coincidence based test attains the optimal generalized error exponents 3 pearson s chi square test has a zero generalized error exponent and thus its probability of error is asymptotically larger than the optimal test

universal divergence estimation for finite alphabet sources this paper studies universal estimation of divergence from the realizations of two unknown finite alphabet sources two algorithms that borrow techniques from data compression are presented the first divergence estimator applies the burrows wheeler block sorting transform to the concatenation of the two realizations consistency of this estimator is shown for all finite memory sources the second divergence estimator is based on the context tree weighting method consistency is shown for all sources whose memory length does not exceed a known bound experimental results show that both algorithms perform similarly and outperform string matching and plug in methods

the efficiency of good s nonparametric coverage estimator the asymptotic efficiency of good s nonparametric coverage estimator is obtained relative to the best estimator derived under the assumption that all classes are equally likely even when that assumption is true good s estimator is quite efficient with an asymptotic relative efficiency of greater than 85 in all cases and greater than 95 if the expected coverage is less than one half

a reliable numerical method to price arithmetic asian options abstract we design and analyze two numerical methods for pricing asian options the first one is an explicit finite difference method and therefore as usual only conditionally stable the second method is an implicit finite difference method and unconditionally stable to explore the basic ideas of analysis we discuss the explicit method in detail and then highlight the crucial steps in the analysis of the implicit method numerical results obtained by these two methods are compared with those obtained by an improved monte carlo method the proposed implicit method is very robust as is evident from the comparative numerical results we also provide additional numerical results that confirm theoretical investigations done by many other researchers

bias analysis in entropy estimation we consider the problem of finite sample corrections for entropy estimation new estimates of the shannon entropy are proposed and their systematic error the bias is computed analytically we find that our results cover correction formulae of current entropy estimates recently discussed in the literature the trade off between bias reduction and the increase of the corresponding statistical error is analysed

estimation of entropy and mutual information we present some new results on the nonparametric estimation of entropy and mutual information first we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators the setup is related to grenander s method of sieves and places no assumptions on the underlying probability measure generating the data second we prove a converse to these consistency theorems demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information even given unlimited data this inconsistency theorem leads to an analytical approximation of the bias valid in surprisingly small sample regimes and more accurate than the usual 1 n formula of miller and madow over a large region of parameter space the two most practical implications of these results are negative 1 information estimates in a certain data regime are likely contaminated by bias even if bias corrected estimators are used and 2 confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods finally we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem by casting bias calculation problems in this approximation theory framework we obtain the best possible generalization of known asymptotic bias results more interesting this framework leads to an estimator with some nice properties the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions and this maximum error turns out to be surprisingly small we demonstrate the application of this new estimator on both real and simulated data

universal compression of memoryless sources over unknown alphabets it has long been known that the compression redundancy of independent and identically distributed i i d strings increases to infinity as the alphabet size grows it is also apparent that any string can be described by separately conveying its symbols and its pattern the order in which the symbols appear concentrating on the latter we show that the patterns of i i d strings over all including infinite and even unknown alphabets can be compressed with diminishing redundancy both in block and sequentially and that the compression can be performed in linear time to establish these results we show that the number of patterns is the bell number that the number of patterns with a given number of symbols is the stirling number of the second kind and that the redundancy of patterns can be bounded using results of hardy and ramanujan on the number of integer partitions the results also imply an asymptotically optimal solution for the good turing probability estimation problem

minimax estimation of functionals of discrete distributions we propose a general methodology for the construction and analysis of essentially minimax estimators for a wide class of functionals of finite dimensional parameters and elaborate on the case of discrete distributions where the support size s is unknown and may be comparable with or even much larger than the number of observations n we treat the respective regions where the functional is nonsmooth and smooth separately in the nonsmooth regime we apply an unbiased estimator for the best polynomial approximation of the functional whereas in the smooth regime we apply a bias corrected version of the maximum likelihood estimator mle we illustrate the merit of this approach by thoroughly analyzing the performance of the resulting schemes for estimating two important information measures 1 the entropy h p sum i 1 s p i ln p i and 2 f alpha p sum i 1 s p i alpha alpha 0 we obtain the minimax l 2 rates for estimating these functionals in particular we demonstrate that our estimator achieves the optimal sample complexity n asymp s ln s for entropy estimation we also demonstrate that the sample complexity for estimating f alpha p 0 is n asymp s 1 alpha ln s which can be achieved by our estimator but not the mle for 1 we show the minimax l 2 rate for estimating f alpha p is n ln n 2 alpha 1 for infinite support size while the maximum l 2 rate for the mle is n 2 alpha 1 for all the above cases the behavior of the minimax rate optimal estimators with n samples is essentially that of the mle plug in rule with n ln n samples which we term effective sample size enlargement we highlight the practical advantages of our schemes for the estimation of entropy and mutual information we compare our performance with various existing approaches and demonstrate that our approach reduces running time and boosts the accuracy moreover we show that the minimax rate optimal mutual information estimator yielded by our framework leads to significant performance boosts over the chow liu algorithm in learning graphical models the wide use of information measure estimation suggests that the insights and estimators obtained in this paper could be broadly applicable