# Presentation: 184423

## A General Theory of Goodness of Fit in Likelihood fits

- Rajendran Raja
- Fermilab
- Feb 10, 2006, FNAL Wine and Cheese
- Started working on problem after Durham ’02 conference
- SLAC PHYSTAT03 Gave a solution to the unbinned goodness of fit problem
- Oxford-Phystat2005-Generalized this to Binned and Unbinned goodness of fit as a general theory
- Calculation of Errors– Once the GoF problem is solved, one can use Bayes’ theorem to calculate posterior densities without  Bayesian Priors.  Mathematical Proof.
- Can provide Fortran Subroutines do perform unbinned goodness of fit, if interest.
- See physics/0509008 for further details-60 page writeup

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Overview of Goodness of fit techniques

- The 2 technique introduced by Karl Pearson is widely used. It is used for binned fitting. Analytic theory fully developed. Error on fitted parameter is obtained in the Gaussian approximation by 2 =1. Probably the most commonly used measure of goodness of fit. Drawback—Does not handle bins with low statistics well.
- G2 method (binned)-Known to Statisticians-in HEP- Baker and Cousins(83).
- G2(Multinomial)=
- where nk is the number observed in the kth bin
- Tk is the number theoretically expected in the kth bin
- nb is the number of bins in the histogram.
- G2(Poisson)=
- The theory normalization and experimental normalization may be different
- No analytic theory of distribution of G2.. However, can show that for large statistics, asymptotes to 2.
- For maximum likelihood fitting, binned and unbinned methods exist. The theory we will derive will bring all of the above under one framework.
- Kolmogorov-Smirnov, Aslan-Zech (energy scheme) are not likelihood techniques and will not be covered here.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## An example where fitting is inadequate- W transverse mass

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Format of talk

- State Maximum likelihood  principle.
- Show why Likelihood cannot act as a goodness of fit variable- Not invariant under transformation of variables
- Define Likelihood ratio of Theoretical likelihood/Data Likelihood as derived from data. Show this is invariant under transformation of variables.
- Apply theory for unbinned goodness of fit using PDE’s
- Illustrative Example for 1d unbinned fits.
- Illustrative example for unbinned fitting for an extreme case.
- Derive Goodness of fit formulae for binned fitting
- Show these asymptote to a 2 like variable for large statistics.
- Show theory applies for 2 fitting.
- Calculation of errors- Problem of inverse probabilities-Derive Bayes’ theorem- Describe Bayesian Paradigm
- Show that Goodness of fit with “Data Likelihood from Data” is incompatible with Bayesian Prior.
- Introduce New Paradigm in Statistics that permits inversion of probabilities without Bayesian priors-Uses Bayes’ theorem and Goodness of fit.
-  Illustrate ideas using examples.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Notation

- s denotes signal. Can be multi-dimensional. Denotes theoretical PARAMETERS. Can be multidimensional.
- c denotes configurations and signifies DATA. Can be multi-dimensional
- P(c|s) signifies the conditional probability density in c, given s. I.e It defines the theoretical model-theory pdf which obeys normalization condition.
- P(s|c) signifies the conditional probability density in s, given c.
- Let       denote the dataset
- Then
- is the likelihood of observing the dataset
- n is the number of elements in a dataset (experiment)
- N denotes the number of datasets in an ensemble.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Maximum Likelihood method for fitting theory to data

- Due to R.A. Fisher(Philo.Trans.Roy.Soc Ser A 222 309-368 (1922))
- Maximize
-  The maximum likelihood point s* yields the best fit to the data of the theoretical model specified by P(c|s)
- This yields the optimum estimate for the true value of s. However, no goodness of fit criterion exists. Likelihood at maximum likelihood point is NOT such a measure.
- Key Observation-  GoF relies on two distributions. Theory and data. Maximum likelihood only uses Theory distribution (evaluated at the data points).

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## To show that Likelihood does not furnish goodness of fit measure

- Goodness of fit must be invariant under change of variable c’=c’(c)

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Likelihood Ratios

- Introduce the concept of “data likelihood” as derived from data. If P(c) is the “frequentist” pdf of the data, the the “data likelihood” for dataset        is defined as
- This object may be thought of as the pdf of the n-object       .
- Then the likelihood ratio
- Is invariant under change of variable since the Jacobians cancel in the numerator and denominator. We will show that it asymptotes to a distribution for large statistics(binned case) and hence will satisfy as a GoF measure.
- Maximum Likelihood point s* and Maximum likelihood ratio point are the same, since denominator does not depend on s.
- Likelihood ratios multiply.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Historical use of Likelihood ratios

- Likelihood ratios are enshrined in statistics by the Neyman-Pearson lemma which states that the cut
-      will have the optimum power in differentiating between the hypotheses s1 and s2 where is a cut chosen to obtain the requisite purity.
- Note that the Neyman-Pearson use of the likelihood ratio is ***between two theoretical distributions*** both in the numerator and the denominator.
- We believe, we are introducing the notion of data likelihood as evaluated using the data pdf alone. This quantity restores the goodness of fit in both the binned and unbinned likelihood fits, yielding a general theory of goodness of fit.
- It provides a method of inverting probabilities without a Bayesian prior
- What is it– It is merely a generalized pdf of the n-object

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## GoF in Unbinned Likelihood Fits

- We need to evaluate
- For the unbinned case. This is done if we have an algorithm to evaluate the denominator for each event c. We accomplish this by means of Probability Density Estimators (PDE’s) also known as Kernel Density Estimators (KDE’s).
- Essentially each event is smeared by a Kernel Density (Gaussians are popular kernels). All the Kernels are added up and the sum divided by n, the number of events. The resulting PDE approximates to the pdf of the data P(c).
- We evaluate the likelihood ratio at the maximum likelihood point s*. In order to quantify the goodness of fit, we evaluate the distribution of the negative log likelihood ratio (NLLR) for an ensemble of fits for Monte Carlo generated to mimic the theoretical curve.  NO analytic theory as yet available.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Probability Density Estimators(PDE)E.Parzen, Ann Math.Statis 32,1065-1072

- If d  is the dimension of the vector c, then
- H=E-1  .  h is a smoothing factor

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Probability Density Estimators

- This is assured by making  the smoothing factor depend on the number of events.
- PDE’s are generalizable to arbitrary dimensions.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Illustrative Example

- Histogram (with errors) showsevents generated as an exponential P(c|s), for s=1.0. Superimposed is the PDE estimator (solid histogram). Boundary effect near c=0.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## PDE tracks data

- Histogram with errors shows exponential (1000 events) with superimposed Gaussian of 500 events (=2.0, =0.2).  The PDE estimator (Solid) histogram.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Improve the smoothing factor

- Smoothing factor should be allowed to vary as a function of event density. Estimate event density using constant smoothing factor and then apply the formula
- t1-t2 are the lower and upper limits of c
- Iterate 3 times till things settle down.

- Variation of h as a function of c for the above example determined iteratively.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Variable vs constant smoothing factor

- Solid Histogram shows NLLR for 500 distributions using iterative smoothing. The dashed curve corresponds to the case for constant smoothing factor h.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Results of unbinned and binned fitting

- 1000 background events

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## An example of an unbinned fit.

- Histogram (with errors) is a Gaussian of 85 events (=2,=0.2) superimposed on an exponential of 1000 events. This causes an NNLR that is 3 away from the mean NLLR for the null hypothesis.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Likelihood vs Likelihood Ratio

- Negative log likelihoods, beside not being invariant are broad. (Top Left)
- Negative Log PDE likelihood are similar (Top right) and contain information from data!
- The two correlate (Bottom left)
- The difference is the narrow GoF, negative log likelihood ratio (Bottom right)

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Unbinned Goodness of Fit-Improving the PDE

- Apply same theory. We now need a way to estimate the “data density” for unbinned data.
- Steps- First, perform an unbinned likelihood fit.
- Second-Evaluate the “data likelihood”            by the method of probability density estimators (PDE)- also known as Kernel Density Estimators (KDE).
- Reported on this in PHYSTAT03.
- To summarize, the best PDE method is to transform to hypercube co-ordinates, where P(c’|s) is flat.
- The likelihood ratio LR is invariant and will not depend on the co-ordinate system.
- For each event ci i=1,n, define a boxcar function centered on ci of width h so that the kernel
- Use periodical boundary conditions on the kernels so that
- Treats every point in hypercube like every other.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Unbinned Likelihood gof

- Then the PDE for the data likelihood is given by
- Exponential + Gaussian- Data (crosses) and PDE histogram

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Dependence of NLLR on h

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Dependence of NLLR on n

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Extreme Example

- Problem-(set by Bruce Knuteson)-3 data points in 3D space have co-odinates (x,y,z)=(0.1,0.2,0.3), (0.2,0.4,0.1),and (0.05,0.6,0.21). What is the goodness of fit to the hypothesis that they are distributed according to P(x,y,z)=exp(-(x+y+z))?
- Likelihood function is given by

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Results

- The data are a poor fit to the hypothesis

| Smoothing
Parameter h | NLLR | Probablilty to exceed NLLR |
| --- | --- | --- |
| 0.2 | 5.36 | 0.5% |
| 0.3 | 5.84 | <0.1% |
| 0.4 | 1.77 | 1% |

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Extreme Example-NLLR vs h n=9

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Binned Likelihood goodness of fit

- Let there be nb bins and let the kth bin whose center abscissa is given by ck contain nk entries.
- Then the probability of obtaining the histogram containing a total of n events is given by the multinomial
- Histogram is degenerate with degeneracy factor
- Each such histogram has the same goodness of fit
- So we can just solve the problem for one of them.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Binned Likelihood goodness of fit

- Then the binned likelihood ratio is
- We can estimate the denominator by
-       where        is the bin width for the kth bin. This yields
- where the Tk is the theoretically expected number of events in the kth bin given by
- This is the same result as given by Baker and Cousins(NIM A221, 1984) for the multinomial but derived using the new theory.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Comparison of derivation with Baker and Cousins

- Baker and Cousins justify the use of a likelihood ratio by the likelihood ratio theorem and employ a LR of  theory to the true value.
- Then they say the  maximum likelihood for the true value is when (True value)k = nk. This yields the same result, but our rationale is very different. We form a ratio between the theoretical likelihood and the new concept the data likelihood as derived from data. Their theory cannot do unbinned likelihood GoF’s.
- We now show that this quantity asymptotes to a 2 distribution for large n.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## To show that NLLR asymptotes to a 2 distribution

- Example of the Likelihood Ratio Theorem

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Maximum likelihood fits only shapes not normalizations

- No information on the theoretical expected number of events. We evaluate the likelihood ratio on the observed data events. Both the theory likelihood and data likelihood integrate to unity. So no information on the normalization.
- Get around this by using the binomial distribution and demanding the observed number of events be in the first bin. Let N be the number of identical experiments in an ensemble. Then for the binomial, the likelihood ratio, using our theory,can be written
- We let N go to infinity in two well known limits.
- Poisson—nt and n finite.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Binned Likelihood gof- Poisson Normalization gof

- Multiplying this LR with binned likelihood LR yields the formulae
- Same as the “multinomial” and “poissonian” results of Baker and Cousins.
- Gaussian Limit of the Binomial, n and nt also go to infinity as N goes to infinity. Yields

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## To show that Pearson 2 distribution is an NLLR

- Let the contents of the kth bin be denoted by ck.
- Let the theoretical expectation for the kth bin be sk.
- Let the standard deviation of the kth bin be  k.
- Then,
- People mistakenly conclude that 2 is the negative log of a likelihood.Correct expression is to note that

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Why is this theory important to HEP?

- Very useful in multivariate analyses. Even though we have shown examples in 1 dimension, theory is easily generalizable to multi-dimensions. Theoretical model can also be Monte Carlo events. Theory and data are interpolated by use of PDE’s.
- Theory can be signal + background Monte Carlo
- NLLR distribution obtained by comparing one set of theory (numerator) vs many such generations of “data from theory” for denominator.
- GoF is useful in in evaluating one set of theoretical curves wrt another.
- Likelihood can be done in correlated variables or uncorrelated variables.
- Excellent tool for LHC analyses.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Errors in Fitted Parameters

- Problem of “inverse probability”.
- Theoretical likelihood is given by
- In order to estimate the errors in s, we need to determine the “inverse probability”
- One needs Bayes’ theorem. In Bayesian language this is also known as “Posterior density”.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Bayes’ Theorem-Simple derivation

- Define a joint probability density P(s,c) such that
- Then define projections P(c), P(s) such that
- Define conditional probability P(c|s) along line AB
- Define conditional probability P(s|c) along line CD

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Bayes’ Theorem-Simple derivation

- Then
- Generalizing to dataset

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## The Bayesian Paradigm

- Note that Bayes’ theorem is a theorem in mathematics and not statistics, since it can be derived for any function of 2 variables with finite integral. The “conditional probabilities” are appropriately normalized “slices” of such a function and the functions P(s) and P(c) are projections of such a function.
- It is up to us how to identify the various functions and map them to our theory.
- In order to invert the probability, one needs to specify one projection and one slice function. This can be used to determine the joint probability, from which the inverse probability may be computed.
- The Bayesian paradigm is to assume that P(s), the projection on the parameter axis of the joint probability is a function that is obtained by prior knowledge of the parameter. This function is supplied during inversion and is known as the Bayesian prior.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## The Bayesian Paradigm

- Several kinds of Bayesian methods– for providing the prior-
- Objective Bayesianism (E.T.Jaynes et al) Assume the prior is flat in the variable of interest within some reasonable bounds– Immediate objection- Flat in which variable, s, s2, loges,loge(loges) etc? Results will vary.
- Subjective Bayesianism (de Finetti et al). Prior knowledge depends on the prior “history” of the experimenter. So different priors should be used. A collider experiment can and will have more priors than experimenters and as many different results!
- Hierarchical Bayesianism- Parameterize the prior by some new parameters. These parameters will have their own priors and so on!
- Empirical Bayesianism- Attempts to terminate the above infinite regression by determining the parameters from data.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## The Bayesian Paradigm

- All Bayesians use Bayes’ theorem. So they term the standard method of obtaining errors from  fitting () and from likelihood fitting (L=1/2) as illegal, since these do not use priors.
- The Bayesians then compute the posterior density as
- Bayesians do not have GoF.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## All Bayesians make this substitution

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## The New Paradigm

- We now try and solve for the joint probability using Bayes’ theorem by retaining the denominator in the likelihood ratio as being obtained from data. This results in an iterative inversion of probability that is reminiscent of “Fisher’s fiducial probability” that preserves the GoF.  R.A.Fisher Annals of Eugenics,6:391-398(1935). Fisher did not solve the GoF problem, so he could not counter Baysians, though he tried.
- This leads to
- This leads to a projection Pn(s) on the parameter axis that is n dependent since
- So the projection on the parameter axis cannot be an n independent Bayesian prior. Need new definitions and paradigm.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## The New Paradigm- Definitions

- We redefine Pn(s) as the distribution of the maximum likelihood value s* when the experiment with data set with n members is repeated N times (ensemble) and N. We re-label it as  Pn(s*) . sT, the true value of s, is the maximum likelihood point of Pn(s*). This is an assumption of unbiasedness in the experiment.  The true value is a number and does not have a distribution. The true value is unknown and unknowable with infinite precision. The function Pn(s*) is also unknowable.
- To calculate errors we assume that given a single dataset        , not only is the maximum likelihood value s* knowable, but there is information present on the distribution of s* as well-i.e errors on s* are computable. We call such a function

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## New Paradigm

- Each member dataset k of the ensemble provides a maximum likelihood value sk* and also has  a function                 associated with it.
- Then the joint probability          is given by
- Not much use to us, since Pn(s*) is only available after a large number of experiments on the ensemble. After one experiment, what can we learn?

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Error Bootstrap

- The value of the distribution Pn(s*) at the true value is denoted by
- It depends on the individual error and the number n in the dataset.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Error Bootstrap

- The quantity Pn(s*) is the ensemble average of all the posterior densities              . Its maximum likelihood value is the true value sT.
- We only have measurements from one member of the ensemble namely
- .We want to describe to the system our lack of knowledge of the true value. I.e. We want to say that it is at s1 OR it is at s2 OR it is at s3.  At each value of s*, we hypothesize that that is the true value.
- The likelihood ratio LR(s*) gives the goodness of fit at that value.
- At the true value s* =sT, the joint probability  P(s*,c) is given by
- As  you change the value of s*, the whole distribution Pn(s*) has to move so that the true value is at the new value of s*. I.e. Pn(sT) in Bayes’ equation is a constant.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Error Bootstrap

- This is the same formula as frequentists use! No Bayesian prior. But it is the first iteration. As we get more members of the ensemble, we will re-use Bayes’ theorem iteratively.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Illustrative example

- Measure a mass whose true value sT is unknown with an apparatus whose standard error  is known to be 5 gms. A single data set consists of n=100 measurements. Then the likelihood for an arbitrary mass s is given by
- Do goodness of fit using the method of unbinned likelihood fits. Obtain NLLR, likelihoods for each individual fit.
- Determine                   for each fit. Average over ensemble N=1000 fits to obtain a better value of Pn(s*). We reuse Bayes’ theorem to re-evaluate posteriors, since we know Pn(s*) from the ensemble better than from individual measurements.
- One more iteration. For the kth element of the ensemble

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

##   Illustrative example

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Illustrative example. Iterated functions

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Illustrative example

- Fit to s* histogram

- Pn(s*) for n=100, 200

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Iterative Example

- Same as above, except n=1. I.e. One measurement. Measurement error is =1 gm. We can project to both s* and c axes. First iteration- Do a maximum likelihood fit-trivial s*=c.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Iterative solution

- Similarly for the theoretical likelihood,

- Iterate the posterior densities first

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## How is         iteratively modified by Bayes’ theorem

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## First iteration

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Second iteration

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Joint probability

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Joint Probability 

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Bayes’ theorem Equations Alternate forms valid for the new paradigm.

- Also, the equation for Pn(s*) becomes an eigenfunction of  PDE with a kernel. See long write-up

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006

## Conclusions

- We have given a general theory of goodness of fit that applies equally well to binned and unbinned likelihood fits. We have shown that the usual is also a special case of this general theory.
- We have shown an iterative  way to obtain posterior densities that does not require the Bayesian prior but uses Bayes’ theorem.
- Method has applications to Tevatron and LHC analyses using multivariate techniques.
- Information from other experiments combined by multiplying likelihood ratios.

- Rajendran Raja, Fermilab Wine and Cheese Seminar

- 10-Feb-2006