# Presentation: 637999

## BAYES and FREQUENTISM: The Return of an Old Controversy

-                       Louis Lyons
-                   Oxford University
- LBL, January 2008

## Commercial break

- Workshop on Multivariate methods for separating Signal from Background
- Caltech, Monday Feb 11th
- Jan 23rd, 29th, 30th:
-    p-values and discovery
-    Learning to love the error matrix
-    Upper Limits

- I am here till end of January
- Room      50-5056B
- Tel          486-4467
- E-mail     l.lyons@physics.ox.ac.uk

## Slide 4

## Slide 5

## Slide 6

## Topics

-   Who cares?
-   What is probability?
-   Bayesian approach
-   Examples
-   Frequentist approach
-   Systematics
-   Summary

- It is possible to spend a lifetime analysing data without realising that there are two very different fundamental approaches to statistics:
- Bayesianism and Frequentism.

## How can textbooks not even mention

-              Bayes / Frequentism?

- For simplest case

-  with no constraint on

- then

- at some probability, for both Bayes and Frequentist

- (but different interpretations)

- See Bob Cousins “Why isn’t every physicist a Bayesian?” Amer Jrnl Phys 63(1995)398

## We need to make a statement about

- Parameters, Given Data

- The basic difference between the two:
- Bayesian :      Probability (parameter, given data)
-                       (an anathema to a Frequentist!)
- Frequentist :   Probability (data, given parameter)
-                        (a likelihood function)

## PROBABILITY

- _ __MATHEMATICAL_
- 	Formal
- 	Based on Axioms
-  _FREQUENTIST_
- 	Ratio of frequencies as  n infinity
- 	Repeated “identical” trials
- 	Not applicable to **single event**  or **physical constant**
-  _BAYESIAN_	Degree of belief
- 	Can be applied to single event or physical constant
- 	(even though these have unique truth)
- 	Varies from person to person      ***
- 	Quantified by “fair bet”

## Bayesian versus Classical

- Bayesian

- P(A and B) = P(A;B) x P(B) = P(B;A) x P(A)

- e.g.  A = event contains t quark
-         B = event contains W boson
- or     A = I am in LBL
-         B = I am giving a lecture

- P(A;B) = P(B;A) x P(A) /P(B)
- Completely uncontroversial, provided....

## Bayesian

- posterior

- likelihood

- prior

- Problems:   p(param)   Has particular value
-                                      “Degree of belief”
-                    Prior          What functional form?
-                    Coverage

- Bayes’ Theorem

- p(param | data)  α   p(data | param) * p(param)

## P(parameter)        Has specific value

-                              “Degree of Belief”
-                               Credible interval
- Prior:       What functional form?

- Uninformative prior:    flat?
- In which variable?
- Even more problematic with more params

- Unimportant if “data overshadows prior”
- Important for limits
- Subjective or Objective prior?

## Slide 15

## Prior

## Prior = zero in unphysical region

## Bayes: Specific example

- Particle decays exponentially:     dn/dt = (1/τ) exp(-t/τ)
- Observe 1 decay at time t1:         **L**(τ)  = (1/τ) exp(-t1/τ)
- Choose prior π(τ) for τ
-      e.g. constant up to some large τ            **L**
- Then posterior p(τ) =**L**(τ) * π(τ)
-          has almost same shape as **L**(τ)
- Use p(τ) to choose interval for                                             τ                                                      τ in usual way
- Contrast frequentist method for same situation later.

## Bayesian posterior  intervals

-                           Upper limit                                 Lower limit
-                       Central interval                         Shortest

## Ilya Narsky, FNAL CLW 2000

## P (Data;Theory)         P (Theory;Data)

-          HIGGS SEARCH at CERN

- Is data consistent with Standard Model?
-           or with Standard Model + Higgs?

- End of Sept 2000:  Data not very consistent with S.M. Prob (Data ; S.M.) < 1%  valid frequentist statement
- Turned by the press into:   Prob (S.M. ; Data) < 1%    and therefore                  Prob (Higgs ; Data) > 99%
- i.e.  “It is almost certain that the Higgs has been seen”

## P (Data;Theory)         P (Theory;Data)

- Theory  =  male or female
- Data     =   pregnant or not pregnant

- P (pregnant ; female) ~ 3%

## P (Data;Theory)         P (Theory;Data)

- Theory  =   male or female
- Data      =   pregnant or not pregnant

- P (pregnant ; female) ~ 3%
- but
- P (female ; pregnant) >>>3%

## Example 1 :    Is coin fair ?

-   Toss coin:  5 consecutive tails
-    What is   P(unbiased; data) ?  i.e. p = 1⁄2
- Depends on Prior(p)
-    If village priest:        prior ~ δ(p = 1/2)
-    If stranger in pub:    prior ~ 1  for 0 < p <1
-                    (also needs cost function)

## Try to separate π’s and protons

- Try to separate π’s and protons
- probability (p tag; real p) = 0.95
- probability (π tag; real p) = 0.05
- probability (p tag; real π) = 0.10
- probability (π tag; real π) = 0.90

- Particle gives proton tag.  What is it?

- If proton beam,                          very likely
- If general secondary particles,  more even
- If pure π beam,                          ~ 0

- Depends on prior = fraction of protons

## Hunter and Dog

- Dog d has 50% probability of being 100 m. of Hunter h
- Hunter h has 50% probability of being within 100m of Dog d

- d

- h

- x

- River x =0

- River x =1 km

## Given that:       a) Dog d has 50% probability of being 100 m. of Hunter, 

- is it true that:    b) Hunter h has 50% probability of being within 100m of Dog d ?

| Additional information
 Rivers at zero & 1 km.  Hunter cannot cross them.  

 Dog can swim across river  - Statement a) still true |
| --- |
| If dog at –101 m, hunter cannot be within 100m of dog
Statement b) untrue |

## Slide 28

## Classical Approach

- Neyman “confidence interval” avoids pdf  for

- Uses only  P( x;    )

- Confidence interval                     :

- P(              contains      ) =

- True for any

- Varying intervals from ensemble of experiments

- fixed

- Gives range of     for which observed value     was “likely” (    )

- Contrast Bayes : Degree of belief =                  is in

## μ≥0                                No prior for μ

- μ≥0                                No prior for μ

## Frequentism: Specific example

- Particle decays exponentially:     dn/dt = (1/τ) exp(-t/τ)
- Observe 1 decay at time t1:         **L**(τ)  = (1/τ) exp(-t1/τ)
- Construct 68% central interval
-                                                     t = .17τ
- dn/dt
-                                            τ
-                  t
-                                                                                 t = 1.8τ
-                                                            t1               t

## 90% Classical interval for Gaussian

- σ = 1     μ ≥ 0      e.g. m2(νe)

## at 90% confidence

|         and          known, but random
                        unknown, but fixed 
Probability statement about         and 
               |
| --- |

- Frequentist

- Bayesian

|        and          known, and fixed

                        unknown, and random 
Probability/credible statement about 
 |  |
| --- | --- |

## Coverage

- Fraction of intervals containing true value
- Property of method, not of result
- Can vary with param
- Frequentist concept.  Built in to Neyman construction
- Some Bayesians reject idea. Coverage not guaranteed
- Integer data (Poisson)  discontinuities
-                                           Ideal coverage plot
-             C
-                                                 μ

## Classical Intervals

- Problems

| Hard to understand   e.g. d’Agostini e-mail
Arbitrary choice of interval
Possibility of empty range
Nuisance parameters  (systematic errors) |
| --- |

-  Advantages

| Widely applicable
Well defined coverage |
| --- |

## FELDMAN - COUSINS 

- Wants to avoid empty classical intervals     
- Uses “**L**-ratio ordering principle” to resolve ambiguity about “which 90% region?”    
-    [Neyman + Pearson say **L**-ratio is best for hypothesis testing]
- No ‘Flip-Flop’ problem

## Xobs = -2 now gives upper limit

- Xobs = -2 now gives upper limit

## Flip-flop

- Black lines      Classical 90% central interval
- Red dashed:   Classical 90% upper limit

## Slide 39

## Poisson confidence intervals.    Background = 3

-    Standard Frequentist               Feldman - Cousins

## Slide 41

## Slide 42

## Importance of Ordering Rule

- Neyman construction in 1 parameter

- 2 measurements

- An aside:  Determination of single parameter p via

- --------------

- Acceptable level of

| Range of parameters given by
Values of      for which data is likely  i.e. p(    ) is acceptable      or
                    |
| --- |

- 2)  is good
- 1) Range depends on

- [“Confidence interval coupled to goodness of fit”]

## *

- Neyman Construction

- For given    , acceptable (     ,     )    satisfy

- =

- Defines cylinder in                space

- Experiment gives                       interval

- Range depends on

- Range and goodness of fit are coupled

## That was using Probability Ordering

- Now change to Likelihood Ratio Ordering

- For              ,no value of       gives very good fit

- For Neyman Construction at fixed      , compare:

- with

- where

- giving

- Cutting on Likelihood Ratio Ordering gives:

## Therefore,  range of       is

- Constant Width
- Independent of

- Confidence Range and Goodness of Fit are completely decoupled

## Standard Frequentist

- _Pros:_
- Coverage
- Widely applicable

- _Cons:_
- Hard to understand
- Small or empty intervals
- Difficult in many variables (e.g. systematics)
- Needs ensemble

## Bayesian

- _Pros:_
- Easy to understand
- Physical interval

- _Cons:_
- Needs prior
- Coverage not guaranteed
- Hard to combine

## SYSTEMATICS

- For example

- Observed

- for statistical errors

- Physics parameter

- we need to know these, probably from other measurements  (and/or theory)
- Uncertainties error in

- Some are arguably statistical errors

- Shift Central Value
- Bayesian
- Frequentist
- Mixed

## Simplest Method

- Evaluate           using              and
- Move nuisance parameters (one at a time) by their errors  
- If nuisance parameters are uncorrelated,
- combine these contributions in quadrature
-                     total systematic

- Shift Nuisance Parameters

## Bayesian

- Without systematics

- prior

- With systematics

- Then integrate over LA and b

## If             = constant and               = truncated Gaussian TROUBLE!

- Upper limit on       from

- Significance from likelihood ratio for             and

## Frequentist

- Full Method
- Imagine just 2 parameters              and  LA
-               and 2 measurements   N  and   M

- Physics

- Nuisance

- Do Neyman construction in 4-D
- Use observed N and M, to give
- Confidence Region for LA and

- LA

- 68%

## Then project onto      axis

- This results in OVERCOVERAGE

- Aim to get better shaped region, by suitable choice of ordering rule

- Example:   Profile likelihood ordering

## Full frequentist method hard to apply in several dimensions

- Used in      3 parameters
-      For example:    Neutrino oscillations (CHOOZ)

- Normalisation of data

- Use approximate frequentist methods that reduce dimensions to just physics parameters
-      e.g.  _Profile_  pdf
-      i.e.

- Contrast Bayes marginalisation
- Distinguish “profile     _ordering_”

- See Giovanni Punzi, PHYSTAT05  page 88

## Limit of C. Lim. as σ  0

-               (March 2000)  by:
-               Gary Feldman
-               Wolfgang Rolke       hep-ph/0005187 version 2

- Acceptance uncertainty worse than Background uncertainty

- Limit of C. Lim. as σ  0

- Need to check Coverage

- Lim

- σ

## Method:  Mixed Frequentist - Bayesian

- Bayesian for nuisance parameters and
- Frequentist to extract range
- Philosophical/aesthetic problems?
- Highland and Cousins
- (Motivation was paradoxical behaviour of Poisson limit when LA not known exactly)

- Bayesian versus Frequentism

| Basis of method | Bayes Theorem 
Posterior probability distribution  | Uses pdf for data,
for fixed parameters |
| --- | --- | --- |
| Meaning of probability | Degree of belief | Frequentist definition |
| Prob of parameters? | Yes | Anathema |
| Needs prior? | Yes | No |
| Choice of interval? | Yes | Yes (except F+C) |
| Data considered | Only data you have | ....+ other possible data |
| Likelihood    principle? | Yes | No |

## Bayesian versus Frequentism

| Ensemble of experiment | No | Yes (but often not explicit) |
| --- | --- | --- |
| Final statement | Posterior probability distribution | Parameter values 
Data is likely |
| Unphysical/
empty ranges | Excluded by prior | Can occur |
| Systematics | Integrate over prior | Extend dimensionality of frequentist construction |
| Coverage | Unimportant | Built-in |
| Decision making | Yes (uses cost function) | Not useful |
|  |  |  |
|  |  |  |

- Bayesian                              Frequentist

## Bayesianism versus Frequentism

- “Bayesians address the question everyone is interested in, by using assumptions no-one believes”

- “Frequentists use impeccable logic to deal with an issue of no interest to anyone”