# Presentation: 331835

## Upper Limits and Priors

- James T. Linnemann
**Michigan State University**
- FNAL CL Workshop
- March 27, 2000

## P(contents|I, finish)prior probability or likelihood?

- Coverage of Cousins + Highand Limits
- mixed Frequentist + Bayesian
- Dependence of Bayesian UL on
- Signal “noninformative” priors
- Efficiency informative priors
- and comparison with C+H limits
- background informative priors
- Summary and Op/Ed Pages

## The Problem

- Observation:   see   **k**   events
- Poisson variable:
- expected mean is   **s+b**     (signal + background)
**s = *********L*******
- efficiency  Luminosity  cross section
- “cross section ” **** really cross section  branching ratio
- Calculate** U**, 95% upper limit on 
- function of k, b, and uncertainties ******b****, ************, *********L***
- focus on **upper limits: searches**

## Some typical cases forCalculation of 95% Upper Limits

- k=0, b=3		The Karmen Problem
- k=3, b=3		Standard Model Rules Again
- k=10, b=3		The Levitation of Gordy Kane?
- 				“seeing no excess, we proceed to 				set an upper limit...”

## The 95% Solution: Reverend Bayes to the Rescue

- Why?  He appeals to our theoretical side
- from statistics, we want “the answer”; as close as it gets?
- Why?  to handle nuisance parameters
***Name your poison***
- Tincture of Bayes
- Cousins and Highland treatment:
- Frequentist signals +  Bayesian nuisance
-  Bayes Full Strength
- The DØ nostrum:
- Both signal and nuisance parameters Bayesian

## Cousins & HighlandTrying to make everyone happy makes no one happy.Not even Bob.

- Treat **signal** in Frequentist fashion  (**counts**)
- Bayesian treatment of nuisance parameters
- 	 modifies probabilities entering signal distribution
**“****weighted average” over degree of belief in unknown parameters**
- Nota Bene
**This is how every physicist I know instinctively**
**approaches this problem.  It’s the “natural” way,**
**particularly when writing a Monte Carlo**

## C+H Coverage Monte Carlo:b=0; sensitivity uncertainty

- Fix true sensitivity,  in outer loop
- sweep through parameter space
- find % of experiments with limits including  at each point
- do MC experiments at each value
- pick observed value for sensitivity, k
- calculate limit based on these
- see if limit covers true value of 

## decreasing sensitivity >>

- increasing cross section >>

## Results for C+H Coverage

**Fails to cover** for large cross section and small efficiency.
- Not too surprising
- a count limit **s****U** could be due to any value of **** since **s****U**** = *********L*******
- if sensitivity small, would need a huge ******U**
- Remember, limit on ****** **must be valid for** **
**any sensitivity--no matter how improbable**
***coverage handles ******statistical fluctuations****** only***

## U = Bayes 95% Upper LimitsCredible Interval

- k =  number of events observed
- b = expected background
- Defined by integral on posterior probability
- Depends on prior probability for signal
- how to express that we don’t know if it exists,
- but would be willing to believe it does?
- 	***This is the Faustian part of the bargain!***
***Posterior: compromise likelihood with prior***

## Expected coverage of Bayesian intervals

- Theorem:<coverage> = 95% for Bayes 95% interval
- < > = average over (possible) true values weighted by prior
- Frequentist definition is_ minimum_ coverage for _any_ value of parameter (especially the true one!)
*not average coverage*
- Classic tech support:  precise, plausible, misleading
- if true for Poisson, why systematically under cover?
- Because k small is infinitely small part of [0,]
- but works beautifully for binomial (finite range)
- coverage varies with parameter but average is right on
- “obvious” if you do it with flat prior in parameter

## The sadness of Fred James: Jim, have you gone astray?

- I am indeed seen to worship at           		Reverend Bayes’ establishment
- I’m not a fully baptized member
- sorry Harrison, not that you haven’t tried!
- A skeptical inquirer...or a reluctant convert?
- Attraction of treating systematics is great
- Is accepting a Prior (*he’s uninformative!*) too high a price?
***	******	******      ******A solution for the tepid?***
- Can we substitute ***convention****** ***for ***conviction***?
- Either one should be _**examined for its consequences**__!_

## Candidate Signal Priors

**Flat**** **up to maximum M  (e.g. TOT)
- (our recommendation--but not invariant!)
- a convention for BR  cross section
**1/********s** (Jeffreys: reparameterization invariant)
- relatively popular “default” prior
**1/s** (one of Jeffreys’ recommendations)
- get expected posterior mean
- limit invariant under power transformation
**e****-****a****s**       not singular at s=0
- Bayes for combining with k=0 prev expt,
-      **a** = relative sensitivity to this experiment

## Slide 14

## sp Prior

- sp Prior

- flat

- Jeffreys

## Power Family sp Results (b=0)

- The **flat prior is not “special”** (stationary)
- But if b=0, Bayes UL =  Frequentist UL  coverage
- 	but lower limit would differ
**1/********s** gives **smaller limit (more weight to s=0)**
- less coverage than flat (though converges for k)
**1/s** gives you **0 **upper limit if b > 0
- too prejudiced towards 0 signal!
- More p dependence for k=0 than k=3 or k=10
- flat (p=0) to 1/s gives 36%, 26% , 6%
- data able to overwhelm prior (b=3)

## eas Prior

- eas Prior

- flat

- 2 expts

## flat

## Slide 19

## Exponential Family Results (b=0)

- Peak at s=0 pulls limit lower than flat prior
- effects larger than 1/s vs. flat: equivalent to *data*
- e-s gives you 1/2 the limit of flat (a=0) for k=0: 		combined 2 equal experiments
- biggest fractional effects on k=10   (=1/2.5)
- because disagrees with previous k=0 measurement
- opposite tendency of power family
- 	k=10 least dependent on power

## Dependence on Efficiency Informative Prior (representation of systematics)

- Input: **estimated efficiency** and **uncertainty**** **
**** uncertainty/estimate
- “efficiency” is really *L**          (*a **nuisance** parameter)
- Consider forms for efficiency prior
**Expect: less fractional dependence on form of prior**
- than on signal prior form
- because of the constraint of the input: informative
- study using flat prior for cross section, b=0
- Warning:    s = *L ** *     (multiplicative form)
- limit in s could mean low efficiency **or** high 

## Expressing      /

- “obvious” Truncated Gaussian (Normal)
- model for additive errors
- we recommend(ed)
- truncate so efficiency  0
**Lognormal** (Gaussian in Ln  )
- model for multiplicative errors
**Gamma**  		(Bayes conjugate prior)
- flat prior + estimate of Poisson variable
**Beta**  		(Bayes Conjugate prior)
- flat prior + estimate of Binomial variable

## Slide 23

## Slide 24

## Slide 25

## Slide 26

## Slide 27

## Results for Truncated Gaussian

- A bad choice, especially if  > .2 or so
- cutoff-dependent (MC: 4 sigma; calc .1<>)
- Otherwise depends on M, range of prior for 
- MC of course cranks out some answer
- dependent on luck, and cutoffs of generators
- WHY!? (same problem as with Coverage)
**Can’t set limit if possibility of no sensitivity**
- Probability of  =0 always finite for a truncated Gaussian
- with flat prior in , gives long tail in  posterior
- Bayes takes this literally:
- 	U reflects heavy weighting of large cross section!

## Slide 29

## Slide 30

## Slide 31

## Slide 32

## Results for alternatives ALL have P(=0) = 0 naturally

- Lognormal, beta, and gamma
**not very different (as expected--informative)**
_**opinion**_**: comparable to “choice of ensemble”**
- Not a Huge effect:
- 	U()/U(0) < 1+   up to   1/3
- .	.	.
- Lognormal, Gamma can be expressed as efficiency scaled to 1.0     (so can Gaussian)
- beta requires absolute scale (1-)j

## Slide 34

## Slide 35

## Slide 36

## Slide 37

## Slide 38

## Slide 39

## Results, compared with C+H (mixed Frequentist-Bayes)

- Truncated Gaussian well-behaved for C+H
- no flat prior to compound with P(=0) > 0 ?
- Fairly close to Bayes Lognormal
**C+H Limits depend on form** of informative prior MORE than Bayes
- 	Lognormal, gamma C+H lower than Bayes!
- C+H limits lower than Bayes limits
- Which is “better”?  coverage study?
- C+H Gaussian undercovers for small  (large )

## Dependence on Background Uncertainty

- Use flat prior, no efficiency uncertainty
- Use truncated Gaussian to represent <b>b
- But isn’t that a disaster?  No--
- additive is very different from multiplicative
- *L* **+ b**
- behavior at b=0 not special

## Slide 42

## Background Prior Results

- Result: **very mild dependence** on b/b
- < 10% change up to b/b = .66
- most sensitive for k=3, b=3; k=1, b=3
- 	absolute maximum: set b=0	20-40% typically
- 		set b=0:  force Frequentist coverage?
- No need to consider more complex models

## Paper in preparation

- With Harrison Prosper and Marc Paterno
- coverage calculation: more DØ help
- Thanks to Louis Lyons for the prod to finish
- and a 2nd chance at understanding all this
- only 1 hour jet lag, maybe I’ll be awake
- Poisson, Fisher....

## Summary (out of things to say)

- Cases studied:  b=3, k=0,3,10 mostly
- studies changed one thing at a time
***All****** ******Bayes upper****** limits ******seen to****** ******monotonically****** increase with uncertainties ***
***(couldn’t quite  ***_***prove***_***:***
***	******	****** ******Goedel’s Theorem for Dummies)***
- Hello PDG/RPP
- nuisance effects 15% or so--please advise us
- 	*ignoring them gives too-optimistic limits*

## Signal Prior Summary

- Flat signal prior a convention
- b=0, =0 matches Frequentist upper limit
- we still recommend it
- 				careful it’s not normalized
- flat vs 1/s **matters at 30% level** when setting limits
- 		So publish what you did!
-       Enough info to deduce **N****U****= ********U****/<*********L*****>** at one point
- 			can see if method or results differ 		**how about posting limits programs on web?**
- exponential family actually is a strong opinion (=data)

## Informative Prior Summary Can’t set limit if possibility of no sensitivity

- C+H mixed prescription doesn’t cover
- how well does Bayes do?  (“better”?)
- Efficiency informative prior **matters** in Bayesian
- at a **level of 10%** differences if you avoid Gaussian
- Prefer Lognormal over Truncated Gaussian
- Keep uncertainty under 30%  (large, ill-defined!)
- limit grows 20-30% for 30% fractional error in efficiency
- growth worse than quadratic
- Bayesian upper limits larger than C+H; more similar
- Publish what you did
- Background uncertainty weaker effect than efficiency
- typically < 15% even at b/b=1

## Is 20% difference in limits worth a religious war ...?(less of a problem if we actually find something!)

- Flat  Prior broadly useful in counting expts?
- Set limits on visible cross section U()
- signal MC for  ()
- stays as close as we can get to raw counts
- here is where scheme-dependence hits; it’s not too bad...
- 	 **resolution corrections, prior dependence ~ 20-30% or less**
- Interpret exclusion limits for :
- compare U to ()
- IF steep parameter dependence: less scheme-dependence
- 		in limits for  than U()...