Repository: gammapy/gammapy

Issue:
What's the correct value for errordef in iminuit?
see https://github.com/gammapy/gammapy/blob/master/gammapy/utils/fitting/iminuit.py#L88

Reminder:
out fit statistic is -2 log (likelihood)
https://github.com/gammapy/gammapy/blob/master/gammapy/stats/fit_statistics.py#L58


Discussion:
I'm moving this to v0.9. Since our likelihood contains the factor 2, `errordef=1` should be correct.

One way to check is to simulate and fit a simple model ~ 100 times, and to check the pull distribution (`(par_fitted - par_true) / par_error`). If it's a unit normal distribution, `errordef` is OK, if it has a different width, `errordef` is incorrect. The model could just have one parameter, like a flux amplitude, or it could have several parameters as in http://docs.gammapy.org/dev/notebooks/simulate_3d.html

@AtreyeeS or @facero or anyone - Do you have time do make a notebook with this check (e.g. pasted in a https://gist.github.com/)?
I can try this check... maybe later today or early tomorrow
@AtreyeeS - Thanks!
I assigned the issue to you.

Maybe you could also send a PR that removes the TODO comment @joleroi linked to above,
and adds a few asserts on the parameter errors in this test? (low precision, with `rtol=1e-2` like we have for the parameter values)? Then we'll have a regression test.

https://github.com/gammapy/gammapy/blob/c886a411f24a011459ca4a3a8d701180ea8e1f19/gammapy/cube/tests/test_fit.py#L108
Hello, 

I tried fitting with two free parameter, index and the longitude.
The pull distribution is NOT a unit normal.
I am not sure if this is expected or there is some problem.

The (mean, sigma) of pull_index = (0.44, 1.09)
(mean,sigma) of pull_lon = (-6.96, 0.61)
![pull_ind](https://user-images.githubusercontent.com/32677370/42945332-6bbbdcbe-8b68-11e8-8995-89260ad6d6d3.png)
![pull_lon](https://user-images.githubusercontent.com/32677370/42945333-6bd7d432-8b68-11e8-9594-8b46c7ab3898.png)

The code is here: https://gist.github.com/AtreyeeS/d5161ebe30f29c1a19b34757a1916301

@AtreyeeS - I now start editing the `make_minuit_par_kwargs` and also moving this `kwargs['errordef'] = 1` line. Hope I'm not interfering with edits you've made in the past days...
Please go ahead! I didn't make any changes to the main repo

Resolving patch:
diff --git a/gammapy/utils/fitting/iminuit.py b/gammapy/utils/fitting/iminuit.py
--- a/gammapy/utils/fitting/iminuit.py
+++ b/gammapy/utils/fitting/iminuit.py
@@ -36,6 +36,10 @@ def fit_iminuit(parameters, function, opts_minuit=None):
         opts_minuit = {}
     opts_minuit.update(make_minuit_par_kwargs(parameters))
 
+    # In Gammapy, we have the factor 2 in the likelihood function
+    # This means `errordef=1` in the Minuit interface is correct
+    opts_minuit.setdefault('errordef', 1)
+
     minuit = Minuit(minuit_func.fcn,
                     forced_parameters=parameters.names,
                     **opts_minuit)
@@ -75,8 +79,7 @@ def make_minuit_par_kwargs(parameters):
     kwargs = {}
     for par in parameters.parameters:
         kwargs[par.name] = par.value
-        if par.frozen:
-            kwargs['fix_{}'.format(par.name)] = True
+
         min_ = None if np.isnan(par.min) else par.min
         max_ = None if np.isnan(par.max) else par.max
         kwargs['limit_{}'.format(par.name)] = (min_, max_)
@@ -86,8 +89,8 @@ def make_minuit_par_kwargs(parameters):
         else:
             kwargs['error_{}'.format(par.name)] = parameters.error(par.name)
 
-        # TODO: Check if we need 0.5 or 1
-        kwargs['errordef'] = 1
+        if par.frozen:
+            kwargs['fix_{}'.format(par.name)] = True
 
     return kwargs
 


Test patch: