Repository: scikit-hep/pyhf

Issue:
iminuit v1.5.0 breaks optimization tests
# Description

With the release of [`iminuit` `v1.5.0`](https://github.com/scikit-hep/iminuit/releases/tag/v1.5.0) on 2020-09-17 the nightly tests are failing in `test_optim.py`. Specifically

https://github.com/scikit-hep/pyhf/blob/8a6ee36da4f566d8a37df01e20201098aa1f8a54/tests/test_optim.py#L47

is failing with errors of 

```pytb
        try:
            assert result.success
        except AssertionError:
            log.error(result)
>           raise exceptions.FailedMinimization(result)
E           pyhf.exceptions.FailedMinimization: Optimization failed. Estimated distance to minimum too large.

src/pyhf/optimize/mixins.py:52: FailedMinimization
------------------------------ Captured log call -------------------------------
ERROR    pyhf.optimize.mixins:mixins.py:51       fun: 15.5887451171875
 hess_inv: array([[1., 1.],
       [1., 1.]])
  message: 'Optimization failed. Estimated distance to minimum too large.'
   minuit: <iminuit._libiminuit.Minuit object at 0x5619c82f90a0>
     nfev: 110
     njev: 0
  success: False
      unc: None
        x: array([0.97325551, 0.91712703])
```

where the `pyhf.exceptions.FailedMinimization` being raised comes from the `raise exceptions.FailedMinimization(result)` in

https://github.com/scikit-hep/pyhf/blob/8a6ee36da4f566d8a37df01e20201098aa1f8a54/src/pyhf/optimize/mixins.py#L31-L53

which are of course coming from

https://github.com/scikit-hep/pyhf/blob/8a6ee36da4f566d8a37df01e20201098aa1f8a54/src/pyhf/optimize/opt_minuit.py#L122-L132

in

https://github.com/scikit-hep/pyhf/blob/8a6ee36da4f566d8a37df01e20201098aa1f8a54/src/pyhf/optimize/opt_minuit.py#L69

# Steps to Reproduce

Run the tests using current master.

![iminuit_breaks](https://user-images.githubusercontent.com/5142394/93731644-8ba78f80-fb93-11ea-8060-9e05dc61b102.png)

To show that this is definitley an issue with `iminuit` `v1.5.0+`

```
$ python -m pip install --upgrade "iminuit<1.5.0"
$ pip list | grep iminuit
iminuit                       1.4.9
$ python -m pytest -sx tests/test_optim.py
```

passes but

```
$ python -m pip install --upgrade iminuit
$ pip list | grep iminuit
iminuit                       1.5.1
$ python -m pytest -sx tests/test_optim.py
```

fails.

# Checklist

- [x] Run `git fetch` to get the most up to date version of `master`
- [x] Searched through existing Issues to confirm this is not a duplicate issue
- [x] Filled out the Description, Expected Behavior, Actual Behavior, and Steps to Reproduce sections above or have edited/removed them in a way that fully describes the issue



Discussion:
Tagging @HDembinski on this not because I expect them to do anything (this is a `pyhf` test issue) but just to give them a heads up that we might not be the only ones with tests that are failing &mdash; so there might be incoming issues on `iminuit` from other projects.
Not sure what could have caused this in iminuit. What exactly is failing?
> Not sure what could have caused this in iminuit. What exactly is failing?

These 8 test parameterizations of https://github.com/scikit-hep/pyhf/blob/8a6ee36da4f566d8a37df01e20201098aa1f8a54/tests/test_optim.py#L47

- `no_grad-minuit-pytorch-32b-no_stitch`
- `no_grad-minuit-pytorch-32b-do_stitch`
- `no_grad-minuit-tensorflow-32b-no_stitch`
- `no_grad-minuit-tensorflow-32b-do_stitch`
- `do_grad-minuit-pytorch-32b-no_stitch`
- `do_grad-minuit-pytorch-32b-do_stitch`
- `do_grad-minuit-tensorflow-32b-no_stitch`
- `do_grad-minuit-tensorflow-32b-do_stitch`

which are essentially the `32b` `iminuit` optimizer tests for PyTorch and TensorFlow. Both the `no_grad`

https://github.com/scikit-hep/pyhf/blob/8a6ee36da4f566d8a37df01e20201098aa1f8a54/tests/test_optim.py#L85-L86

and `do_grad` 

https://github.com/scikit-hep/pyhf/blob/8a6ee36da4f566d8a37df01e20201098aa1f8a54/tests/test_optim.py#L95-L96

are failing with the `'Optimization failed. Estimated distance to minimum too large.'` errors seen above in the Issue report body.

I can follow up more on this when it is morning in my timezone.
@HDembinski This is (strangely?) related to https://github.com/scikit-hep/iminuit/pull/462 it seems. If I change `setup.py` to require `iminuit~=1.4.0` (to get `v1.4.9`) and then in CI update `iminuit` to the digest for `iminuit` PR 461:

```yaml
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip setuptools wheel
        python -m pip install --ignore-installed -U -q --no-cache-dir -e .[test] # So have iminuit v1.4.9 here
        python -m pip install cython
        python -m pip install --upgrade --no-cache-dir git+git://github.com/scikit-hep/iminuit.git@0a035e8da276bd64a106f4b10537fde884ae8eba
```

then the CI will pass, but if I change that to the digest for `iminuit` PR 462

```
python -m pip install --upgrade --no-cache-dir git+git://github.com/scikit-hep/iminuit.git@31799ac188798ad8166375cc4c3a0cfcb2825995
```

the CI will fail with the error described above.
Something's off for sure as we don't rely on `nsplit`. If somehow dropping `nsplit` changed the minimum distance calculation - then this is the issue. We probably need a way to bump up the calculation to make it happen...

@matthewfeickert does increasing `ncalls` in the tests work? We do default to 100k iterations which is... a large number. Especially since for these tests, minuit converged in < 100 in the past... To change, something like `pyhf.optimizer.minuit_optimizer(maxiter=1e6)` would change the ncalls.
Dropping nsplit should not change the minimization in any way. nsplit is something that Piti added to artificially interrupt Migrad to get intermediate output. It was an ugly hack and not very useful, removing it simplified the code a lot.
The new version passes all the iminuit tests, including minimization tests on difficult functions, so I am not sure what is going on here.
In any case, I will have a sharp look at the code again.
> The new version passes all the iminuit tests, including minimization tests on difficult functions, so I am not sure what is going on here.

That's what I'm confused about here. These tests we're doing are using rather simple likelihoods where we have

```
    m = pyhf.simplemodels.hepdata_like([50.0], [100.0], [10.0])
    data = pyhf.tensorlib.astensor([125.0] + m.config.auxdata)
```

This is where we expect the test to give us parameter values of `[mu=0.5, bkg_uncert=1.0]`. For 64b it looks like we have no failures -- but when we pass in 32b straight into minuit, I guess there's no more support there, or it's unable to do the minimization at that float precision?
It would be useful to inspect the FMin object, you can get it from the minimize output as result["minuit"].fmin.

If the error message you got is correct, then increasing the call limit should not help. Minuit should complain with a different message when the limit is reached. It basically says that convergence could not be reached. Usually that is caused by numerical issues in the cost function.
dumping here for now (@HDembinski -- also providing code so you can explore as well. We have a nice api for getting the minuit object back out).

```python
Python 3.7.3 (default, Mar 27 2019, 09:23:32) 
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyhf
>>> pyhf.version.__version__
'0.5.2'
>>> import iminuit
>>> iminuit.version.__version__
'1.4.9'
>>> pyhf.set_backend(pyhf.tensor.pytorch_backend(precision='32b'), 'minuit')
>>> m = pyhf.simplemodels.hepdata_like([50.0], [100.0], [10.0])
>>> data = pyhf.tensorlib.astensor([125.0] + m.config.auxdata)
>>> try:
...   result, obj = pyhf.infer.mle.fit(data, m, do_grad=False, do_stitch=False, return_result_obj=True)
... except pyhf.exceptions.FailedMinimization as e:
...   obj = e.result
>>> obj.minuit.fmin
FMin(fval=14.68798828125, edm=3.4475309051665455e-05, tolerance=0.1, nfcn=145, ncalls=255, up=1.0, is_valid=True, has_valid_parameters=True, has_accurate_covar=True, has_posdef_covar=True, has_made_posdef_covar=False, hesse_failed=False, has_covariance=True, is_above_max_edm=False, has_reached_call_limit=False)
```

failure

```python
Python 3.7.3 (default, Mar 27 2019, 09:23:32) 
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyhf
>>> pyhf.version.__version__
'0.5.2'
>>> import iminuit
>>> iminuit.version.__version__
'1.5.0'
>>> pyhf.set_backend(pyhf.tensor.pytorch_backend(precision='32b'), 'minuit')
>>> m = pyhf.simplemodels.hepdata_like([50.0], [100.0], [10.0])
>>> data = pyhf.tensorlib.astensor([125.0] + m.config.auxdata)
>>> try:
...   result, obj = pyhf.infer.mle.fit(data, m, do_grad=False, do_stitch=False, return_result_obj=True)
... except pyhf.exceptions.FailedMinimization as e:
...   obj = e.result
>>> obj.minuit.fmin
FMin(fval=15.5887451171875, edm=300.64070491474513, tolerance=0.1, nfcn=99, ncalls=110, up=1.0, is_valid=False, has_valid_parameters=True, has_accurate_covar=False, has_posdef_covar=False, has_made_posdef_covar=True, hesse_failed=False, has_covariance=True, is_above_max_edm=True, has_reached_call_limit=False, has_parameters_at_limit=False)
```

certainly some differences. Notably, the `edm` is significantly larger.
here's a quick script that runs both precision values:

```python
import pyhf
print(f'pyhf.version   = {pyhf.version.__version__}')
import iminuit
print(f'iminuit.version = {iminuit.version.__version__}')


for precision in ['32b', '64b']:
    print(f'precision = {precision}')
    pyhf.set_backend(pyhf.tensor.pytorch_backend(precision=precision), 'minuit')
    m = pyhf.simplemodels.hepdata_like([50.0], [100.0], [10.0])
    data = pyhf.tensorlib.astensor([125.0] + m.config.auxdata)
    try:
      result, obj = pyhf.infer.mle.fit(data, m, do_grad=False, do_stitch=False, return_result_obj=True)
    except pyhf.exceptions.FailedMinimization as e:
      obj = e.result
    print(obj.minuit.fmin)
```

```
$ python minuit.py 
pyhf.version   = 0.5.2
iminuit.version = 1.4.9
precision = 32b
------------------------------------------------------------------
| FCN = 14.69                   |     Ncalls=145 (255 total)     |
| EDM = 3.45e-05 (Goal: 0.0002) |            up = 1.0            |
------------------------------------------------------------------
|  Valid Min.   | Valid Param.  | Above EDM | Reached call limit |
------------------------------------------------------------------
|     True      |     True      |   False   |       False        |
------------------------------------------------------------------
| Hesse failed  |   Has cov.    | Accurate  | Pos. def. | Forced |
------------------------------------------------------------------
|     False     |     True      |   True    |   True    | False  |
------------------------------------------------------------------
precision = 64b
------------------------------------------------------------------
| FCN = 13.11                   |      Ncalls=43 (43 total)      |
| EDM = 6.95e-08 (Goal: 0.0002) |            up = 1.0            |
------------------------------------------------------------------
|  Valid Min.   | Valid Param.  | Above EDM | Reached call limit |
------------------------------------------------------------------
|     True      |     True      |   False   |       False        |
------------------------------------------------------------------
| Hesse failed  |   Has cov.    | Accurate  | Pos. def. | Forced |
------------------------------------------------------------------
|     False     |     True      |   True    |   True    | False  |
------------------------------------------------------------------
```

versus 1.5.0

```
$ python minuit.py 
pyhf.version   = 0.5.2
iminuit.version = 1.5.0
precision = 32b
┌──────────────────────────────────┬──────────────────────────────────────┐
│ FCN = 15.59                      │       Ncalls = 99 (110 total)        │
│ EDM = 301 (Goal: 0.0002)         │               up = 1.0               │
├───────────────┬──────────────────┼──────────────────────────────────────┤
│INVALID Minimum│ Valid Parameters │        No Parameters at limit        │
├───────────────┴──────────────────┼──────────────────────────────────────┤
│          ABOVE EDM goal          │           Below call limit           │
├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤
│   Hesse ok    │  Has Covariance  │APPROXIMATE│NOT pos. def.│   FORCED   │
└───────────────┴──────────────────┴───────────┴─────────────┴────────────┘
precision = 64b
┌──────────────────────────────────┬──────────────────────────────────────┐
│ FCN = 13.11                      │        Ncalls = 43 (43 total)        │
│ EDM = 6.95e-08 (Goal: 0.0002)    │               up = 1.0               │
├───────────────┬──────────────────┼──────────────────────────────────────┤
│ Valid Minimum │ Valid Parameters │        No Parameters at limit        │
├───────────────┴──────────────────┼──────────────────────────────────────┤
│          Below EDM goal          │           Below call limit           │
├───────────────┬──────────────────┼───────────┬─────────────┬────────────┤
│   Hesse ok    │  Has Covariance  │ Accurate  │  Pos. def.  │ Not forced │
└───────────────┴──────────────────┴───────────┴─────────────┴────────────┘
```
MINUIT expects the calculation to be accurate to double precision. I expect the calculation in 32 bit to fail, actually. The question is rather why it was working before.
Note that you can override the precision that MINUIT assumes by default with the precision keyword in Minuit.migrad. I have never tried this myself, though...
Ok, I think I get it now. The original code by Piti called Migrad in a loop until either the call limit is reached or a valid minimum is found. Basically it was overriding the stopping condition and went on to "try harder". I am not super happy with this idea, because it hides problems such as this, but apparently it was working and people rely on this feature.
> Ok, I think I get it now. The original code by Piti called Migrad in a loop until either the call limit is reached or a valid minimum is found. Basically it was overriding the stopping condition and went on to "try harder". I am not super happy with this idea, because it hides problems such as this, but apparently it was working and people rely on this feature.

We would be happy to have this be a configurable option. At least to have a "try_harder=True" or something like this. Can you point to the code changes that were affecting this? I looked through the linked PR but had a hard time figuring out how it might have changed.
Thanks for taking the time to investigate this @HDembinski.

> We would be happy to have this be a configurable option. At least to have a "try_harder=True" or something like this.

I like @kratsg's suggestion here.

> Can you point to the code changes that were affecting this? I looked through the linked PR but had a hard time figuring out how it might have changed.

Same. We're happy to try to help out here if possible/needed.

@HDembinski Also, just a note. With PR #1071 we now have a nightly test of the `HEAD` of `iminuit`'s default branch (`develop`) with our test suite so we should detect in the future if there are any changes to `iminuit` that affect us far in advance of any release.
The test against develop is good, thanks. We need to cooperate a bit on bringing the old behavior back. I will try to check out pyhf to use its test in my debug cycle.
If you have the time, it would be great if you could try to set precision manually to something like float32 epsilon, to see whether that restores convergence in the 32b case.
> If you have the time, it would be great if you could try to set precision manually to something like float32 epsilon, to see whether that restores convergence in the 32b case.

Okay. I'll try to get to that this evening, but probably won't be able to do much testing before that sadly.
I am working on restoring something like the old behavior in iminuit. I can also tell you that running migrad() several times in a row fixes the failing test.

@kratsg 
> Can you point to the code changes that were affecting this?

https://github.com/scikit-hep/iminuit/commit/31799ac188798ad8166375cc4c3a0cfcb2825995#diff-555f11eca5f91abf962969c058e0edebL886-L893
Since I referenced your issue in my PR, it was automatically closed. Please confirm that develop works again.
> Since I referenced your issue in my PR, it was automatically closed. Please confirm that develop works again.

I just manually triggered a run of the "HEAD of dependencies" workflow and now there are only two tests still failing with the `HEAD` of `iminuit`: https://github.com/scikit-hep/pyhf/actions/runs/267372882

- `test_minimize[no_grad-minuit-tensorflow-32b-no_stitch]`
- `test_minimize[no_grad-minuit-tensorflow-32b-do_stitch]`

The rest are passing now. 
I cannot confirm this with my local checkout of pyhf. All tests run through for me.
I currently run 3 iterations before giving up, the old code potentially ran up to infinitely many. I will increase to 5 in this patch
https://github.com/scikit-hep/iminuit/pull/487
Let's see whether that helps.
We likely need to patch pyhf to set the 32b float mode when running over 32b. This may be the last piece needed on our side. I think 3 is enough, but not sure.
I just increased the number of iterations to 5 in the develop branch.
> I just increased the number of iterations to 5 in the develop branch.

With that (https://github.com/scikit-hep/iminuit/pull/487) `iminuit` now passes: https://github.com/scikit-hep/pyhf/actions/runs/269087634

> We likely need to patch pyhf to set the 32b float mode when running over 32b. This may be the last piece needed on our side. I think 3 is enough, but not sure.

@HDembinski If it would be preferable to have a lower number of iterations, we can work on the above as @kratsg mentioned.


No it is fine. Glad to hear that fixed it.
> No it is fine. Glad to hear that fixed it.

Thanks very much for your help here @HDembinski. You were very fast to iterate and we greatly appreciate you being willing to make changes to `iminuit` on our behalf.

We'll also lift the restrictions added in PR #1070 after the next release. :+1: (We can have that PR close this Issue).

@kratsg we still might want to revisit

> set[ing] the 32b float mode when running over 32

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@
     'torch': ['torch~=1.2'],
     'jax': ['jax~=0.2.4', 'jaxlib~=0.1.56'],
     'xmlio': ['uproot3~=3.14'],  # Future proof against uproot4 API changes
-    'minuit': ['iminuit~=1.4.3'],  # v1.5.0 breaks pyhf for 32b TensorFlow and PyTorch
+    'minuit': ['iminuit~=1.5.3'],
 }
 extras_require['backends'] = sorted(
     set(
diff --git a/src/pyhf/optimize/opt_minuit.py b/src/pyhf/optimize/opt_minuit.py
--- a/src/pyhf/optimize/opt_minuit.py
+++ b/src/pyhf/optimize/opt_minuit.py
@@ -113,7 +113,7 @@ def _minimize(
         minimizer.tol = tolerance
         minimizer.migrad(ncall=maxiter)
         # Following lines below come from:
-        # https://github.com/scikit-hep/iminuit/blob/22f6ed7146c1d1f3274309656d8c04461dde5ba3/src/iminuit/_minimize.py#L106-L125
+        # https://github.com/scikit-hep/iminuit/blob/64acac11cfa2fb91ccbd02d1b3c51f8a9e2cc484/src/iminuit/_minimize.py#L102-L121
         message = "Optimization terminated successfully."
         if not minimizer.valid:
             message = "Optimization failed."
@@ -141,7 +141,7 @@ def _minimize(
             fun=minimizer.fval,
             hess_inv=hess_inv,
             message=message,
-            nfev=minimizer.ncalls,
-            njev=minimizer.ngrads,
+            nfev=minimizer.ncalls_total,
+            njev=minimizer.ngrads_total,
             minuit=minimizer,
         )


Test patch:
diff --git a/tests/test_optim.py b/tests/test_optim.py
--- a/tests/test_optim.py
+++ b/tests/test_optim.py
@@ -420,20 +420,6 @@ def test_minuit_failed_optimization(
     monkeypatch, mocker, has_reached_call_limit, is_above_max_edm
 ):
     class BadMinuit(iminuit.Minuit):
-        @classmethod
-        def from_array_func(cls, *args, **kwargs):
-            """
-            from_array_func won't need mocker in a newer version of iminuit
-
-            See scikit-hep/iminuit#464 for more details
-            """
-            self = super().from_array_func(*args, **kwargs)
-            mock = mocker.MagicMock(wraps=self)
-            mock.valid = False
-            mock.fmin.has_reached_call_limit = has_reached_call_limit
-            mock.fmin.is_above_max_edm = is_above_max_edm
-            return mock
-
         @property
         def valid(self):
             return False