Repository: ICB-DCM/pyPESTO

Issue:
Add more tests to validation intervals
**Feature description**
It would be nice to check, if the prediction for `x_opt` leads to the significance `1` and if the significance approaches zero for  larger and larger deviations. (I had some problems with that for one application of mine, but I would not exclude that the issue there is outside pyPESTO) 


Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM) Report
> Merging [#676](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM) (1c08614) into [develop](https://codecov.io/gh/ICB-DCM/pyPESTO/commit/160c2a824419eaec6dd9ff66a242e495e5479567?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM) (160c2a8) will **decrease** coverage by `41.38%`.
> The diff coverage is `34.85%`.

[![Impacted file tree graph](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/graphs/tree.svg?width=650&height=150&src=pr&token=vaUsU0G1LY&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM)](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM)

```diff
@@             Coverage Diff              @@
##           develop     #676       +/-   ##
============================================
- Coverage    88.16%   46.78%   -41.39%     
============================================
  Files           79       96       +17     
  Lines         5257     6363     +1106     
============================================
- Hits          4635     2977     -1658     
- Misses         622     3386     +2764     
```


| [Impacted Files](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM) | Coverage Δ | |
|---|---|---|
| [pypesto/ensemble/covariance\_analysis.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by9lbnNlbWJsZS9jb3ZhcmlhbmNlX2FuYWx5c2lzLnB5) | `18.36% <0.00%> (-0.39%)` | :arrow_down: |
| [pypesto/objective/aesara.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by9vYmplY3RpdmUvYWVzYXJhLnB5) | `0.00% <0.00%> (ø)` | |
| [pypesto/optimize/\_\_init\_\_.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by9vcHRpbWl6ZS9fX2luaXRfXy5weQ==) | `100.00% <ø> (ø)` | |
| [pypesto/profile/profile.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by9wcm9maWxlL3Byb2ZpbGUucHk=) | `32.43% <0.00%> (-59.46%)` | :arrow_down: |
| [pypesto/sample/theano.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by9zYW1wbGUvdGhlYW5vLnB5) | `20.00% <0.00%> (-55.68%)` | :arrow_down: |
| [pypesto/startpoint/latin\_hypercube.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by9zdGFydHBvaW50L2xhdGluX2h5cGVyY3ViZS5weQ==) | `16.66% <0.00%> (-79.34%)` | :arrow_down: |
| [pypesto/visualize/parameters.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by92aXN1YWxpemUvcGFyYW1ldGVycy5weQ==) | `14.14% <0.00%> (-77.78%)` | :arrow_down: |
| [pypesto/visualize/waterfall.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by92aXN1YWxpemUvd2F0ZXJmYWxsLnB5) | `14.81% <0.00%> (-81.49%)` | :arrow_down: |
| [pypesto/store/read\_from\_hdf5.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by9zdG9yZS9yZWFkX2Zyb21faGRmNS5weQ==) | `44.96% <2.63%> (-12.94%)` | :arrow_down: |
| [pypesto/visualize/misc.py](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM#diff-cHlwZXN0by92aXN1YWxpemUvbWlzYy5weQ==) | `13.18% <11.11%> (-71.75%)` | :arrow_down: |
| ... and [119 more](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM). Last update [5cce409...1c08614](https://codecov.io/gh/ICB-DCM/pyPESTO/pull/676?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ICB-DCM).


Patch:
diff --git a/pypesto/profile/validation_intervals.py b/pypesto/profile/validation_intervals.py
index 15de409..c60a437 100644
--- a/pypesto/profile/validation_intervals.py
+++ b/pypesto/profile/validation_intervals.py
@@ -125,13 +125,13 @@ def validation_profile_significance(
     # => survival function chi.sf
     if return_significance:
         if lsq_objective:
-            return chi2.sf(nllh_new-nllh_old, 1)
+            return chi2.sf(nllh_old-nllh_new, 1)
         else:
-            return chi2.sf(2*(nllh_new-nllh_old), 1)
+            return chi2.sf(2*(nllh_old-nllh_new), 1)
     # compute the probability, that the validation data set is inside the CI
     # => cumulative density function chi.cdf
     else:
         if lsq_objective:
-            return chi2.cdf(nllh_new-nllh_old, 1)
+            return chi2.cdf(nllh_old-nllh_new, 1)
         else:
-            return chi2.cdf(2*(nllh_new-nllh_old), 1)
+            return chi2.cdf(2*(nllh_old-nllh_new), 1)


Test patch:
diff --git a/test/profile/test_validation_intervals.py b/test/profile/test_validation_intervals.py
index 623153c..a129b19 100644
--- a/test/profile/test_validation_intervals.py
+++ b/test/profile/test_validation_intervals.py
@@ -16,18 +16,18 @@ class ValidationIntervalTest(unittest.TestCase):
     @classmethod
     def setUp(cls):
 
-        lb = np.array([-1])
-        ub = np.array([5])
+        cls.lb = np.array([-1])
+        cls.ub = np.array([5])
 
         cls.problem_training_data = pypesto.Problem(
             lsq_residual_objective(0),
-            lb, ub)
+            cls.lb, cls.ub)
 
         cls.problem_all_data = pypesto.Problem(
             pypesto.objective.AggregatedObjective(
                 [lsq_residual_objective(0),
                  lsq_residual_objective(2)]),
-            lb, ub)
+            cls.lb, cls.ub)
 
         # optimum f(0)=0
         cls.result_training_data = optimize.minimize(cls.problem_training_data,
@@ -45,8 +45,58 @@ class ValidationIntervalTest(unittest.TestCase):
                                                 self.result_all_data)
 
         # fit with refitting inside function
-        profile.validation_profile_significance(self.problem_all_data,
-                                                self.result_training_data)
+        res_with_significance_true = profile.validation_profile_significance(
+            self.problem_all_data,
+            self.result_training_data)
+
+        # test return_significance = False leads the result to 1-significance
+        res_with_significance_false = profile.validation_profile_significance(
+            self.problem_all_data,
+            self.result_training_data,
+            return_significance=False)
+
+        self.assertAlmostEqual(1-res_with_significance_true,
+                               res_with_significance_false,
+                               places=4)
+
+        # test if significance=1, if the validation experiment coincides with
+        # the Max Likelihood prediction
+
+        problem_val = pypesto.Problem(pypesto.objective.AggregatedObjective(
+                [lsq_residual_objective(0),
+                 lsq_residual_objective(0)]),
+            self.lb, self.ub)
+
+        significance = profile.validation_profile_significance(
+            problem_val,
+            self.result_training_data)
+
+        self.assertAlmostEqual(significance, 1)
+
+        # Test if the significance approaches zero (monotonously decreasing)
+        # if the discrepancy between Max. Likelihood prediction and observed
+        # validation interval is increased
+
+        for d_val in [0.1, 1, 10, 100]:
+
+            # problem including d_val
+            problem_val = pypesto.Problem(
+                pypesto.objective.AggregatedObjective(
+                    [lsq_residual_objective(0),
+                     lsq_residual_objective(d_val)]),
+                self.lb, self.ub)
+
+            sig_d_val = profile.validation_profile_significance(
+                problem_val,
+                self.result_training_data)
+
+            # test, if the more extreme measurement leads to a more
+            # significant result
+            self.assertLessEqual(sig_d_val, significance)
+            significance = sig_d_val
+
+        # test if the significance approaches 0 for d_val=100
+        self.assertAlmostEqual(significance, 0)
 
 
 def lsq_residual_objective(d: float):