Repository: scipy/scipy

Issue:
MAINT: Fix Box-Cox transform
* Fixes #6873
* Short-circuits MLE and Pearson in certain edge cases
* Adds test for Box-Cox transform for valid corner cases
* Adds test for Box-Cox transform for invalid corner cases


Discussion:
Just for the future reference, please make separate branches to work on new features and to send PRs such that your standard working repo does not interfere with the PRs you have submitted. Once a PR is merged you can safely delete that branch and keep working on other branches.
> Just for the future reference, please make separate branches to work on new features and to send PRs such that your standard working repo does not interfere with the PRs you have submitted. Once a PR is merged you can safely delete that branch and keep working on other branches.

Makes sense. Fortunately this PR is the only one I have in flight right now. Thanks!
Friendly ping on this PR. Hoping to have this merged soon so I can remove workarounds that cause SciPy to crash in my code.
I currently don't have time to review in detail, just a few comments:

- can you describe the cases you are trying to fix in more detail and the rationale for the solution?
  - if the transformation returns non-finite values, returning nan looks fine
  - if the transformed values have MSE of zero, why set the MSE to 1e-21 in that case? if all values are constant, I guess there is no optimal lambda?
Sure thing,

> can you describe the cases you are trying to fix in more detail and the rationale for the solution?

Intermediate values reach infinity due to division by zero or lack of enough precision in some scenarios. This simply means that the optimizer tried out a value that was not optimal, and letting the exception raise prevents the optimizer from searching further and returning a valid value.

> if the transformation returns non-finite values, returning nan looks fine

As far as I can tell, transformation should always return finite values and a valid lambda. In some edge cases the optimizer may not be able to find an optimal lambda (or there may not be an optimal lambda that makes sense, e.g. if all values are identical), but I don't think that returning nan is a valid output.

> if the transformed values have MSE of zero, why set the MSE to 1e-21 in that case? if all values are constant, I guess there is no optimal lambda?

Good question. I'm setting MSE to a very, very small number to prevent division by zero in the intermediate values, while letting the optimizer do its thing searching for a different value in whichever direction the result of that intermediate value tells it to go. If MSE is zero, it does not necessarily mean that there is no optimal lambda, only that the lambda that was tested in that iteration results in a transformed dataset that is equal to the input dataset. Unless I'm mistaken, that could be the most optimal way to transform the dataset, or there could be other lambdas yet to be tested by the optimizer that result in a better transformation.

Another thing that is being discussed, brought up by you initially and with concerns shared by @ilayn, is whether the output values should be checked. In the cases that I'm testing for, I'm only trying to make sure that the transformation succeeds. What transformation is expected in some corner cases -- some of which are identical to the corner cases that produce invalid intermediate values e.g. division by zero -- such as when all inputs are the same, is in my opinion undefined at the moment. Since the function in question is a statistical function, I would suggest to have a tighter definition of the expected output in those corner cases but I find that to be unrelated work to this PR which should be tested separately.
OK, thanks for the clarification, so llf should not raise an error, so the optimizer can continue to search for a solution.in one case (non-finite values), you return NaN, in the other case (MSE == 0), you tweak the MSE. Would NaN work in the latter case as well? (Might be more consistent)
I think that returning NaN would not always work, specifically in the case where the input is already all equal values. By tweaking the MSE, we are making sure that there is at least one valid output. We could short-circuit if all inputs are the same, but I'm afraid that there may be contrived non-equal inputs where the solver is unable to find a single good solution if NaN is returned for MSE == 0.

Returning NaN for non-finite values does not appear to be a conscious choice, and I think in that case we would be better off hardening the internal functions and preventing all exceptions by checking inputs and values along the way; then keep searching for a valid lambda or return NaN when the search space has been exhausted.

But that would require a much larger change. For the time being, this fix appears to work for all the reports of Box-Cox misbehaving that I have seen on #6873.
Added an additional test as requested. Failing test seems to be flaky, unrelated to this change.
Friendly ping. Is there anything left unaddressed that prevents this PR from being merged?
Just a mechanical comment -- you'll probably need to rebase to get the Appveyor failure resolved.
Addressed requested changes. Test failures appear to be unrelated to this change as they can be seen in other PRs and I can confirm that all tests pass when I run them locally (using a fresh Docker image).
@omtinez did you see my main comment in https://github.com/scipy/scipy/pull/9271#pullrequestreview-180631962 ("this issue isn't actually solved ...")? 
> @omtinez did you see my main comment in [#9271 (review)](https://github.com/scipy/scipy/pull/9271#pullrequestreview-180631962) ("this issue isn't actually solved ...")?

Sorry, I forgot to address that. Maybe I'm not grasping that specific issue fully, but I think that there may be more than one problem being discusses in that thread. At least the issue that I experienced, which was being unable to find an lmax altogether, is fixed with this patch.
> Maybe I'm not grasping that specific issue fully, but I think that there may be more than one problem being discusses in that thread. At least the issue that I experienced, which was being unable to find an lmax altogether, is fixed with this patch.

Probably, this PR does like an improvement. The `pearsonr` rewrite may help more though. I'm inclined to merge this but keep the issue open. Alternative is to merge the `pearsonr` rewrite first, and then revisit this. @WarrenWeckesser @chrisb83 thoughts?
The documentation of `boxcox_normmax` says the following:

> The starting interval for a downhill bracket search with optimize.brent. Note that this is in most cases not critical; the final result is allowed to be outside this bracket.

So the behaviour described by Ralf above appears to be fine.

Otherwise I cannot say much about the proposed solution, I don't fully understand the problem and I find the test cases confusing. For example, if a single value is supplied to boxcox, what should a meaningful output be? I would even say one should just raise an exception in that case.

Also, `assert_(max(output) - min(output) > 0)` does not appear to be a good conditon to check that the correct solution is found. Maybe use R as a benchmark (I could do that as a test in the next days.)?

I think the handling of pearsonr here needs to stay even if Warren's PR on pearson is fixed (i.e. raise an error if input is constant), whereas here the optimizer should just continue to search for a solution.


> For example, if a single value is supplied to boxcox, what should a meaningful output be? I would even say one should just raise an exception in that case.

In that case where is the line drawn? What about lists of size two? I prefer not to raise an exception since that adds complexity to the users of that function (which also would need to be documented properly.)

> Also, assert_(max(output) - min(output) > 0) does not appear to be a good conditon to check that the correct solution is found. Maybe use R as a benchmark (I could do that as a test in the next days.)?

I'm open to suggestions. The current test was written to support that the quoted issue was fixed as requested.
@rgommers will this PR be merged? Otherwise it should probably be closed as rejected since I won't be able to spend any more time on this.
> @rgommers will this PR be merged? Otherwise it should probably be closed as rejected since I won't be able to spend any more time on this.

yes I think we'll merge it, possibly with some modifications. it's a useful contributions, please leave it open for now.
Don't merge this.  I don't think this is the right way to solve #6873.  I'm working on a different solution that reformulates the log-likelihood function to avoid the subtraction that leads to the loss of precision.  I'll submit an alternative pull request soon.
Sigh... when will I learn?  @omtinez, the *first* thing I should have said in my previous comment was *thank you for working on this!*  It is always great to have people who care enough about the project to submit bug fixes and other improvements.

In this case, though, I think I've worked out a better way to solve the problem.  I'll explain the details when I submit a new pull request.

Resolving patch:
diff --git a/scipy/stats/morestats.py b/scipy/stats/morestats.py
--- a/scipy/stats/morestats.py
+++ b/scipy/stats/morestats.py
@@ -896,11 +896,18 @@ def boxcox_llf(lmb, data):
     if N == 0:
         return np.nan
 
-    y = boxcox(data, lmb)
-    y_mean = np.mean(y, axis=0)
-    llf = (lmb - 1) * np.sum(np.log(data), axis=0)
-    llf -= N / 2.0 * np.log(np.sum((y - y_mean)**2. / N, axis=0))
-    return llf
+    logdata = np.log(data)
+
+    # Compute the variance of the transformed data.
+    if lmb == 0:
+        variance = np.var(logdata, axis=0)
+    else:
+        # Transform without the constant offset 1/lmb.  The offset does
+        # not effect the variance, and the subtraction of the offset can
+        # lead to loss of precision.
+        variance = np.var(data**lmb / lmb, axis=0)
+
+    return (lmb - 1) * np.sum(logdata, axis=0) - N/2 * np.log(variance)
 
 
 def _boxcox_conf_interval(x, lmax, alpha):


Test patch:
diff --git a/scipy/stats/tests/test_morestats.py b/scipy/stats/tests/test_morestats.py
--- a/scipy/stats/tests/test_morestats.py
+++ b/scipy/stats/tests/test_morestats.py
@@ -1211,6 +1211,66 @@ def test_2d_input(self):
     def test_empty(self):
         assert_(np.isnan(stats.boxcox_llf(1, [])))
 
+    def test_gh_6873(self):
+        # Regression test for gh-6873.
+        # This example was taken from gh-7534, a duplicate of gh-6873.
+        data = [198.0, 233.0, 233.0, 392.0]
+        llf = stats.boxcox_llf(-8, data)
+        # The expected value was computed with mpmath.
+        assert_allclose(llf, -17.93934208579061)
+
+
+# This is the data from github user Qukaiyi, given as an example
+# of a data set that caused boxcox to fail.
+_boxcox_data = [
+    15957, 112079, 1039553, 711775, 173111, 307382, 183155, 53366, 760875,
+    207500, 160045, 473714, 40194, 440319, 133261, 265444, 155590, 36660,
+    904939, 55108, 138391, 339146, 458053, 63324, 1377727, 1342632, 41575,
+    68685, 172755, 63323, 368161, 199695, 538214, 167760, 388610, 398855,
+    1001873, 364591, 1320518, 194060, 194324, 2318551, 196114, 64225, 272000,
+    198668, 123585, 86420, 1925556, 695798, 88664, 46199, 759135, 28051,
+    345094, 1977752, 51778, 82746, 638126, 2560910, 45830, 140576, 1603787,
+    57371, 548730, 5343629, 2298913, 998813, 2156812, 423966, 68350, 145237,
+    131935, 1600305, 342359, 111398, 1409144, 281007, 60314, 242004, 113418,
+    246211, 61940, 95858, 957805, 40909, 307955, 174159, 124278, 241193,
+    872614, 304180, 146719, 64361, 87478, 509360, 167169, 933479, 620561,
+    483333, 97416, 143518, 286905, 597837, 2556043, 89065, 69944, 196858,
+    88883, 49379, 916265, 1527392, 626954, 54415, 89013, 2883386, 106096,
+    402697, 45578, 349852, 140379, 34648, 757343, 1305442, 2054757, 121232,
+    606048, 101492, 51426, 1820833, 83412, 136349, 1379924, 505977, 1303486,
+    95853, 146451, 285422, 2205423, 259020, 45864, 684547, 182014, 784334,
+    174793, 563068, 170745, 1195531, 63337, 71833, 199978, 2330904, 227335,
+    898280, 75294, 2011361, 116771, 157489, 807147, 1321443, 1148635, 2456524,
+    81839, 1228251, 97488, 1051892, 75397, 3009923, 2732230, 90923, 39735,
+    132433, 225033, 337555, 1204092, 686588, 1062402, 40362, 1361829, 1497217,
+    150074, 551459, 2019128, 39581, 45349, 1117187, 87845, 1877288, 164448,
+    10338362, 24942, 64737, 769946, 2469124, 2366997, 259124, 2667585, 29175,
+    56250, 74450, 96697, 5920978, 838375, 225914, 119494, 206004, 430907,
+    244083, 219495, 322239, 407426, 618748, 2087536, 2242124, 4736149, 124624,
+    406305, 240921, 2675273, 4425340, 821457, 578467, 28040, 348943, 48795,
+    145531, 52110, 1645730, 1768364, 348363, 85042, 2673847, 81935, 169075,
+    367733, 135474, 383327, 1207018, 93481, 5934183, 352190, 636533, 145870,
+    55659, 146215, 73191, 248681, 376907, 1606620, 169381, 81164, 246390,
+    236093, 885778, 335969, 49266, 381430, 307437, 350077, 34346, 49340,
+    84715, 527120, 40163, 46898, 4609439, 617038, 2239574, 159905, 118337,
+    120357, 430778, 3799158, 3516745, 54198, 2970796, 729239, 97848, 6317375,
+    887345, 58198, 88111, 867595, 210136, 1572103, 1420760, 574046, 845988,
+    509743, 397927, 1119016, 189955, 3883644, 291051, 126467, 1239907, 2556229,
+    411058, 657444, 2025234, 1211368, 93151, 577594, 4842264, 1531713, 305084,
+    479251, 20591, 1466166, 137417, 897756, 594767, 3606337, 32844, 82426,
+    1294831, 57174, 290167, 322066, 813146, 5671804, 4425684, 895607, 450598,
+    1048958, 232844, 56871, 46113, 70366, 701618, 97739, 157113, 865047,
+    194810, 1501615, 1765727, 38125, 2733376, 40642, 437590, 127337, 106310,
+    4167579, 665303, 809250, 1210317, 45750, 1853687, 348954, 156786, 90793,
+    1885504, 281501, 3902273, 359546, 797540, 623508, 3672775, 55330, 648221,
+    266831, 90030, 7118372, 735521, 1009925, 283901, 806005, 2434897, 94321,
+    309571, 4213597, 2213280, 120339, 64403, 8155209, 1686948, 4327743,
+    1868312, 135670, 3189615, 1569446, 706058, 58056, 2438625, 520619, 105201,
+    141961, 179990, 1351440, 3148662, 2804457, 2760144, 70775, 33807, 1926518,
+    2362142, 186761, 240941, 97860, 1040429, 1431035, 78892, 484039, 57845,
+    724126, 3166209, 175913, 159211, 1182095, 86734, 1921472, 513546, 326016,
+    1891609
+]
 
 class TestBoxcox(object):
 
@@ -1266,6 +1326,14 @@ def test_boxcox_bad_arg(self):
     def test_empty(self):
         assert_(stats.boxcox([]).shape == (0,))
 
+    def test_gh_6873(self):
+        # Regression test for gh-6873.
+        y, lam = stats.boxcox(_boxcox_data)
+        # The expected value of lam was computed with the function
+        # powerTransform in the R library 'car'.  I trust that value
+        # to only about five significant digits.
+        assert_allclose(lam, -0.051654, rtol=1e-5)
+
 
 class TestBoxcoxNormmax(object):
     def setup_method(self):