Repository: scipy/scipy

Issue:
Request: transformation functions - Yeo-Johnson
It would be great if you could expand on the transformation functions. 

At the moment BoxCox, center, scale and range are supported and a handful other from scikit-learn. However, some popular models, such as Yeo-Johnson, are missing.

Thanks.

References:
Kuhn and Johnson (2013), Applied Predictive Modeling, Springer
Kuhn (2008), Building predictive models in R using the caret 



Discussion:
As discussed on SO, http://stackoverflow.com/questions/37037837/python-library-yeo-johnson-tranformation, this is quite easy to do from the user side. You might need to add some motivation of why this is a worthy add to the library. If this is reasonably widely used, we can add it; if it's a one-shot thing, then it belongs to the user code.

Thanks for the quick reply. 
I agree with your view, it is not a priority, and it is very hard to say how popular it is. 
Thanks

Why was this never done? The linked question only contains the actual transform, which while trivial is useless without a function as stats.boxcox() to find lambda. This is not 'easy to do from the user side' 
> Why was this never done?

Because no one argued for its including here, which would be step 1. And no one stepped up to actually implement this, which would be step 2.
> Because no one argued for its including here, which would be step 1. And no one stepped up to actually implement this, which would be step 2.

This would be helpful anytime we have a mix of negative and positive values (Box-Cox works only for all positive or all negative). The fact it's inclusion is already available in R suggests it's a helpful function for statistics. As to step 2, I don't have time to contribute right away, but I might be able to later.

I implemented the Yeo-Johnson transform in scikit-learn (scikit-learn/scikit-learn#11520). It will be released with version 0.20.

The implementation is based on that of `scipy.stats.boxcox` and uses the brent solver with MLE. 

Would it be a welcome addition to scipy? If yes, I could work on this.

Like previously mentioned, Yeo-Johnson can deal with negative values while Box-Cox cannot, which is the main reason it was included in scikit learn.
It would also be useful for statsmodels (although it looks like I never opened an issue for it)
So, I'm in favor of getting it into scipy.
PR welcome seems to be the consensus:)

Resolving patch:
diff --git a/scipy/stats/__init__.py b/scipy/stats/__init__.py
--- a/scipy/stats/__init__.py
+++ b/scipy/stats/__init__.py
@@ -280,6 +280,9 @@
    boxcox
    boxcox_normmax
    boxcox_llf
+   yeojohnson
+   yeojohnson_normmax
+   yeojohnson_llf
    obrientransform
    sigmaclip
    trimboth
@@ -327,6 +330,7 @@
    ppcc_plot
    probplot
    boxcox_normplot
+   yeojohnson_normplot
 
 
 Masked statistics functions
diff --git a/scipy/stats/morestats.py b/scipy/stats/morestats.py
--- a/scipy/stats/morestats.py
+++ b/scipy/stats/morestats.py
@@ -26,7 +26,9 @@
            'boxcox_llf', 'boxcox', 'boxcox_normmax', 'boxcox_normplot',
            'shapiro', 'anderson', 'ansari', 'bartlett', 'levene', 'binom_test',
            'fligner', 'mood', 'wilcoxon', 'median_test',
-           'circmean', 'circvar', 'circstd', 'anderson_ksamp'
+           'circmean', 'circvar', 'circstd', 'anderson_ksamp',
+           'yeojohnson_llf', 'yeojohnson', 'yeojohnson_normmax',
+           'yeojohnson_normplot'
            ]
 
 
@@ -1150,6 +1152,43 @@ def _all(x, brack):
     return optimfunc(x, brack)
 
 
+def _normplot(method, x, la, lb, plot=None, N=80):
+    """Compute parameters for a Box-Cox or Yeo-Johnson normality plot,
+    optionally show it. See `boxcox_normplot` or `yeojohnson_normplot` for
+    details."""
+
+    if method == 'boxcox':
+        title = 'Box-Cox Normality Plot'
+        transform_func = boxcox
+    else:
+        title = 'Yeo-Johnson Normality Plot'
+        transform_func = yeojohnson
+
+    x = np.asarray(x)
+    if x.size == 0:
+        return x
+
+    if lb <= la:
+        raise ValueError("`lb` has to be larger than `la`.")
+
+    lmbdas = np.linspace(la, lb, num=N)
+    ppcc = lmbdas * 0.0
+    for i, val in enumerate(lmbdas):
+        # Determine for each lmbda the square root of correlation coefficient
+        # of transformed x
+        z = transform_func(x, lmbda=val)
+        _, (_, _, r) = probplot(z, dist='norm', fit=True)
+        ppcc[i] = r
+
+    if plot is not None:
+        plot.plot(lmbdas, ppcc, 'x')
+        _add_axis_labels_title(plot, xlabel='$\\lambda$',
+                               ylabel='Prob Plot Corr. Coef.',
+                               title=title)
+
+    return lmbdas, ppcc
+
+
 def boxcox_normplot(x, la, lb, plot=None, N=80):
     """Compute parameters for a Box-Cox normality plot, optionally show it.
 
@@ -1215,28 +1254,333 @@ def boxcox_normplot(x, la, lb, plot=None, N=80):
     >>> plt.show()
 
     """
+    return _normplot('boxcox', x, la, lb, plot, N)
+
+
+def yeojohnson(x, lmbda=None):
+    r"""
+    Return a dataset transformed by a Yeo-Johnson power transformation.
+
+    Parameters
+    ----------
+    x : ndarray
+        Input array.  Should be 1-dimensional.
+    lmbda : float, optional
+        If ``lmbda`` is ``None``, find the lambda that maximizes the
+        log-likelihood function and return it as the second output argument.
+        Otherwise the transformation is done for the given value.
+
+    Returns
+    -------
+    yeojohnson: ndarray
+        Yeo-Johnson power transformed array.
+    maxlog : float, optional
+        If the `lmbda` parameter is None, the second returned argument is
+        the lambda that maximizes the log-likelihood function.
+
+    See Also
+    --------
+    probplot, yeojohnson_normplot, yeojohnson_normmax, yeojohnson_llf, boxcox
+
+    Notes
+    -----
+    The Yeo-Johnson transform is given by::
+
+        y = ((x + 1)**lmbda - 1) / lmbda,                for x >= 0, lmbda != 0
+            log(x + 1),                                  for x >= 0, lmbda = 0
+            -((-x + 1)**(2 - lmbda) - 1) / (2 - lmbda),  for x < 0, lmbda != 2
+            -log(-x + 1),                                for x < 0, lmbda = 2
+
+    Unlike `boxcox`, `yeojohnson` does not require the input data to be
+    positive.
+
+    .. versionadded:: 1.2.0
+
+
+    References
+    ----------
+    I. Yeo and R.A. Johnson, "A New Family of Power Transformations to
+    Improve Normality or Symmetry", Biometrika 87.4 (2000):
+
+
+    Examples
+    --------
+    >>> from scipy import stats
+    >>> import matplotlib.pyplot as plt
+
+    We generate some random variates from a non-normal distribution and make a
+    probability plot for it, to show it is non-normal in the tails:
+
+    >>> fig = plt.figure()
+    >>> ax1 = fig.add_subplot(211)
+    >>> x = stats.loggamma.rvs(5, size=500) + 5
+    >>> prob = stats.probplot(x, dist=stats.norm, plot=ax1)
+    >>> ax1.set_xlabel('')
+    >>> ax1.set_title('Probplot against normal distribution')
+
+    We now use `yeojohnson` to transform the data so it's closest to normal:
+
+    >>> ax2 = fig.add_subplot(212)
+    >>> xt, lmbda = stats.yeojohnson(x)
+    >>> prob = stats.probplot(xt, dist=stats.norm, plot=ax2)
+    >>> ax2.set_title('Probplot after Yeo-Johnson transformation')
+
+    >>> plt.show()
+
+    """
+
     x = np.asarray(x)
     if x.size == 0:
         return x
 
-    if lb <= la:
-        raise ValueError("`lb` has to be larger than `la`.")
+    if lmbda is not None:
+        return _yeojohnson_transform(x, lmbda)
 
-    lmbdas = np.linspace(la, lb, num=N)
-    ppcc = lmbdas * 0.0
-    for i, val in enumerate(lmbdas):
-        # Determine for each lmbda the correlation coefficient of transformed x
-        z = boxcox(x, lmbda=val)
-        _, r2 = probplot(z, dist='norm', fit=True)
-        ppcc[i] = r2[-1]
+    # if lmbda=None, find the lmbda that maximizes the log-likelihood function.
+    lmax = yeojohnson_normmax(x)
+    y = _yeojohnson_transform(x, lmax)
 
-    if plot is not None:
-        plot.plot(lmbdas, ppcc, 'x')
-        _add_axis_labels_title(plot, xlabel='$\\lambda$',
-                               ylabel='Prob Plot Corr. Coef.',
-                               title='Box-Cox Normality Plot')
+    return y, lmax
 
-    return lmbdas, ppcc
+
+def _yeojohnson_transform(x, lmbda):
+    """Return x transformed by the Yeo-Johnson power transform with given
+    parameter lmbda."""
+
+    out = np.zeros_like(x)
+    pos = x >= 0  # binary mask
+
+    # when x >= 0
+    if abs(lmbda) < np.spacing(1.):
+        out[pos] = np.log1p(x[pos])
+    else:  # lmbda != 0
+        out[pos] = (np.power(x[pos] + 1, lmbda) - 1) / lmbda
+
+    # when x < 0
+    if abs(lmbda - 2) > np.spacing(1.):
+        out[~pos] = -(np.power(-x[~pos] + 1, 2 - lmbda) - 1) / (2 - lmbda)
+    else:  # lmbda == 2
+        out[~pos] = -np.log1p(-x[~pos])
+
+    return out
+
+
+def yeojohnson_llf(lmb, data):
+    r"""The yeojohnson log-likelihood function.
+
+    Parameters
+    ----------
+    lmb : scalar
+        Parameter for Yeo-Johnson transformation. See `yeojohnson` for
+        details.
+    data : array_like
+        Data to calculate Yeo-Johnson log-likelihood for. If `data` is
+        multi-dimensional, the log-likelihood is calculated along the first
+        axis.
+
+    Returns
+    -------
+    llf : float
+        Yeo-Johnson log-likelihood of `data` given `lmb`.
+
+    See Also
+    --------
+    yeojohnson, probplot, yeojohnson_normplot, yeojohnson_normmax
+
+    Notes
+    -----
+    The Yeo-Johnson log-likelihood function is defined here as
+
+    .. math::
+
+        llf = N/2 \log(\hat{\sigma}^2) + (\lambda - 1)
+              \sum_i \text{ sign }(x_i)\log(|x_i| + 1)
+
+    where :math:`\hat{\sigma}^2` is estimated variance of the the Yeo-Johnson
+    transformed input data ``x``.
+
+    .. versionadded:: 1.2.0
+
+    Examples
+    --------
+    >>> from scipy import stats
+    >>> import matplotlib.pyplot as plt
+    >>> from mpl_toolkits.axes_grid1.inset_locator import inset_axes
+    >>> np.random.seed(1245)
+
+    Generate some random variates and calculate Yeo-Johnson log-likelihood
+    values for them for a range of ``lmbda`` values:
+
+    >>> x = stats.loggamma.rvs(5, loc=10, size=1000)
+    >>> lmbdas = np.linspace(-2, 10)
+    >>> llf = np.zeros(lmbdas.shape, dtype=float)
+    >>> for ii, lmbda in enumerate(lmbdas):
+    ...     llf[ii] = stats.yeojohnson_llf(lmbda, x)
+
+    Also find the optimal lmbda value with `yeojohnson`:
+
+    >>> x_most_normal, lmbda_optimal = stats.yeojohnson(x)
+
+    Plot the log-likelihood as function of lmbda.  Add the optimal lmbda as a
+    horizontal line to check that that's really the optimum:
+
+    >>> fig = plt.figure()
+    >>> ax = fig.add_subplot(111)
+    >>> ax.plot(lmbdas, llf, 'b.-')
+    >>> ax.axhline(stats.yeojohnson_llf(lmbda_optimal, x), color='r')
+    >>> ax.set_xlabel('lmbda parameter')
+    >>> ax.set_ylabel('Yeo-Johnson log-likelihood')
+
+    Now add some probability plots to show that where the log-likelihood is
+    maximized the data transformed with `yeojohnson` looks closest to normal:
+
+    >>> locs = [3, 10, 4]  # 'lower left', 'center', 'lower right'
+    >>> for lmbda, loc in zip([-1, lmbda_optimal, 9], locs):
+    ...     xt = stats.yeojohnson(x, lmbda=lmbda)
+    ...     (osm, osr), (slope, intercept, r_sq) = stats.probplot(xt)
+    ...     ax_inset = inset_axes(ax, width="20%", height="20%", loc=loc)
+    ...     ax_inset.plot(osm, osr, 'c.', osm, slope*osm + intercept, 'k-')
+    ...     ax_inset.set_xticklabels([])
+    ...     ax_inset.set_yticklabels([])
+    ...     ax_inset.set_title('$\lambda=%1.2f$' % lmbda)
+
+    >>> plt.show()
+
+    """
+    data = np.asarray(data)
+    n_samples = data.shape[0]
+
+    if n_samples == 0:
+        return np.nan
+
+    trans = _yeojohnson_transform(data, lmb)
+
+    loglike = -n_samples / 2 * np.log(trans.var(axis=0))
+    loglike += (lmb - 1) * (np.sign(data) * np.log(np.abs(data) + 1)).sum(axis=0)
+
+    return loglike
+
+
+def yeojohnson_normmax(x, brack=(-2, 2)):
+    """Compute optimal Yeo-Johnson transform parameter for input data, using
+    maximum likelihood estimation.
+
+    Parameters
+    ----------
+    x : array_like
+        Input array.
+    brack : 2-tuple, optional
+        The starting interval for a downhill bracket search with
+        `optimize.brent`. Note that this is in most cases not critical; the
+        final result is allowed to be outside this bracket.
+
+    Returns
+    -------
+    maxlog : float
+        The optimal transform parameter found.
+
+    Notes
+    -----
+    .. versionadded:: 1.2.0
+
+    See Also
+    --------
+    yeojohnson, yeojohnson_llf, yeojohnson_normplot
+
+    Examples
+    --------
+    >>> from scipy import stats
+    >>> import matplotlib.pyplot as plt
+    >>> np.random.seed(1234)  # make this example reproducible
+
+    Generate some data and determine optimal ``lmbda``
+
+    >>> x = stats.loggamma.rvs(5, size=30) + 5
+    >>> lmax = stats.yeojohnson_normmax(x)
+
+    >>> fig = plt.figure()
+    >>> ax = fig.add_subplot(111)
+    >>> prob = stats.yeojohnson_normplot(x, -10, 10, plot=ax)
+    >>> ax.axvline(lmax, color='r')
+
+    >>> plt.show()
+
+    """
+
+    def _neg_llf(lmbda, data):
+        return -yeojohnson_llf(lmbda, data)
+
+    return optimize.brent(_neg_llf, brack=brack, args=(x,))
+
+
+def yeojohnson_normplot(x, la, lb, plot=None, N=80):
+    """Compute parameters for a Yeo-Johnson normality plot, optionally show it.
+
+    A Yeo-Johnson normality plot shows graphically what the best
+    transformation parameter is to use in `yeojohnson` to obtain a
+    distribution that is close to normal.
+
+    Parameters
+    ----------
+    x : array_like
+        Input array.
+    la, lb : scalar
+        The lower and upper bounds for the ``lmbda`` values to pass to
+        `yeojohnson` for Yeo-Johnson transformations. These are also the
+        limits of the horizontal axis of the plot if that is generated.
+    plot : object, optional
+        If given, plots the quantiles and least squares fit.
+        `plot` is an object that has to have methods "plot" and "text".
+        The `matplotlib.pyplot` module or a Matplotlib Axes object can be used,
+        or a custom object with the same methods.
+        Default is None, which means that no plot is created.
+    N : int, optional
+        Number of points on the horizontal axis (equally distributed from
+        `la` to `lb`).
+
+    Returns
+    -------
+    lmbdas : ndarray
+        The ``lmbda`` values for which a Yeo-Johnson transform was done.
+    ppcc : ndarray
+        Probability Plot Correlelation Coefficient, as obtained from `probplot`
+        when fitting the Box-Cox transformed input `x` against a normal
+        distribution.
+
+    See Also
+    --------
+    probplot, yeojohnson, yeojohnson_normmax, yeojohnson_llf, ppcc_max
+
+    Notes
+    -----
+    Even if `plot` is given, the figure is not shown or saved by
+    `boxcox_normplot`; ``plt.show()`` or ``plt.savefig('figname.png')``
+    should be used after calling `probplot`.
+
+    .. versionadded:: 1.2.0
+
+    Examples
+    --------
+    >>> from scipy import stats
+    >>> import matplotlib.pyplot as plt
+
+    Generate some non-normally distributed data, and create a Yeo-Johnson plot:
+
+    >>> x = stats.loggamma.rvs(5, size=500) + 5
+    >>> fig = plt.figure()
+    >>> ax = fig.add_subplot(111)
+    >>> prob = stats.yeojohnson_normplot(x, -20, 20, plot=ax)
+
+    Determine and plot the optimal ``lmbda`` to transform ``x`` and plot it in
+    the same plot:
+
+    >>> _, maxlog = stats.yeojohnson(x)
+    >>> ax.axvline(maxlog, color='r')
+
+    >>> plt.show()
+
+    """
+    return _normplot('yeojohnson', x, la, lb, plot, N)
 
 
 def shapiro(x):


Test patch:
diff --git a/scipy/stats/tests/test_morestats.py b/scipy/stats/tests/test_morestats.py
--- a/scipy/stats/tests/test_morestats.py
+++ b/scipy/stats/tests/test_morestats.py
@@ -1214,6 +1214,144 @@ def test_empty(self):
         assert_(stats.boxcox_normplot([], 0, 1).size == 0)
 
 
+class TestYeojohnson_llf(object):
+
+    def test_array_like(self):
+        np.random.seed(54321)
+        x = stats.norm.rvs(size=100, loc=0)
+        lmbda = 1
+        llf = stats.yeojohnson_llf(lmbda, x)
+        llf2 = stats.yeojohnson_llf(lmbda, list(x))
+        assert_allclose(llf, llf2, rtol=1e-12)
+
+    def test_2d_input(self):
+        np.random.seed(54321)
+        x = stats.norm.rvs(size=100, loc=10)
+        lmbda = 1
+        llf = stats.yeojohnson_llf(lmbda, x)
+        llf2 = stats.yeojohnson_llf(lmbda, np.vstack([x, x]).T)
+        assert_allclose([llf, llf], llf2, rtol=1e-12)
+
+    def test_empty(self):
+        assert_(np.isnan(stats.yeojohnson_llf(1, [])))
+
+
+class TestYeojohnson(object):
+
+    def test_fixed_lmbda(self):
+        np.random.seed(12345)
+
+        # Test positive input
+        x = stats.loggamma.rvs(5, size=50) + 5
+        assert np.all(x > 0)
+        xt = stats.yeojohnson(x, lmbda=1)
+        assert_allclose(xt, x)
+        xt = stats.yeojohnson(x, lmbda=-1)
+        assert_allclose(xt, 1 - 1 / (x + 1))
+        xt = stats.yeojohnson(x, lmbda=0)
+        assert_allclose(xt, np.log(x + 1))
+        xt = stats.yeojohnson(x, lmbda=1)
+        assert_allclose(xt, x)
+
+        # Test negative input
+        x = stats.loggamma.rvs(5, size=50) - 5
+        assert np.all(x < 0)
+        xt = stats.yeojohnson(x, lmbda=2)
+        assert_allclose(xt, -np.log(-x + 1))
+        xt = stats.yeojohnson(x, lmbda=1)
+        assert_allclose(xt, x)
+        xt = stats.yeojohnson(x, lmbda=3)
+        assert_allclose(xt, 1 / (-x + 1) - 1)
+
+        # test both positive and negative input
+        x = stats.loggamma.rvs(5, size=50) - 2
+        assert not np.all(x < 0)
+        assert not np.all(x >= 0)
+        pos = x >= 0
+        xt = stats.yeojohnson(x, lmbda=1)
+        assert_allclose(xt[pos], x[pos])
+        xt = stats.yeojohnson(x, lmbda=-1)
+        assert_allclose(xt[pos], 1 - 1 / (x[pos] + 1))
+        xt = stats.yeojohnson(x, lmbda=0)
+        assert_allclose(xt[pos], np.log(x[pos] + 1))
+        xt = stats.yeojohnson(x, lmbda=1)
+        assert_allclose(xt[pos], x[pos])
+
+        neg = ~pos
+        xt = stats.yeojohnson(x, lmbda=2)
+        assert_allclose(xt[neg], -np.log(-x[neg] + 1))
+        xt = stats.yeojohnson(x, lmbda=1)
+        assert_allclose(xt[neg], x[neg])
+        xt = stats.yeojohnson(x, lmbda=3)
+        assert_allclose(xt[neg], 1 / (-x[neg] + 1) - 1)
+
+    @pytest.mark.parametrize('lmbda', [0, .1, .5, 2])
+    def test_lmbda_None(self, lmbda):
+        # Start from normal rv's, do inverse transform to check that
+        # optimization function gets close to the right answer.
+
+        def _inverse_transform(x, lmbda):
+            x_inv = np.zeros(x.shape, dtype=x.dtype)
+            pos = x >= 0
+
+            # when x >= 0
+            if abs(lmbda) < np.spacing(1.):
+                x_inv[pos] = np.exp(x[pos]) - 1
+            else:  # lmbda != 0
+                x_inv[pos] = np.power(x[pos] * lmbda + 1, 1 / lmbda) - 1
+
+            # when x < 0
+            if abs(lmbda - 2) > np.spacing(1.):
+                x_inv[~pos] = 1 - np.power(-(2 - lmbda) * x[~pos] + 1,
+                                           1 / (2 - lmbda))
+            else:  # lmbda == 2
+                x_inv[~pos] = 1 - np.exp(-x[~pos])
+
+            return x_inv
+
+        np.random.seed(1234567)
+        n_samples = 20000
+        x = np.random.normal(loc=0, scale=1, size=(n_samples))
+
+        x_inv = _inverse_transform(x, lmbda)
+        xt, maxlog = stats.yeojohnson(x_inv)
+
+        assert_allclose(maxlog, lmbda, atol=1e-2)
+
+        assert_almost_equal(0, np.linalg.norm(x - xt) / n_samples, decimal=2)
+        assert_almost_equal(0, xt.mean(), decimal=1)
+        assert_almost_equal(1, xt.std(), decimal=1)
+
+    def test_empty(self):
+        assert_(stats.yeojohnson([]).shape == (0,))
+
+    def test_array_like(self):
+        np.random.seed(54321)
+        x = stats.norm.rvs(size=100, loc=0)
+        lmbda = 1.5
+        xt1, _ = stats.yeojohnson(x)
+        xt2, _ = stats.yeojohnson(list(x))
+        assert_allclose(xt1, xt2, rtol=1e-12)
+
+
+class TestYeojohnsonNormmax(object):
+    def setup_method(self):
+        np.random.seed(12345)
+        self.x = stats.loggamma.rvs(5, size=50) + 5
+
+    def test_mle(self):
+        maxlog = stats.yeojohnson_normmax(self.x)
+        assert_allclose(maxlog, 1.876393, rtol=1e-6)
+
+    def test_darwin_example(self):
+        # test from original paper "A new family of power transformations to
+        # improve normality or symmetry" by Yeo and Johnson.
+        x = [6.1, -8.4, 1.0, 2.0, 0.7, 2.9, 3.5, 5.1, 1.8, 3.6, 7.0, 3.0, 9.3,
+             7.5, -6.0]
+        lmbda = stats.yeojohnson_normmax(x)
+        assert np.allclose(lmbda, 1.305, atol=1e-3)
+
+
 class TestCircFuncs(object):
     def test_circfuncs(self):
         x = np.array([355, 5, 2, 359, 10, 350])