Repository: scipy/scipy

Issue:
Improve documentation for smoothing in interpolate.UnivariateSpline and interpolate.splrep
**Problem:**
For [`scipy.interpolate.UnivariateSpline`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.UnivariateSpline.html#scipy.interpolate.UnivariateSpline) and [`scipy.interpolate.splrep`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.splrep.html#scipy.interpolate.splrep), the user is asked to supply a smoothing factor (`s`) that is used to smooth the spline fit through noisy data. At first glance, this is easy, because `s=0` provides no smoothing, only interpolation. A larger value of `s` provides more smoothing. However, in practice, I have been finding `s` through a tedious process of trial and error, and I imagine that many users are doing the same. As discussed in https://github.com/scipy/scipy/issues/4975#issuecomment-112870150 and (more recently) in 
https://github.com/PyAbel/PyAbel/issues/276#issuecomment-616966199, the documentation for how to select an appropriate `s` is somewhat confusing. 

In the case where users correctly input the weights array (`w`) as the inverse of the standard deviation, then `s` is correctly set to `s = len(w)`, which is a good starting point and covered in the documentation. However, for users that use the default weights array (`w = ones(len(x))`), then the default `s` is much too high and they aren't offered any guidance on how to select an appropriate `s`.

**Suggested solution:**
As discussed in the docs for [splrep ](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpolate.splrep.html#scipy.interpolate.splrep), the amount of smoothness is given by `sum((w * (y - g))**2,axis=0) <= s`, where w is the weights, y is the data, and g is the spline. So, if the weights are equal to 1 everywhere, then we are left with `sum((y-g)**2) <= s`, which is `(number of points) * (standard dev of residuals)**2`. If we assume that we want the standard deviation of the fit residuals to be equal to the noise (standard deviation) of the data, then we would want `s = (number of points) * (standard deviation)**2`.

So, I think that the documentation for `s` should include something like the following:
"If the weights are all set to 1 (default), then the appropriate value of `s` will be somewhere near len(x)*sd**2, where `sd` is is an estimate of the standard deviation of the data points." Another sentence should be included to emphasize that the user *must* set either the `w` or the `s` to achieve sensible results.

In addition, the documentation for `UnivariateSpline` should mention that the default value of the `w` array is `ones(len(x))`. `splrep` already does this, but `UnivariateSpline` just says that "If None (default), weights are all equal.". Since the absolute value of the weights matters for the selection of `s`, it's important to say what the actual value of `w[i]` is. 

**Alternatives:**
I don't think that these are great ideas, but since the form asks for alternatives, here is what comes to mind:

1) More invasive: One could argue that UnivariateSpline should work reasonably, out-of-the-box, without the user needing to input the weights or smoothing variables. This would amount to setting the default `s = len(x) * 1e-4`, or assuming a 1% error in the datapoints.

2) More invasive: the functions could show a warning if the user hasn't set either `w` or `s`.

**Side-note:**
`splrep` sets the default `s = len(x) - sqrt(2*(len(x))` while `UnivariateSpline` sets the default `s = len(x)`. These functions are both fitting the same splines, right? Shouldn't they use the same default? Their documentation is also quite different, and it might make sense to take the best of both docs and combine them. Pragmatically, I'm guessing that most users will be using UnivariateSpline (it's the most user-friendly), but splrep has more comprehensive documentation.

EDIT: Here is a previous issue from many years ago when `s = None` by default: https://github.com/scipy/scipy/issues/726

EDIT2: Here is the link to the section of the FITPACK code where there is a lengthy (but fairly confusing) discussion of the smoothing factor: https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack/curfit.f#L141


Discussion:
It'd be great to improve the docs, care to send the PR? I'd be happy to review it.

Changing the defaults is not an option anymore, I'm afraid.
Alternative smoothing/control routines is more then welcome, but should be added to make_* routines.
> care to send the PR? I'd be happy to review it.

Sure, I'll make a PR for the revised docs.

> but should be added to make_* routines.

What are "make_* routines"? I think that improving the docs is fine at this point. I was just thinking about alternative approaches since the form prompted me to do that :)

> What are "make_* routines"? I think that improving the docs is fine at this point. I was just thinking about alternative approaches since the form prompted me to do that :)

Here's a previous issue with some discussion: 
https://github.com/scipy/scipy/issues/6730
Another thing related to smoothing and documentation.

The “smoothing condition” in `splrep()` has very different effects with and without the periodic conditions (the `per` parameter):
```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import splrep, splev

N = 20
tol = 0.5
s = N * tol**2

xd = np.linspace(0, 1, N, endpoint=False)
yd = np.sin(2 * np.pi * xd) + \
     np.random.RandomState(0).normal(scale=0.1, size=N)

x = np.linspace(0, 1, 100, endpoint=False)

nonperiodic = splrep(xd, yd, s=s)
periodic = splrep(np.append(xd, xd[0] + 1), np.append(yd, 0), s=s, per=True)

plt.subplot(211)

plt.plot(xd, yd, 'C0o', label='data')
plt.plot(xd + 1, yd, 'C0o', fillstyle='none')

plt.plot(x, splev(x, nonperiodic), 'C1', label='per=False')
plt.plot(x + 1, splev(x, nonperiodic), 'C1:')

plt.plot(x, splev(x, periodic), 'C2', label='per=True')
plt.plot(x + 1, splev(x, periodic), 'C2')

plt.ylabel('value')
plt.legend()
plt.title('tol = {}'.format(tol))

plt.subplot(212)

plt.plot(xd, yd - splev(xd, nonperiodic), 'C1o', label='per=False')
plt.plot(xd + 1, yd - splev(xd, nonperiodic), 'C1o', fillstyle='none')

plt.plot(xd, yd - splev(xd, periodic), 'C2o', label='per=True')
plt.plot(xd + 1, yd - splev(xd, periodic), 'C2o', fillstyle='none')

plt.ylabel('deviation')
plt.legend()

plt.tight_layout()
plt.show()
```
![0 5](https://user-images.githubusercontent.com/12854133/80046362-a16f3680-84c7-11ea-825b-2b98211c0911.png)
Without periodicity, `splrep()` apparently tries to find the best match to the data points without exceeding the “smoothing condition”, but with periodicity, it apparently tries to reduce the total variation of the spline as much as possible without exceeding the “smoothing condition”. I can even agree that the green spline (`per=True`) can be called “smoother” than the orange one (`per=False`), but why the interpretation of “smoothness” depends on whether `per` is `True` or `False` and why this is not documented?

For closer tolerances (smaller `s`), the non-periodic and periodic splines do not show such drastic differences:
![0 1](https://user-images.githubusercontent.com/12854133/80046924-40e0f900-84c9-11ea-862d-4bbc8fa1161a.png)

The difference in behavior between the periodic conditions and the non-periodic conditions is strange. If anything, I would have expected the periodic conditions to force the fit to be better, instead of the opposite. 

I'm not sure where the difference arises. As you can see here, there is a difference in the "nest" variable between the two cases: https://github.com/scipy/scipy/blob/master/scipy/interpolate/_fitpack_impl.py#L478

Also, the non-periodic case uses [curfit](https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack/curfit.f), while the periodic case uses [percur](https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack/percur.f).
Interesting. The `nest` differences seem to be due to different requirements of [percur](https://github.com/scipy/scipy-svn/blob/master/scipy/interpolate/fitpack/percur.f#L56) and [curfit](https://github.com/scipy/scipy-svn/blob/master/scipy/interpolate/fitpack/curfit.f#L56), but this just suggests the maximal storage-space requirements and should not affect the results.

The whole documentation for [percur](https://github.com/scipy/scipy-svn/blob/master/scipy/interpolate/fitpack/percur.f) and [curfit](https://github.com/scipy/scipy-svn/blob/master/scipy/interpolate/fitpack/curfit.f) looks almost identical. With one minor detail:

[percur](https://github.com/scipy/scipy-svn/blob/b3d70966190d759261bf0b2ddc8cddef5d4336a6/scipy/interpolate/fitpack/percur.f#L102-L104):
```
c    ier=-2 : normal return. the spline returned is the weighted least-
c             squares constant. in this extreme case fp gives the upper
c             bound fp0 for the smoothing factor s.
```
[curfit](https://github.com/scipy/scipy-svn/blob/b3d70966190d759261bf0b2ddc8cddef5d4336a6/scipy/interpolate/fitpack/curfit.f#L102-L104):
```
c    ier=-2 : normal return. the spline returned is the weighted least-
c             squares polynomial of degree k. in this extreme case fp
c             gives the upper bound fp0 for the smoothing factor s.
```
Notice the “weighted least-squares **constant**” and “weighted least-squares **polynomial of degree k**”! This happens “**if s is very large**” (according to “further comments” there)!

So this is apparently what we see. Using the same example, but with a higher-frequency oscillation in the data:
```python
yd = np.sin(4 * np.pi * xd) + \
```
which cannot be approximated by a _single cubic polynomial_, reveals that in this case both non-periodic and periodic splines behave similarly:
![4pi](https://user-images.githubusercontent.com/12854133/80058992-00dd3e80-84e8-11ea-97a0-7121482b7212.png)
They both indeed try to be “smoother” (have smaller variation) rather than fit the data better. _If s is **not** too large!_

From this perspective, this behavior kind of makes sense, but it definitely must be documented. I mean that:
1. Smaller spline variation is preferred over better data reproduction.
2. If s is too large, the best-fit polynomial is returned. For the non-periodic case, it will have degree k, but for the periodic case it is a constant, since polynomials of order > 0 cannot be periodic.
Improving the documentation on what happens when `s` is set to its default value would definitely be helpful. Was getting some very weird splines with s=`None`, so started to dig deeper into the smoothness parameter to find out that it was way too high.
I probably got spoiled by scikit-learn, where default values usually give a good enough initial result.
Pull request are certainly very welcome.

Resolving patch:
diff --git a/scipy/interpolate/_fitpack2.py b/scipy/interpolate/_fitpack2.py
--- a/scipy/interpolate/_fitpack2.py
+++ b/scipy/interpolate/_fitpack2.py
@@ -84,7 +84,7 @@ class UnivariateSpline:
         1-D array of dependent input data, of the same length as `x`.
     w : (N,) array_like, optional
         Weights for spline fitting.  Must be positive.  If `w` is None,
-        weights are all equal. Default is None.
+        weights are all 1. Default is None.
     bbox : (2,) array_like, optional
         2-sequence specifying the boundary of the approximation interval. If
         `bbox` is None, ``bbox=[x[0], x[-1]]``. Default is None.
@@ -97,9 +97,20 @@ class UnivariateSpline:
 
             sum((w[i] * (y[i]-spl(x[i])))**2, axis=0) <= s
 
-        If `s` is None, ``s = len(w)`` which should be a good value if
-        ``1/w[i]`` is an estimate of the standard deviation of ``y[i]``.
-        If 0, spline will interpolate through all data points. Default is None.
+        If `s` is None, `s` will be set as `len(w)` for a smoothing spline
+        that uses all data points.
+        If 0, spline will interpolate through all data points. This is
+        equivalent to `InterpolatedUnivariateSpline`.
+        Default is None.
+        The user can use the `s` to control the tradeoff between closeness
+        and smoothness of fit. Larger `s` means more smoothing while smaller
+        values of `s` indicate less smoothing.
+        Recommended values of `s` depend on the weights, `w`. If the weights
+        represent the inverse of the standard-deviation of `y`, then a good
+        `s` value should be found in the range (m-sqrt(2*m),m+sqrt(2*m))
+        where m is the number of datapoints in `x`, `y`, and `w`. This means
+        ``s = len(w)`` should be a good value if ``1/w[i]`` is an
+        estimate of the standard deviation of ``y[i]``.
     ext : int or str, optional
         Controls the extrapolation mode for elements
         not in the interval defined by the knot sequence.
@@ -612,7 +623,7 @@ class InterpolatedUnivariateSpline(UnivariateSpline):
 
     Fits a spline y = spl(x) of degree `k` to the provided `x`, `y` data.
     Spline function passes through all provided points. Equivalent to
-    `UnivariateSpline` with  s=0.
+    `UnivariateSpline` with  `s` = 0.
 
     Parameters
     ----------
@@ -622,7 +633,7 @@ class InterpolatedUnivariateSpline(UnivariateSpline):
         input dimension of data points
     w : (N,) array_like, optional
         Weights for spline fitting.  Must be positive.  If None (default),
-        weights are all equal.
+        weights are all 1.
     bbox : (2,) array_like, optional
         2-sequence specifying the boundary of the approximation interval. If
         None (default), ``bbox=[x[0], x[-1]]``.
@@ -740,7 +751,7 @@ class LSQUnivariateSpline(UnivariateSpline):
 
     w : (N,) array_like, optional
         weights for spline fitting. Must be positive. If None (default),
-        weights are all equal.
+        weights are all 1.
     bbox : (2,) array_like, optional
         2-sequence specifying the boundary of the approximation interval. If
         None (default), ``bbox = [x[0], x[-1]]``.


Test patch: