What are the coefficients returned by Polynomial Ridge Regression (or any regression)?

### Describe the issue linked to the documentation

I asked and answered a question about Regression in [Stack Overflow](https://stackoverflow.com/questions/79691953/ridge-polynomial-regression-how-to-get-parameters-for-equation-found).  Here is a summary and question.

I ran several regressions using pipeline and gridsearch.  The winning regression was polynomial ridge regression.  What I then wanted to do was extract the coefficients of the successful regression so I could pass them on for an implementation that uses just python (no libraries) and Java (no libraries).  That was not straightforward.

I eventually found the coefficients under `steps` after someone pointed that out.  Even the answers I got on Google indicated they were under the attribute `coef` but I couldn't find them though I thought I had read the docs sufficiently.

As explained at the link above, I expected coefficients for an equation: `a + bx + cx^2 + dx^3`.  If I looked at the coefficients under the attribute `coef_` I got: `[ 0.00000000e+00  9.17291774e-01 -4.25186367e-09  9.06355625e-18]`, from which I assumed that meant that `a=0`,` b=9.17291774e-01`, etc.  It turned out that was only partially true, `b-d` are correct but `a` is not.  `a` is actually the interecept which is another attribute `intercept_`.  At least, that is how I got things to work (code below for an example)

Question:  what is the first element in the coefficients from Polynomial Ridge Regression or have I completely misunderstood?

```
import pandas as pd
import warnings

# regression libs
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures

# useful initializations
warnings.filterwarnings('ignore')

p = [0, 10, -20, .30]

# Create fake data using the preceding coefficients with some noise
def regr_noise(x, p):
    mu = np.random.uniform(0,50E6)
    return (p[0] + p[1]*x + p[2]*x**2 + p[3]*x**3 + mu)

x = range(0,1000, 50)
df_fake = pd.DataFrame({'x':x})
df_fake['y'] = df_fake['x'].apply(lambda x: regr_noise(x, p))

# polynomial ridge regression
x = df_fake[['x']]
y = df_fake[['y']]

ridge_reg_poly = make_pipeline(PolynomialFeatures(degree=3),Ridge(alpha=0, solver="cholesky"))
temp = ridge_reg_poly.fit(x,y)
yp = temp.predict(x)

# get the coefficients and reproduce the results
p_found = ridge_reg_poly.steps[1][1].coef_
p_interc = ridge_reg_poly.steps[1][1].intercept_
print('coeff:',p_found,'\nintercep:' ,p_interc)

def regr_clean(x, p):
    return (p[0] + p[1]*x + p[2]*x**2 + p[3]*x**3 + p_interc)

y_fmcoef = regr_clean(x, p_found).rename(columns={'x':'y'})

Will get a column of zeros b/c same results as sklearn
yp - y_fmcoef.y
```

### Suggest a potential alternative/fix

Clear statements:

1. coefficients are produced and available
2. what the coefficients are, their relationship to an equation
3. how to use them in an equation

COMMENT:
According to the [Pipeline documentation](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html), a `Pipeline` object doesn't have a no `coef_` attribute. [Likewise](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html), a `PolynomialFeatures` objects also lacks a `coef_` attribute. Therefore, if `coef_` exists, it must belong to the `Ridge` estimator, which is [exactly the case](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge).

Now, let's look at your example, @MikeB2019x. We start with a dataset `x` of 1000 samples and 1 feature:

```python
x = range(0,1000, 50)
```
When passed to `PolynomialFeatures(degree=3)`, it transformed into a datset `x_new` of 1000 samples and 4 features. Particularly, the columns of `x_new` correspond to:

```python
[1, x, x** 2, x**3]
```

These monomials are all monic (each has coefficient 1), as stated in the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html).

So, it's clear that the array `[ 0.00000000e+00  9.17291774e-01 -4.25186367e-09  9.06355625e-18]` does not come from `PolynomialFeatures` itself. Instead, these are the learned coefficients of the `Ridge` regressor in the `Pipeline`.

That is, `Ridge` is trained on `x_new` (with its 4 features), and its `coef_` vector is 4-dimensional: `coef_[i]` corresponds to the feature `x ** i` from `x_new`, for `i = 0, 1, 2, 3`.

Thus, for a single sample `z`, the prediction looks like:

`ridge_reg_poly.predict(z) = coef_[0] * 1 + coef_[1] * z + coef_[2] * (z ** 2) + coef_[3] * (z ** 3) + Ridge.intercept_` .

That said, I think this is just a matter of following the documentation closely rather than a need of updating it.

Edit: Fixed RHS of `predict`.

Labels: Documentation