Repository: scipy/scipy

Issue:
Covariance matrix returned by ODR needs to be scaled by the residual variance
Originally reported on SO [here][1].

```python
import numpy as np
from scipy.odr import ODR, Model, RealData

def f(B, x):
    return B[0] + B[1]*x + B[2]*x**3

n = 100
sx = 0.3
sy = 2.0
B_real = [-2.3, 7.0, -4.0]
gen = np.random.RandomState(0)
x = np.linspace(-5, 5, n)
y = f(B_real, x)
x +=  gen.randn(n) * sx
y += gen.randn(n) * sy

model = Model(f)
data = RealData(x, y, sx=sx, sy=sy)
odr = ODR(data, model, beta0=[1., 0., 0.])
odr.set_job(fit_type=0)
output = odr.run()

output.pprint()
# Beta: [-2.56437069  6.97055557 -4.11430261]
# Beta Std Error: [ 0.48929313  0.66751299  0.14785663]
# Beta Covariance: [[ 0.2321291  -0.02974486  0.00394905]
#  [-0.02974486  0.43202691 -0.05337819]
#  [ 0.00394905 -0.05337819  0.02119693]]
# Residual Variance: 1.03135611652
# Inverse Condition #: 0.0768799779673
# Reason(s) for Halting:
#   Sum of squares convergence

print("           sd_beta: {}".format(output.sd_beta))
print("sqrt(diag(sd_cov)): {}".format(np.sqrt(np.diag(output.cov_beta))))
#            sd_beta: [ 0.48929313  0.66751299  0.14785663]
# sqrt(diag(sd_cov)): [ 0.48179778  0.65728754  0.14559166]
```

The discrepancy arises because `__odrpack.c` extracts `cov_beta` directly from the FORTRAN routine's `work` vector using the  [ `vcv` index][2]. However, according to the [ODRPACK documentation][3] (p85):

> **`WORK(VCVI)`** is the first element of a p × p array `VCV` containing the values of the covariance matrix of the parameters β *prior to scaling by the residual variance*, where  `WORK(VCVI-1+I+(J-1)*(NP)) = VCV(I,J)` = ̂σ⁻²V β(I, J) for I = 1,... ,p and J = 1,... ,p.

If we multiply the covariance matrix by the residual variance then we get the expected result:

```python
print("                sd_beta: {}".format(output.sd_beta))
print("sqrt(diag(sd_cov * rv)): {}".format(
      np.sqrt(np.diag(output.cov_beta * output.res_var))))
#                 sd_beta: [ 0.48929313  0.66751299  0.14785663]
# sqrt(diag(sd_cov * rv)): [ 0.48929313  0.66751299  0.14785663]
```

[1]: http://stackoverflow.com/q/41028846/1461210
[2]: https://github.com/scipy/scipy/blob/2526df72e5d4ca8bad6e2f4b3cbdfbc33e805865/scipy/odr/__odrpack.c#L363
[3]: https://docs.scipy.org/doc/external/odrpack_guide.pdf


Discussion:
I came across a related post on SO: https://stackoverflow.com/questions/62460399/comparison-of-curve-fit-and-scipy-odr-absolute-sigma

Trying to understand the results I compared ODR in ftype=2 mode (least squares) with scipy.optimize.curve_fit:

```
import numpy as np
from scipy.odr import ODR, Model, RealData

def f(B, x):
    return B[0] + B[1]*x + B[2]*x**3

n = 100
sx = 0.3
sy = 2.0
B_real = [-2.3, 7.0, -4.0]
gen = np.random.RandomState(0)
x = np.linspace(-5, 5, n)
y = f(B_real, x)
x +=  gen.randn(n) * sx
y += gen.randn(n) * sy

model = Model(f)
data = RealData(x, y, sx=sx, sy=sy)
odr = ODR(data, model, beta0=[1., 0., 0.])
odr.set_job(fit_type=2)
output = odr.run()

print("odr      : sd_beta: {}".format(output.sd_beta))
print("odr      : sqrt(diag(sd_cov)): {}".format(np.sqrt(np.diag(output.cov_beta))))

def func(x, a, b, c):
    return a + b*x + c*x**3

from scipy.optimize import curve_fit

popt, pcov = curve_fit(func, x, y, sigma=np.tile([sy], len(y)), absolute_sigma=True)
print("curve_fit: sqrt(diag(cov)): {}".format(np.sqrt(np.diag(pcov))))
```

and the output is

```
odr      : sd_beta: [3.95086679 3.33293371 0.1969002 ]
odr      : sqrt(diag(sd_cov)): [0.2018686  0.17029546 0.01006057]
curve_fit: sqrt(diag(cov)): [0.20186863 0.17029594 0.01006059]

```

So the unweighted covariance matrix returned by ODR is (almost) the same as that returned by curve_fit but `sd_beta` is obviously different.

What is the right way to get the error of the paremeters then?


@kurisutsukato Thank you for your comments.

> What is the right way to get the error of the paremeters then?

In my understanding, the odr covariance is not scaled by residual variance, so curve_fit covariance is too.
However, it seems that `sd_beta` is scaled by residual variance, there is inconsistency.

I think `sd_beta` should be unscaled or `sd_beta` should be changed to `scaled_sd_beta` or something like that.
But both options might break backward compatibility.
So, I added an optional argument for scaled covariance matrix of ODR and I updated `sd_beta` doc which show it is scaled in #12207 

Modifying  your code, we can check this easily.
```
import numpy as np

from scipy.odr import ODR, Model, RealData


def f(B, x):
    return B[0] + B[1] * x + B[2] * x ** 3


n = 100
sx = 0.3
sy = 2.0
B_real = [-2.3, 7.0, -4.0]
gen = np.random.RandomState(0)
x = np.linspace(-5, 5, n)
y = f(B_real, x)
x += gen.randn(n) * sx
y += gen.randn(n) * sy

model = Model(f)
data = RealData(x, y, sx=sx, sy=sy)
odr = ODR(data, model, beta0=[1., 0., 0.])
odr.set_job(fit_type=2)
output = odr.run()

print("odr      : sd_beta: {}".format(output.sd_beta))
print("odr(unscaled): sqrt(diag(sd_cov)): {}".format(
    np.sqrt(np.diag(output.cov_beta))))
print("odr(scaled): sqrt(diag(sd_cov * res_var)): {}".format(
    np.sqrt(np.diag(output.cov_beta * output.res_var))))


def func(x, a, b, c):
    return a + b * x + c * x ** 3


from scipy.optimize import curve_fit

popt, pcov = curve_fit(func, x, y, sigma=np.tile([sy], len(y)),
                       absolute_sigma=True)
print("curve_fit: sqrt(diag(cov)): {}".format(np.sqrt(np.diag(pcov))))


```

The output is
```
odr      : sd_beta: [3.9508668  3.33294797 0.19690089]
odr(unscaled): sqrt(diag(sd_cov)): [0.2018686  0.17029619 0.0100606 ]
odr(scaled): sqrt(diag(sd_cov * res_var)): [3.9508668  3.33294797 0.19690089]
curve_fit: sqrt(diag(cov)): [0.20186863 0.17029594 0.01006059]
```



@AtsushiSakai Thanks for looking into this.

What I was trying to say is, that I would expect that ODR's `sd_beta` is the same as `sqrt(diag(cov))` returned by `curve_fit`, which is apparently not the case. But I found out that it depends on the `absolute_sigma` argument of `curve_fit`. Setting it to `False` makes `sqrt(diag(cov))` coincide with ODR's `sd_beta`.

```
import numpy as np
from scipy.odr import ODR, Model, RealData
def f(B, x):
    return B[0] + B[1]*x + B[2]*x**3

n = 100
sx = 0.3
sy = 2.0
B_real = [-2.3, 7.0, -4.0]
gen = np.random.RandomState(0)
x = np.linspace(-5, 5, n)
y = f(B_real, x)
x += gen.randn(n) * sx
y += gen.randn(n) * sy
# since we are in leastsq mode all error is interpreted as error in y

model = Model(f)
data = Data(x, y)
odr = ODR(data, model, beta0=[1., 1., 1.])
odr.set_job(fit_type=2)
output = odr.run()
print(output.beta)
print("odr      : sd_beta: {}".format(output.sd_beta))
print("odr      : sqrt(diag(sd_cov)): {}".format(np.sqrt(np.diag(output.cov_beta))))

def func(x, a, b, c):
    return a + b*x + c*x**3

from scipy.optimize import curve_fit

popt, pcov = curve_fit(func, x, y, absolute_sigma=False) 
print("curve_fit(absolute_sigma=False) sqrt(diag(cov)): {}".format(np.sqrt(np.diag(pcov))))
popt, pcov = curve_fit(func, x, y, absolute_sigma=True)
print("curve_fit(absolute_sigma=True) sqrt(diag(cov)): {}".format(np.sqrt(np.diag(pcov))))

```

yields

```
odr      : sd_beta: [3.95086651 3.33293698 0.19689997]
odr      : sqrt(diag(sd_cov)): [0.10093429 0.08514781 0.00503028]
curve_fit(absolute_sigma=False) sqrt(diag(cov)): [3.95086723 3.33294311 0.19690061]
curve_fit(absolute_sigma=True) sqrt(diag(cov)): [0.10093431 0.08514797 0.0050303 ]
```

Sorry for the noise.
@kurisutsukato Oh. OK. I misunderstood what you mean.  

> But I found out that it depends on the absolute_sigma argument

That's great : )
Bump.  This confused the hell out of me before I found the Stack Overflow thread (https://stackoverflow.com/questions/41028846/how-to-compute-standard-error-from-odr-results).  At the very least, we should specify in the docstring for cov_beta that it should be multiplied by the residual variance
From what I remember when I looked at measurement error model many years ago. I did not look at the ODR literature, the literature I was reading criticized ODR for this.

ODR treats lhs and rhs in the same way, i.e. all variables have measurement errors. The measurement error is assumed to be known.  (Similar to the curve_fit option of NOT scaling the cov by the residual variance.)

An alternative treatment is to have the "dependent" lhs variable taking the slack. That is, the measurement error of regressors is assumed to be known but not the measurement error of the dependent variable.
In this case, the residual variance includes the measurement error of the dependent random variable plus any left over variance from model approximation or model misspecifiction errors.
However, AFAIU that literature, it said that ODR is NOT doing this. So, multiplying by the residual variance is not in the design of ODR.

This does not mean it could not be added, but I never looked at the scipy ODR code.



The square root of the diagonals of "cov_beta" should be made equal to "sd_beta".  Whether ODR treats LHS and RHS the same way or not, there's no reason for cov_beta and sd_beta to be inconsistent with each other.

If we don't want to make this change because of backward compatibility, the docstring for cov_beta should explicitly say that it isn't multiplied by the residual variance.
Documentation enhancements are welcome. I made a choice to output exactly what ODRPACK outputs so that people can use its documentation, and they decided to scale `sd_beta` but not `cov_beta`.

Resolving patch:
diff --git a/scipy/odr/_odrpack.py b/scipy/odr/_odrpack.py
--- a/scipy/odr/_odrpack.py
+++ b/scipy/odr/_odrpack.py
@@ -553,6 +553,10 @@ class Output:
         Standard deviations of the estimated parameters, of shape (p,).
     cov_beta : ndarray
         Covariance matrix of the estimated parameters, of shape (p,p).
+        Note that this `cov_beta` is not scaled by the residual variance 
+        `res_var`, whereas `sd_beta` is. This means 
+        ``np.sqrt(np.diag(output.cov_beta * output.res_var))`` is the same 
+        result as `output.sd_beta`.
     delta : ndarray, optional
         Array of estimated errors in input variables, of same shape as `x`.
     eps : ndarray, optional


Test patch: