Repository: scipy/scipy

Issue:
ENH: `multivariate_normal.rvs` extremely slow
### Describe your issue.

Due to using the default options for `rng.multivariate_normal` ([here](https://github.com/scipy/scipy/blob/c1ed5ece8ffbf05356a22a8106affcd11bd3aee0/scipy/stats/_multivariate.py#L753)), `scipy.stats.multivariate_normal.rvs` is quite slow for large inputs. This causes `scipy.stats.multivariate_t` to also be very slow, as it calls `multivariate_normal.rvs`. 

Potential solutions - expose the `method` keyword argument from `np.random.Generator.multivariate_normal` - though this is a bit suboptimal, as you'd have to expose in methods that internally use `multivariate_normal.rvs`, too.

Another is to use the `allow_singular` argument to determine if `method` in the `np.random.Generator.multivariate_normal` call should be `eigh` or `cholesky`. 

### Reproducing Code Example

```python
import numpy as np
from scipy.stats import multivariate_normal

data = np.random.standard_normal((4096, 4096))
cov = data @ data.T

%time multivariate_normal.rvs(np.zeros(4096), cov)
CPU times: user 2min 45s, sys: 56 s, total: 3min 41s
Wall time: 31.5 s

# this slowness of the above is primarily due to this internal call, which uses the `svd` method
%time rng.multivariate_normal(np.zeros(4096), cov)
CPU times: user 2min 12s, sys: 50 s, total: 3min 2s
Wall time: 24.7 s

# switching it to this would be faster
%time rng.multivariate_normal(np.zeros(4096), cov, method="cholesky")
CPU times: user 1.91 s, sys: 979 ms, total: 2.89 s
Wall time: 710 ms

# even this is a lot faster
%time rng.multivariate_normal(np.zeros(4096), cov, method="eigh")
CPU times: user 38.8 s, sys: 12.4 s, total: 51.2 s
Wall time: 6.93 s
```


### Error message

```shell
N/A
```


### SciPy/NumPy/Python version and system information

```shell
1.10.1 1.23.5 sys.version_info(major=3, minor=9, micro=15, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: cmake
    found: true
    include directory: unknown
    lib directory: unknown
    name: OpenBLAS
    openblas configuration: unknown
    pc file directory: unknown
    version: 0.3.18
  lapack:
    detection method: cmake
    found: true
    include directory: unknown
    lib directory: unknown
    name: OpenBLAS
    openblas configuration: unknown
    pc file directory: unknown
    version: 0.3.18
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 0.29.33
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  pythran:
    include directory: /tmp/pip-build-env-yjjniym9/overlay/lib/python3.9/site-packages/pythran
    version: 0.12.1
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /opt/python
```



Discussion:
Please use [`scipy.stats.Covariance`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.Covariance.html) for `multivariate_normal`. (It will make its way to `multivariate_t` in time.)

```python3
%timeit rng.multivariate_normal(mean, cov, method="cholesky")
# 1.69 s ± 247 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# this part takes a moment, but you only have to do it once
L = np.linalg.cholesky(cov)
cov_object = stats.Covariance.from_cholesky(L)  # <- this can be sped up a lot

%timeit stats.multivariate_normal.rvs(mean, cov_object)
# 12.4 ms ± 1.1 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

<details>

```python3
import numpy as np
from scipy import stats, linalg 

rng = np.random.default_rng(4292569835937)
data = rng.standard_normal((4096, 4096))
cov = data @ data.T
mean = np.zeros(4096)

L = np.linalg.cholesky(cov)
cov_object = stats.Covariance.from_cholesky(L)

%timeit rng.multivariate_normal(mean, cov, method="cholesky")
%timeit stats.multivariate_normal.rvs(mean, cov_object)
```

</details>

Actually, `from_cholesky` is much slower than it needs to be because it computes the `_covariance` attribute even if it's not needed. Instead, it can be an `@cached_property`. I will fix that, and then I think that the speed here will be comparable to NumPy even including the Cholesky decomposition.
Amazing, thanks! As it is, my current use case is `multivariate_t` - it looks like the logic for handling `Covariance` objects here is pretty simple: https://github.com/scipy/scipy/blob/c1ed5ece8ffbf05356a22a8106affcd11bd3aee0/scipy/stats/_multivariate.py#L397 

I could open a PR replicating that logic for `shape` in `multivariate_t` if that would help.
The bottleneck is typically on the review side. @tirthasheshpatel would you be willing to review this?

Note that the work is a bit more involved than what that one line can convey.  Please take a look at the changes made to `multivariate_normal` in gh-16002 to get a sense of what would be required. The basic idea is that if the input is an array instead of a `Covariance` object, it gets converted to one (`CovViaPSD`), and then all the methods of `multivariate_t` need to be refactored to use the methods of a `Covariance` object.

But actually @rishi-kulkarni could you start by changing this:

https://github.com/scipy/scipy/blob/c1ed5ece8ffbf05356a22a8106affcd11bd3aee0/scipy/stats/_covariance.py#L544

to do what some of the other classes do, e.g. 

https://github.com/scipy/scipy/blob/c1ed5ece8ffbf05356a22a8106affcd11bd3aee0/scipy/stats/_covariance.py#L599-L601

? I don't think it even needs a test because it is already covered in `scipy/stats/tests/test_multivariate.py::TestCovariance::test_covariance`:

https://github.com/scipy/scipy/blob/c1ed5ece8ffbf05356a22a8106affcd11bd3aee0/scipy/stats/tests/test_multivariate.py#L127

That would be a good start, and I could easily review that.
Yeah, that's easy enough - I can likely get to that this evening or so. 

As an aside, any reason these are all subclasses rather than class methods of `Covariance`?
Only the `Covariance` class is public. The `from_covariance` and similar are public static methods of `Covariance`. I thought having separate, private subclasses for the implementations was useful to keep things organized, since they all work a bit differently under the hood. Users can actually create their own subclasses... once this all settles for a bit, we will document how.

Resolving patch:
diff --git a/scipy/stats/_covariance.py b/scipy/stats/_covariance.py
--- a/scipy/stats/_covariance.py
+++ b/scipy/stats/_covariance.py
@@ -541,10 +541,13 @@ def __init__(self, cholesky):
         self._factor = L
         self._log_pdet = 2*np.log(np.diag(self._factor)).sum(axis=-1)
         self._rank = L.shape[-1]  # must be full rank for cholesky
-        self._covariance = L @ L.T
         self._shape = L.shape
         self._allow_singular = False
 
+    @cached_property
+    def _covariance(self):
+        return self._factor @ self._factor.T
+
     def _whiten(self, x):
         res = linalg.solve_triangular(self._factor, x.T, lower=True).T
         return res


Test patch: