Repository: sktime/sktime

Issue:
[BUG] ConformalInterval Memory Usage
**Describe the bug**
Disclaimer: Not sure if really a bug or more a request.

Currently, the ConformalInterval is not very memory efficient, since it constructs internally a dataframe with the size of len(time series) x len(time series) (line 420). However, the matrix represented by this dataframe is sparse for two reasons:
1. It is an upper triangular matrix by definition.
2. Due to the `sample_frac` parameter, multiple rows will stay completely empty. 

Thus, I would propose to use a sparse matrix here instead of a normal dataframe. Thus, the definition of the dataframe would need to be replaced with: 

```python
residuals_matrix = pd.DataFrame.sparse.from_spmatrix(scipy.sparse.csr_array((len(full_y_index), len(full_y_index))), 
                                                             index=full_y_index, columns=full_y_index)
```



Discussion:
Small experiment to show the difference/effect:

```python
import pandas as pd
import time 
import sys
import scipy

t0 = time.time()
df_sparse = pd.DataFrame.sparse.from_spmatrix(scipy.sparse.csr_array((10000, 10000)), index=range(10000), columns=range(10000))
print("Time to create sparse df:", time.time() - t0, "seconds")

t0 = time.time()
df_dense = pd.DataFrame(index=range(10000), columns=range(10000))
print("Time to create dense df:", time.time() - t0, "seconds")

print("Size of sparse df:", sys.getsizeof(df_sparse), "bytes")
print("Size of dense df:", sys.getsizeof(df_dense), "bytes")
```

**Output**

```
Time to create sparse df: 1.962019443511963 seconds
Time to create dense df: 5.920591592788696 seconds
Size of sparse df: 164 bytes
Size of dense df: 3200000164 bytes
```

Resolving patch:
diff --git a/sktime/forecasting/conformal.py b/sktime/forecasting/conformal.py
--- a/sktime/forecasting/conformal.py
+++ b/sktime/forecasting/conformal.py
@@ -11,6 +11,7 @@
 
 import numpy as np
 import pandas as pd
+import scipy
 from joblib import Parallel, delayed
 from sklearn.base import clone
 
@@ -417,8 +418,10 @@ def _compute_sliding_residuals(
 
         full_y_index = y.iloc[n_initial_window:].index
 
-        residuals_matrix = pd.DataFrame(
-            columns=full_y_index, index=full_y_index, dtype="float"
+        residuals_matrix = pd.DataFrame.sparse.from_spmatrix(
+            scipy.sparse.csr_array((len(full_y_index), len(full_y_index))),
+            index=full_y_index,
+            columns=full_y_index,
         )
 
         if update and hasattr(self, "residuals_matrix_") and not sample_frac:
@@ -427,9 +430,9 @@ def _compute_sliding_residuals(
                 overlapping_index = pd.Index(
                     self.residuals_matrix_.index.intersection(full_y_index)
                 ).sort_values()
-                residuals_matrix.loc[
+                residuals_matrix = self.residuals_matrix_.loc[
                     overlapping_index, overlapping_index
-                ] = self.residuals_matrix_.loc[overlapping_index, overlapping_index]
+                ].combine_first(residuals_matrix)
             else:
                 overlapping_index = None
             y_index = remaining_y_index
@@ -466,8 +469,10 @@ def _get_residuals_matrix_row(forecaster, y, X, id):
             delayed(_get_residuals_matrix_row)(forecaster.clone(), y, X, id)
             for id in y_index
         )
-        for idx, id in enumerate(y_index):
-            residuals_matrix.loc[id] = all_residuals[idx]
+        all_residuals = pd.concat(all_residuals, axis=1, ignore_index=True).T
+        all_residuals.index = y_index
+        all_residuals = all_residuals.astype(pd.SparseDtype("float", 0))
+        residuals_matrix = all_residuals.combine_first(residuals_matrix)
 
         if overlapping_index is not None:
 
@@ -494,9 +499,10 @@ def _extend_residuals_matrix_row(y, X, id):
                 delayed(_extend_residuals_matrix_row)(y, X, id)
                 for id in overlapping_index
             )
-
-            for idx, id in enumerate(overlapping_index):
-                residuals_matrix.loc[id] = extend_residuals[idx]
+            extend_residuals = pd.concat(extend_residuals, axis=1, ignore_index=True).T
+            extend_residuals.index = overlapping_index
+            extend_residuals = extend_residuals.astype(pd.SparseDtype("float", 0))
+            residuals_matrix = extend_residuals.combine_first(residuals_matrix)
 
         return residuals_matrix
 


Test patch: