Repository: sktime/sktime

Issue:
[BUG] Wrong result for `predict_residuals` of`TransformedTargetForecaster` on training data
**Describe the bug**
If `predict_residuals` is called  of a `TransformedTargetForecaster` on the training data, the results seems to be wrong. My guess it that internally, the  `self._y` of the `TransformedTargetForecaster` is used to calculate the residuals instead of the transformed target data. 


**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
from datasets import load_longley
from forecasting.model_selection import temporal_train_test_split
from forecasting.sarimax import SARIMAX
from transformations.series.difference import Differencer

method = "predict_residuals"
y, X = load_longley()
y_train, y_test, X_train, X_test = temporal_train_test_split(y, X)
pipe = Differencer() * SARIMAX()
pipe.fit(y=y_train.to_frame(), X=X_train, fh=[1, 2, 3, 4])
result = pipe.predict_residuals()
result_test = pipe.predict_residuals(y=y_test, X=X_test)
```

In this example, result_test looks like residuals. However, result is far higher and not the residuals.

**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
Also on the training data the residuals should be calculated of the TransformedTargetForecaster applied on the training data.


<!-- Thanks for contributing! -->

[BUG] 'Differencer' returning strange results
**Describe the bug**

When calling fit_transform and inverse_transform sequentially, I don't get the original series back when applying one or several lags. It seems to work fine with a PeriodIndex (like the example in the documentation), but does not work with a datetimeindex. I reproduced the issue with one of the default datasets included with SKTime (but I need to run a simple groupby to get it to daily level to match my own).

**To Reproduce**
```python
from sktime.transformations.series.difference import Differencer
from sktime.datasets import load_solar
y = load_solar()
y = y.groupby(by=pd.Grouper(level=0, freq="D")).sum()
display(y.plot())
```
![image](https://github.com/sktime/sktime/assets/99316631/a546da5b-f4b9-4628-9840-fec9c20bda2f)

```python
transformer = Differencer(lags=[1, 7])
y_transform = transformer.fit_transform(y)
display(y_transform.plot())
```
![image](https://github.com/sktime/sktime/assets/99316631/505b06f7-dfcd-47d2-a6ec-45c913eaf00e)


```python
y_revert = transformer.inverse_transform(y_transform)
y_revert = pd.DataFrame(y_revert, index=y_transform.index)
y_revert.plot()
```
![image](https://github.com/sktime/sktime/assets/99316631/20d16adf-4a5b-4d31-a4db-13d177a8d421)


**Expected behavior**
The returned series should be identical (or very close)

**Additional context**
I have the same problem with a much longer timeseries.

**Versions**
<details>
System:
    python: 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 17:59:51) [MSC v.1935 64 bit (AMD64)]
executable: [c:\ProgramData\Miniconda3\envs\DLL_Forecast\python.exe](file:///C:/ProgramData/Miniconda3/envs/DLL_Forecast/python.exe)
   machine: Windows-10-10.0.19042-SP0

Python dependencies:
          pip: 23.2
       sktime: 0.21.0
      sklearn: 1.3.0
       skbase: 0.4.6
        numpy: 1.24.4
        scipy: 1.11.1
       pandas: 2.0.3
   matplotlib: 3.7.2
       joblib: 1.3.0
  statsmodels: 0.14.0
        numba: 0.57.1
     pmdarima: None
      tsfresh: None
   tensorflow: None
tensorflow_probability: None

</details>

<!-- Thanks for contributing! -->



Discussion:
My suspicion is that this combines two issues:

* `predict_residuals` without args computes the in-sample residuals on `y_train`
* `Differencer` may produce counterintuitive results in-sample https://github.com/sktime/sktime/issues/4965

As said, this is an unverified guess.
reproduced on current `main`, windows, python 3.11
@MCRE-BE, I've investigated this and it is quite subtle.

The point is, `y_transform` has zeroes at the first eight indices, since you ran `Differencer` with the zero-fill default, for values that cannot be known (since earlier values are unknown).
Inverting that series with consecutive lags [1, 7] does not, mathematically, produce the original series.
(this is true only for a single lag, not multiple consecutive lags)

The differencer is smart enough to remember the original values *if they are not seen as part of `y_transform`* and fills them in, but in your case `y_transform` does have zeroes there.

If you were to run `Differencer` with `na_handling="drop_na"` or `"keep_na"`, those values would not be there, and the inversion would give you what you expect:

```python
import pandas as pd
from sktime.transformations.series.difference import Differencer
from sktime.datasets import load_airline

y = load_airline()
y = y.reset_index(drop=True)

transformer = Differencer(lags=[1, 7], na_handling="keep_na")

# these twp are now equal
y_transform = transformer.fit_transform(y)
y_revert = transformer.inverse_transform(y_transform)
```

Please let me know if it's not clear what happens, and whether you consider this problematic or counterintuitive.

An alternative behaviour for the `Differencer`, when given multiple consecutive lags, could be to fill the last sequential difference that is known, as opposed to zero.

That is, in the example, in `y_transform`, the zero-th value would be identical with the zero-th value of the original series, values 1 to 6 would be filled with the first differences of the original series, and value 7 would be the difference between the zero-th value of the original series and the 7-th first difference. This series, when inverted with the consecutive cumsum at lags [7, 1] would produce the original series.

But the non-zero values at the start would be weird to explain and understand - is this worse or better than the "fill zero" default?

What do you think?
hmmm the difference is indeed quite subtile. I have to admit, this bug only exists because of coding errors on my side. I don't use SKTime Piplines (yet) so I did code some of the steps individually.

- I did fill NaN's and zero's (but poorly, so some remained causing the bug)
- In the example above, I didn't think about checking the default parameter for na_handling. In my code I chose "keep_na". 

What I can tell you is : I am personnally never in favour to automatically doing or filling stuff. I would rather recommend setting errors to limit the use of the Differencer to known conditions that work :

- Don't inverse a timeseries where there are zero's in the first X and na_handling = "fill_zero"
- Don't inverse a timeseries with nan's (you check the freq, but that doesn't solve nan's)
- etc...

Thanks for taking the time to look at the problem though ! I'll see if I can adapt my code to make everything work now :-)
Ok. I've been running some more tests on my own dataset and I can't figure out why it's bugging.
The code is a bit messy but : 

- I have a timeseries that represents sales.
- I dropped all outliers, nan's and zero's
- Filled all the missing values with a simple XGBoost model
- This results in the following target variable. As far as I can see : No Zero's, No Nan's, no missing data
[data.csv](https://github.com/sktime/sktime/files/12212708/data.csv)

- I run an XGBoost forecast and I get the output below
[forecast.csv](https://github.com/sktime/sktime/files/12212758/forecast.csv)

- Reverting the forecasted values gives something strange. Any idea why?

```python
from sktime.transformations.series.difference import Differencer
transformer = Differencer(lags=[1, 7], na_handling="keep_na")
y_diff = transformer.fit_transform(y)

y_diff = y_diff.dropna()

model = xgboost.XGBRegressor()
model = model.fit(x[x.index.isin(y_diff.index)], y_diff)
yhat = model.predict(.x)

yhat = pd.Series(yhat, xgb.FeatureDataframe.index, name="yhat")
yhat = yhat.asfreq("D")
transformer.inverse_transform(yhat).plot()
```

![image](https://github.com/sktime/sktime/assets/99316631/62749e6f-deb9-419f-aafa-1b793897eb48)


Resolving patch:
diff --git a/sktime/transformations/series/difference.py b/sktime/transformations/series/difference.py
--- a/sktime/transformations/series/difference.py
+++ b/sktime/transformations/series/difference.py
@@ -142,10 +142,24 @@ def _inverse_diff(X, lags, X_diff_seq=None):
 
     # invert last lag index
     if X_diff_seq is not None:
+        # Get the train time series before the last difference
         X_diff_orig = X_diff_seq[len(lags)]
+        # Shift the differenced time series index by the last lag
+        # to match the original time series index
         X_ix_shift = _shift(X.index, -lag_last)
+        # Get the original time series values for the intersecting
+        # indices between the shifted index and the original index
         X_update = X_diff_orig.loc[X_ix_shift.intersection(X_diff_orig.index)]
-
+        # Set the values of the differenced time series to nan for all indices
+        # that are in the indices of the original and the by the sum of all lags
+        # shifted original time series that are available in the differenced time
+        # series (intersection). These are the indices for which no valid differenced
+        # values exist.
+        X.loc[
+            X_diff_orig.index.difference(
+                _shift(X_diff_orig.index, sum(lags) + lag_last)
+            ).intersection(X.index)
+        ] = np.nan
         X = X.combine_first(X_update)
 
     X_diff_last = X.copy()


Test patch:
diff --git a/sktime/transformations/series/tests/test_differencer.py b/sktime/transformations/series/tests/test_differencer.py
--- a/sktime/transformations/series/tests/test_differencer.py
+++ b/sktime/transformations/series/tests/test_differencer.py
@@ -168,6 +168,17 @@ def test_differencer_cutoff():
     gscv.fit(train_model, X=X_train)
 
 
+@pytest.mark.parametrize("lags", lags_to_test)
+@pytest.mark.parametrize("index_type", ["int", "datetime"])
+def test_inverse_train_data_fill_zero(lags, index_type):
+    y = y_airline
+    if index_type == "int":
+        y = y.reset_index(drop=True)
+    diff = Differencer(lags).fit(y)
+    result = diff.inverse_transform(diff.transform(y))
+    _assert_array_almost_equal(result, y)
+
+
 def test_differencer_inverse_does_not_memorize():
     """Tests that differencer inverse always computes inverse via cumsum.