Repository: sktime/sktime

Issue:
[BUG] NaNs returned by transformations.series.difference.Differencer
**Describe the bug**
this was spotted by Lukasz Mentel (@lmmentel) thanks for this. The anomaly arises when running transformations testing locally. Strangely, this is not picked up with the repo tests. Differencer in transformations.series.difference return the same size data frame as input, which means for first order difference it returns nans in the first row
 
**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
from sktime.transformations.series.difference import Differencer
df = pd.DataFrame({'a': [1, 2, 3, 4, 5, 6],
                   'b': [1, 1, 2, 3, 5, 8],
                   'c': [1, 4, 9, 16, 25, 36]})
diff = Differencer()
print(df.diff())
```
outputs 
![image](https://user-images.githubusercontent.com/9594042/143411260-750b4921-f771-4737-aaa7-56d7c7a9be7c.png)
 Is it meant to have nans? If so, the tests need changing. If you run the test function in transformations/tests/test_all_transformers locally you get this output

![image](https://user-images.githubusercontent.com/9594042/143410700-efd6949b-0831-47d1-938d-1c167642d46c.png)



**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->

**Additional context**
<!--
Add any other context about the problem here.
-->
![image](https://user-images.githubusercontent.com/9594042/143410994-9761746e-11e2-4164-83c0-30097f975a3a.png)

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:
 
from sktime import show_versions; show_versions()
![image](https://user-images.githubusercontent.com/9594042/143410994-9761746e-11e2-4164-83c0-30097f975a3a.png)


-->


</details>

<!-- Thanks for contributing! -->



Discussion:
so this is a pandas version issue I think. It works on build-and-test because that installs earlier pandas version (for 3.6 test it installs pandas == 1.1.5). When I install 1.1.5, the tests all pass
The error I'm getting is 
```
______________________________________________________________________ test_differencer_prediction[lags7-y1] _______________________________________________________________________

y =          Passengers 1  Passengers 2
1949-01         112.0         112.0
1949-02         118.0         118.0
1949-03   ...    461.0         461.0
1960-11         390.0         390.0
1960-12         432.0         432.0

[144 rows x 2 columns]
lags = array([5, 7, 1])

    @pytest.mark.parametrize("y", test_cases)
    @pytest.mark.parametrize("lags", lags_to_test)
    def test_differencer_prediction(y, lags):
        y_train = y.iloc[:-12].copy()
        y_true = y.iloc[-12:].copy()
    
        transformer = Differencer(lags=[1, 12])
>       y_transform = transformer.fit_transform(y)

../sktime-venv/lib/python3.8/site-packages/sktime/transformations/series/tests/test_differencer.py:54: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../sktime-venv/lib/python3.8/site-packages/sktime/transformations/base.py:99: in fit_transform
    return self.fit(Z).transform(Z)
../sktime-venv/lib/python3.8/site-packages/sktime/transformations/series/difference.py:293: in transform
    Zt = self._transform(Z, X=X)
../sktime-venv/lib/python3.8/site-packages/sktime/transformations/series/difference.py:198: in _transform
    Zt = _diff_transform(Z, self._lags)
../sktime-venv/lib/python3.8/site-packages/sktime/transformations/series/difference.py:51: in _diff_transform
    Zt = Zt.diff(lag)

...

    def diff(self, periods: int = 1, axis: Axis = 0) -> DataFrame:
        if not isinstance(periods, int):
            if not (is_float(periods) and periods.is_integer()):
>               raise ValueError("periods must be an integer")
E               ValueError: periods must be an integer

../sktime-venv/lib/python3.8/site-packages/pandas/core/frame.py:8428: ValueError
```

So it seems that `lag` variable in the call `Zt = Zt.diff(lag)` is something that pandas does not like. 

It seems that there has been some changes to the `diff` method that introduce type checking. I tried to narrow it down to a specific version of pandas and found this https://github.com/pandas-dev/pandas/pull/36710 I couldn't find any mention of it in the release notes but it's tagged with 1.2 milestone.
I've isolated a minimal example

```
import pandas as pd

from sktime.datasets import load_airline
from sktime.transformations.series.difference import Differencer


y_airline = load_airline()
y_airline_df = pd.concat([y_airline, y_airline], axis=1)
y_airline_df.columns = ["Passengers 1", "Passengers 2"]

test_cases = [y_airline, y_airline_df]

transformer = Differencer(lags=1)
y_transform = transformer.fit_transform(y_airline_df)
```

which fails with

```
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_172444/3364667273.py in <module>
     12 
     13 transformer = Differencer(lags=1)
---> 14 y_transform = transformer.fit_transform(y_airline_df)

~/projects/sktime/sktime/transformations/base.py in fit_transform(self, Z, X)
     97         if X is None:
     98             # Fit method of arity 1 (unsupervised transformation)
---> 99             return self.fit(Z).transform(Z)
    100         else:
    101             # Fit method of arity 2 (supervised transformation)

~/projects/sktime/sktime/transformations/series/difference.py in transform(self, Z, X)
    291         Z = check_series(Z)
    292 
--> 293         Zt = self._transform(Z, X=X)
    294 
    295         return Zt

~/projects/sktime/sktime/transformations/series/difference.py in _transform(self, Z, X)
    196             The transformed timeseries.
    197         """
--> 198         Zt = _diff_transform(Z, self._lags)
    199         if self.drop_na:
    200             Zt = Zt.iloc[self._cumulative_lags[-1] :]

~/projects/sktime/sktime/transformations/series/difference.py in _diff_transform(Z, lags)
     49     else:
     50         for lag in lags:
---> 51             Zt = Zt.diff(lag)
     52         return Zt
     53 

~/projects/sktime-venv/lib/python3.8/site-packages/pandas/core/frame.py in diff(self, periods, axis)
   8426         if not isinstance(periods, int):
   8427             if not (is_float(periods) and periods.is_integer()):
-> 8428                 raise ValueError("periods must be an integer")
   8429             periods = int(periods)
   8430 

ValueError: periods must be an integer
```

However if I use the univariate series `y_airline` instead of `y_airline_df` it works just fine. My suspicion is that diff might need both arguments `periods` and `axis` set to work properly for a dataframe instead of a positional argument. Will investigate further.

Resolving patch:
diff --git a/sktime/transformations/series/difference.py b/sktime/transformations/series/difference.py
--- a/sktime/transformations/series/difference.py
+++ b/sktime/transformations/series/difference.py
@@ -5,6 +5,8 @@
 __author__ = ["Ryan Kuhns"]
 __all__ = ["Differencer"]
 
+from typing import Union
+
 import numpy as np
 import pandas as pd
 from sklearn.utils import check_array
@@ -40,16 +42,16 @@ def _check_lags(lags):
     return lags
 
 
-def _diff_transform(Z, lags):
+def _diff_transform(Z: Union[pd.Series, pd.DataFrame], lags: np.array):
+    """Perform differencing on Series or DataFrame."""
     Zt = Z.copy()
 
-    if len(lags) == 0:
-        return Zt
-
-    else:
+    if len(lags) != 0:
         for lag in lags:
-            Zt = Zt.diff(lag)
-        return Zt
+            # converting lag to int since pandas complains if it's np.int64
+            Zt = Zt.diff(periods=int(lag))
+
+    return Zt
 
 
 def _inverse_diff(Z, lag):
@@ -290,9 +292,7 @@ def transform(self, Z, X=None):
         self.check_is_fitted()
         Z = check_series(Z)
 
-        Zt = self._transform(Z, X=X)
-
-        return Zt
+        return self._transform(Z, X=X)
 
     def inverse_transform(self, Z, X=None):
         """Reverse transformation on input series `Z`.
@@ -310,6 +310,4 @@ def inverse_transform(self, Z, X=None):
         self.check_is_fitted()
         Z = check_series(Z)
 
-        Z_inv = self._inverse_transform(Z, X=X)
-
-        return Z_inv
+        return self._inverse_transform(Z, X=X)


Test patch: