Repository: blue-yonder/tsfresh

Issue:
ValueError: The following ids are in the time series container but are missing in y:
Hello, I'm using tsfresh to extract features from univariate energy time series. On the newer versions >0.15 the same code that was working fine some months ago outputs the error above and a huge list of ids (input data is a year of 15-min sampled energy readings).

x=pd.Series(data=dfs[0].values.flatten(),index=dfs[0].index)
df_shift, y = make_forecasting_frame(x, kind="price", max_timeshift=10, rolling_direction=1)
X = extract_relevant_features(df_shift,y, column_id="id", column_sort="time", column_value="value", 
                     show_warnings=False)

If I downgrade to 0.12 for example it works fine. Could you please tell me what changed between versions with regard to the rolling mechanism or with the column ids for the feature extraction step? Thank you!

Discussion:
github-actions[bot]: You have style errors. See them below.

./tests/units/utilities/test_dataframe_functions.py:812:121: E501 line too long (130 > 120 characters)
./tests/units/utilities/test_dataframe_functions.py:813:72: E128 continuation line under-indented for visual indent
./tests/units/utilities/test_dataframe_functions.py:829:121: E501 line too long (143 > 120 characters)
github-actions[bot]: You have style errors. See them below.

./tests/units/utilities/test_dataframe_functions.py:812:121: E501 line too long (130 > 120 characters)
./tests/units/utilities/test_dataframe_functions.py:813:72: E128 continuation line under-indented for visual indent
./tests/units/utilities/test_dataframe_functions.py:829:121: E501 line too long (143 > 120 characters)
codecov-commenter: # [Codecov](https://codecov.io/gh/blue-yonder/tsfresh/pull/758?src=pr&el=h1) Report
> Merging [#758](https://codecov.io/gh/blue-yonder/tsfresh/pull/758?src=pr&el=desc) into [main](https://codecov.io/gh/blue-yonder/tsfresh/commit/1a4199b46ffdf623f7d9647c1385751cc33d4959?el=desc) will **decrease** coverage by `0.23%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/blue-yonder/tsfresh/pull/758/graphs/tree.svg?width=650&height=150&src=pr&token=SJbXAVYKe1)](https://codecov.io/gh/blue-yonder/tsfresh/pull/758?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##             main     #758      +/-   ##
==========================================
- Coverage   95.70%   95.47%   -0.24%     
==========================================
  Files          18       18              
  Lines        1654     1656       +2     
  Branches      330      331       +1     
==========================================
- Hits         1583     1581       -2     
- Misses         35       39       +4     
  Partials       36       36              
```


| [Impacted Files](https://codecov.io/gh/blue-yonder/tsfresh/pull/758?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [tsfresh/utilities/dataframe\_functions.py](https://codecov.io/gh/blue-yonder/tsfresh/pull/758/diff?src=pr&el=tree#diff-dHNmcmVzaC91dGlsaXRpZXMvZGF0YWZyYW1lX2Z1bmN0aW9ucy5weQ==) | `96.72% <100.00%> (+0.03%)` | :arrow_up: |
| [tsfresh/feature\_selection/significance\_tests.py](https://codecov.io/gh/blue-yonder/tsfresh/pull/758/diff?src=pr&el=tree#diff-dHNmcmVzaC9mZWF0dXJlX3NlbGVjdGlvbi9zaWduaWZpY2FuY2VfdGVzdHMucHk=) | `91.42% <0.00%> (-2.86%)` | :arrow_down: |
| [tsfresh/utilities/distribution.py](https://codecov.io/gh/blue-yonder/tsfresh/pull/758/diff?src=pr&el=tree#diff-dHNmcmVzaC91dGlsaXRpZXMvZGlzdHJpYnV0aW9uLnB5) | `88.88% <0.00%> (-1.86%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/blue-yonder/tsfresh/pull/758?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/blue-yonder/tsfresh/pull/758?src=pr&el=footer). Last update [1a4199b...e25017c](https://codecov.io/gh/blue-yonder/tsfresh/pull/758?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

github-actions[bot]: ## Result of Benchmark Tests
| Benchmark | Min | Max | Mean | Mean on Repo `HEAD` |
| :--- | :---: | :---: | :---: | :---: |
| tests/benchmark.py::test_benchmark_small_data| 11.22| 11.31| 11.26 +- 0.04 | 9.28 +- 0.03 |
| tests/benchmark.py::test_benchmark_large_data| 6.20| 6.28| 6.24 +- 0.03 | 5.97 +- 0.04 |
| tests/benchmark.py::test_benchmark_with_selection| 8.06| 8.16| 8.11 +- 0.04 | 7.38 +- 0.12 |


Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 974d248..15a7631 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -9,6 +9,7 @@ Unreleased
 
 - Bugfixes:
     - Fix typos in notebook (#757)
+    - Fix output format of `make_forecasting_frame` (#758)
 
 Version 0.17.0
 ==============
diff --git a/tsfresh/utilities/dataframe_functions.py b/tsfresh/utilities/dataframe_functions.py
index 2e40262..b8fe5c0 100644
--- a/tsfresh/utilities/dataframe_functions.py
+++ b/tsfresh/utilities/dataframe_functions.py
@@ -559,7 +559,16 @@ def make_forecasting_frame(x, kind, max_timeshift, rolling_direction):
     mask = df_shift.groupby(['id'])['id'].transform(mask_first).astype(bool)
     df_shift = df_shift[mask]
 
-    return df_shift, df["value"][1:]
+    # Now create the target vector out of the values
+    # of the input series - not including the first one
+    # (as there is nothing to forecast from)
+    y = df["value"][1:]
+
+    # make sure that the format is the same as the
+    # df_shift index
+    y.index = map(lambda x: ("id", x), y.index)
+
+    return df_shift, y
 
 
 def add_sub_time_series_index(df_or_dict, sub_length, column_id=None, column_sort=None, column_kind=None):


Test patch:
diff --git a/tests/units/utilities/test_dataframe_functions.py b/tests/units/utilities/test_dataframe_functions.py
index 5130875..c8dc883 100644
--- a/tests/units/utilities/test_dataframe_functions.py
+++ b/tests/units/utilities/test_dataframe_functions.py
@@ -9,6 +9,8 @@ import pandas as pd
 from pandas.testing import assert_frame_equal, assert_series_equal
 
 from tsfresh.utilities import dataframe_functions
+from tsfresh import extract_relevant_features
+from tsfresh.feature_extraction.settings import MinimalFCParameters
 from tests.fixtures import warning_free
 
 
@@ -786,7 +788,7 @@ class MakeForecastingFrameTestCase(TestCase):
                                     "value": [0, 1, 2],
                                     "time": [0, 1, 2]})
 
-        expected_y = pd.Series(data=[1, 2, 3], index=[1, 2, 3], name="value")
+        expected_y = pd.Series(data=[1, 2, 3], index=[("id", 1), ("id", 2), ("id", 3)], name="value")
         assert_frame_equal(df.sort_index(axis=1).reset_index(drop=True), expected_df.sort_index(axis=1))
         assert_series_equal(y, expected_y)
 
@@ -797,7 +799,9 @@ class MakeForecastingFrameTestCase(TestCase):
                                     "kind": ["test"] * 3,
                                     "value": np.arange(3),
                                     "time": [0, 1, 2]})
+        expected_y = pd.Series(data=[1, 2, 3], index=[("id", 1), ("id", 2), ("id", 3)], name="value")
         assert_frame_equal(df.sort_index(axis=1).reset_index(drop=True), expected_df.sort_index(axis=1))
+        assert_series_equal(y, expected_y)
 
     def test_make_forecasting_frame_pdSeries(self):
 
@@ -805,8 +809,8 @@ class MakeForecastingFrameTestCase(TestCase):
         df, y = dataframe_functions.make_forecasting_frame(x=pd.Series(data=range(4), index=t_index),
                                                            kind="test", max_timeshift=1, rolling_direction=1)
 
-        expected_y = pd.Series(data=[1, 2, 3], index=pd.DatetimeIndex(["2011-01-01 01:00:00", "2011-01-01 02:00:00",
-                                                                       "2011-01-01 03:00:00"], freq="H"), name="value")
+        time_shifts = pd.DatetimeIndex(["2011-01-01 01:00:00", "2011-01-01 02:00:00", "2011-01-01 03:00:00"], freq="H")
+        expected_y = pd.Series(data=[1, 2, 3], index=zip(["id"]*3, time_shifts), name="value")
         expected_df = pd.DataFrame({"id": list(zip(["id"] * 3, pd.DatetimeIndex(["2011-01-01 01:00:00",
                                                                                  "2011-01-01 02:00:00",
                                                                                  "2011-01-01 03:00:00"]))),
@@ -817,6 +821,14 @@ class MakeForecastingFrameTestCase(TestCase):
         assert_frame_equal(df.sort_index(axis=1).reset_index(drop=True), expected_df.sort_index(axis=1))
         assert_series_equal(y, expected_y)
 
+    def test_make_forecasting_frame_feature_extraction(self):
+        t_index = pd.date_range('1/1/2011', periods=4, freq='H')
+        df, y = dataframe_functions.make_forecasting_frame(x=pd.Series(data=range(4), index=t_index),
+                                                           kind="test", max_timeshift=1, rolling_direction=1)
+
+        extract_relevant_features(df, y, column_id="id", column_sort="time", column_value="value",
+                                  default_fc_parameters=MinimalFCParameters())
+
 
 class GetIDsTestCase(TestCase):