Pytorch Forecasting: Loading a custom dataset

Keyboard_Crasher: <p>Greetings, everyone!</p>
<p>I’m having trouble with loading custom datasets into PyTorch Forecasting. I already posted the question to <a href="https://stackoverflow.com/questions/69530665/pytorch-forecasting-loading-a-custom-dataset" rel="noopener nofollow ugc">Stack Overflow</a> but it seems that I might find the answer here here’s the message pasted for your convenience:</p>
<p>I’m trying to load a custom dataset to PyTorch Forecasting by modifying the example given in <a href="https://github.com/jdb78/pytorch-forecasting/blob/master/docs/source/tutorials/building.ipynb" rel="noopener nofollow ugc">this Github repository</a>. However I’m stuck at instantiating the <code>TimeSeriesDataSet</code>. The relevant parts of the code are as follows:</p>
<pre><code class="lang-python">import numpy as np
import pandas as pd

df = pd.read_csv("data.csv")
print(df.shape) # (300, 8)

# Divide the timestamps so that they are incremented by one each row.
df["unix"] = df["unix"].apply(lambda n: int(n / 86400))

# Set "unix" as the index
#df = df.set_index("unix")

# Add *integer* indices.
df["index"] = np.arange(300)
df = df.set_index("index")

# Add group column.
df["group"] = np.repeat(np.arange(30), 10)

from pytorch_forecasting import TimeSeriesDataSet

target = ["foo", "bar", "baz"]

# Create the dataset from the pandas dataframe
dataset = TimeSeriesDataSet(
    df,
    group_ids                  = ["group"],
    target                     = target,
    time_idx                   = "unix",
    min_encoder_length         = 50,
    max_encoder_length         = 50,
    min_prediction_length      = 20,
    max_prediction_length      = 20,
    time_varying_unknown_reals = target,
    allow_missing_timesteps    = True
)
</code></pre>
<p>And the error message plus traceback:</p>
<pre><code class="lang-auto">/home/user/.virtualenvs/torch/lib/python3.9/site-packages/pytorch_forecasting/data/timeseries.py:1241: UserWarning: Min encoder length and/or min_prediction_idx and/or min prediction length and/or lags are too large for 30 series/groups which therefore are not present in the dataset index. This means no predictions can be made for those series. First 10 removed groups: [{'__group_id__group': 0}, {'__group_id__group': 1}, {'__group_id__group': 2}, {'__group_id__group': 3}, {'__group_id__group': 4}, {'__group_id__group': 5}, {'__group_id__group': 6}, {'__group_id__group': 7}, {'__group_id__group': 8}, {'__group_id__group': 9}]
  warnings.warn(

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
/tmp/ipykernel_822/3402560775.py in &lt;module&gt;
      4 
      5 # create the dataset from the pandas dataframe
----&gt; 6 dataset = TimeSeriesDataSet(
      7     df,
      8     group_ids                  = ["group"],

~/.virtualenvs/torch/lib/python3.9/site-packages/pytorch_forecasting/data/timeseries.py in __init__(self, data, time_idx, target, group_ids, weight, max_encoder_length, min_encoder_length, min_prediction_idx, min_prediction_length, max_prediction_length, static_categoricals, static_reals, time_varying_known_categoricals, time_varying_known_reals, time_varying_unknown_categoricals, time_varying_unknown_reals, variable_groups, constant_fill_strategy, allow_missing_timesteps, lags, add_relative_time_idx, add_target_scales, add_encoder_length, target_normalizer, categorical_encoders, scalers, randomize_length, predict_mode)
    437 
    438         # create index
--&gt; 439         self.index = self._construct_index(data, predict_mode=predict_mode)
    440 
    441         # convert to torch tensor for high performance data loading later

~/.virtualenvs/torch/lib/python3.9/site-packages/pytorch_forecasting/data/timeseries.py in _construct_index(self, data, predict_mode)
   1247                 UserWarning,
   1248             )
-&gt; 1249         assert (
   1250             len(df_index) &gt; 0
   1251         ), "filters should not remove entries all entries - check encoder/decoder lengths and lags"

AssertionError: filters should not remove entries all entries - check encoder/decoder lengths and lags
</code></pre>
<p>I have tried tweaking the initializer arguments without success. The file <code>timeseries.py</code> can be found in the same Github repository, at <code>pytorch_forecasting/data/timeseries.py</code>, line 1246. Cannot give a third link in a single post, unfortunately.</p>

ptrblck: <p>What happens if you remove the filtering by using 0 for the min. values and a huge number for the max. ones?</p>

Keyboard_Crasher: <p>That solves the loading issue. Thank you!</p>

Shahal_Kp: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I am facing the the same issue, It is working if i set min_encoder_length as 0 which leads to an error in building model.</p>
<p>training = TimeSeriesDataSet(data[lambda x: x.time_idx &lt;= training_cutoff],<br>
time_idx=“time_idx”,<br>
target=“P-L”,<br>
group_ids=[“MS”, “Amb”,“Load”],<br>
max_encoder_length=349,<br>
min_encoder_length=1,<br>
min_prediction_length=1,<br>
max_prediction_length=10,<br>
time_varying_known_categoricals=[“MS”, “Amb”,“Load”],<br>
allow_missing_timesteps=True,<br>
categorical_encoders={<br>
‘MS’: pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),<br>
‘Amb’:pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True),<br>
‘Load’:pytorch_forecasting.data.encoders.NaNLabelEncoder(add_nan=True)<br>
},<br>
time_varying_known_reals = <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
time_varying_unknown_reals = [“P-L”],<br>
)</p>
<p>Error:- AssertionError: filters should not remove entries all entries - check encoder/decoder lengths and lags</p>

ptrblck: <p>What’s the error when the min length is 0?</p>

Shahal_Kp: <p>if i use timeseriesdataset with min_encoder_length = 0, it is returning the following error while training the model:-</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abeec7598eb268bdf1bfc573162fed62ea5b4a48.png" data-download-href="https://discuss.pytorch.org/uploads/default/abeec7598eb268bdf1bfc573162fed62ea5b4a48" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abeec7598eb268bdf1bfc573162fed62ea5b4a48_2_690x113.png" alt="image" data-base62-sha1="owZeafNrntJ9BoE4LeoZRNlhuuA" width="690" height="113" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abeec7598eb268bdf1bfc573162fed62ea5b4a48_2_690x113.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abeec7598eb268bdf1bfc573162fed62ea5b4a48_2_1035x169.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abeec7598eb268bdf1bfc573162fed62ea5b4a48_2_1380x226.png 2x" data-dominant-color="F3F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1406×232 24 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Some encoder lengths seem to be empty, so you might need to make sure valid data is used.</p>