PyTorch Forecasting Dataloader/TimeSeriesDataSet with two differently scaled DataFrames (by unix) as Input/Output

username: eTuDpy
cooked: <p>I am trying to apply <code>pytorch_forecasting.TimeSeriesDataSet</code> from PyTorch Forecasting. My difficulty is, that I have two differently <strong>scaled &amp; shaped</strong> DataFrames as input and output data.</p>
<p>Simply put, my <code>input_df</code> looks like this:</p>
<pre><code>    unix_timestamp[ms]         value_a         value_b
0        1609455600000               2               3 
1        1609455600010               2               4
2        1609455600020               4               5
3        1609455600030               6               6
...                ...             ...             ...
</code></pre>
<p>Where the <code>unix_timestamp</code> is a running integer in milliseconds and each row represents the value of a 10ms interval.</p>
<p>Whereas my <code>output_df</code> looks like this:</p>
<pre><code>    unix_timestamp[ms]    target_value
0        1609455600000               9
1        1609455660000               8
2        1609455720000               7
3        1609455780000               6
...                ...             ... 
</code></pre>
<p>In this case, each row represents the value of a 1-minute interval!</p>
<p>Now I would like to use a time window of 10 minutes from the <code>input_df</code> (so 600000 ms and therefore 60000 rows) to predict 1 minute of the <code>output_df</code> (therefore 1 row).</p>
<p>How do I use <code>pytorch_forecasting.TimeSeriesDataSet</code> to prepare these two DataFrames this way?</p>
<p><strong>Important Note I:</strong><br>
The <code>unix_timestamp</code> of the two DataFrames do not necessarily overlap like shown it the example above. So for instance, if the <code>input_df</code> has a timestamp of 1601596805783 which corresponds to the ‘2020-10-02T00:00:05.783’, it does not mean that this exact timestamp exists in the <code>output_df</code>. It might be very close, but mostly not on point to the exact millisecond and off by a couple of milliseconds.</p>
<p><strong>Important Note II:</strong><br>
I thought about just scaling up the <code>output_df</code> to the same scale by repeating the value within the affiliated time interval, however, as far as I can judge this should distort the prediction result, shouldn´t it?</p>

username: nivek
cooked: <aside class="quote no-group" data-username="eTuDpy" data-post="1" data-topic="142776">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/etudpy/48/46155_2.png" class="avatar"> eTuDpy:</div>
<blockquote>
<p>pytorch_forecasting</p>
</blockquote>
</aside>
<p>Since the <code>pytorch_forecasting</code> library is not maintained by the PyTorch team, you may get a better response if you ask this question in their repository/forum.</p>
<p><code>DataLoader</code> does allow you to pass in a custom <code>batch_sampler</code>, which will allow you to specify how your sampling process work and get 10 minutes of input data at a time. You can find more details on this page <a href="https://pytorch.org/docs/stable/data.html#loading-batched-and-non-batched-data" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 2.1 documentation</a>.</p>