Using datapipes over variables already in memory

franchesoni: <p>The only <code>IterDataPipe</code> that carries an object is <code>IterableWrapper</code>. However, I found that the following (which loads subtables from inside a dataframe) works</p>
<pre><code class="lang-python">def build_simplest_dp(input_df, table_rows=1, batch_size=1):
    df  = input_df  # shortname
    dp = IterableWrapper(df.index)
    dp = dp.batch(table_rows)
    dp = dp.map(lambda idx: df.loc[idx])  # get batch
    dp = dp.batch(batch_size).map(list)
    return dp
</code></pre>
<p>as you can see, the final <code>dp</code> carries <code>input_df</code> after it has been passed on <code>lambda idx: df.loc[idx]</code>.<br>
Is this too hacky?<br>
Which is the correct way to do this?<br>
One drawback is that I don’t know if <code>input_df</code> is accessible from <code>dp</code>, even if it’s inside. What are the other drawbacks?</p>

ejguan: <p>This is interesting use case. Technically, I don’t see drawback. But, it won’t gain the benefit to stream data. Since you have held dataframe into memory, you could simply iterate over the in-memory dataframe.</p>

franchesoni: <p>That is exactly what I would like to do,</p>
<aside class="quote no-group" data-username="ejguan" data-post="2" data-topic="150675">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/48.png" class="avatar"> ejguan:</div>
<blockquote>
<p>simply iterate over the in-memory dataframe</p>
</blockquote>
</aside>
<p>The thing is, batching, casting or augmenting are easier using datapipes. I would also like to get the dataloader benefits of prefetching and multiprocessing. This usecase is much more needed when using heavy augmentations.</p>
<p>My personal exp is that when using distributed training (e.g. training using pytorch-lightning) the dataloading was broken at some point because of the lambda functions. The following code did work instead:</p>
<pre data-code-wrap="python"><code class="lang-python">def get_batch(idx, df):
    return df.loc[idx]

def build_simplest_dp(input_df, table_rows=1, batch_size=1):
    df  = input_df  # shortname
    get_batch_fn = partial(get_batch, df=df)
    dp = IterableWrapper(df.index)
    dp = dp.batch(table_rows)
    dp = dp.map(get_batch_fn)  # get batch
    dp = dp.batch(batch_size, wrapper_class=list)
    return dp
</code></pre>
<p>To be honest, I didn’t want to dig deep enough to understand the main problem, but it was something about the multiprocessing (or multithreading) and the pickling of some functions.</p>
<p>As I have solved the problem, I will close this issue, but it is for you to evaluate how common will be the “use datapipes to work with things in memory” usecase, and also which would be the “proper” way to do it.</p>

ejguan: <aside class="quote no-group" data-username="franchesoni" data-post="3" data-topic="150675">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/franchesoni/48/32282_2.png" class="avatar"> franchesoni:</div>
<blockquote>
<p>My personal exp is that when using distributed training (e.g. training using pytorch-lightning) the dataloading was broken at some point because of the lambda functions. The following code did work instead:</p>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>Yeah. Lambda function doesn’t work well with multiprocessing. We do have a mechanism to override the behavior, but you need to have <code>dill</code> installed.</p>