akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Help understanding data collectors

icarosadero: <p>I am following the RNN tutorial for reinforcement learning in PyTorch from this <a href="https://docs.pytorch.org/rl/stable/tutorials/dqn_with_rnn.html" rel="noopener nofollow ugc">link</a>. In there, I came across the <code>SyncDataCollector</code> class, whose documentation looked a bit confusing. I have the following questions:</p>
<ul>
<li>What is a <code>frame</code>? Is a single frame just the state of the environment?</li>
<li>Assuming that a frame is just the state, what is a <code>batch</code>? If I choose a fixed number for the batch, is it going to concatenate simulations back to back until the number of states match it? Does it always start a fresh new simulation on every batch?</li>
<li>Is an <code>iter</code> a set of <code>batches</code>? Would the batches start a fresh new environment if I pass <code>reset_at_each_iter</code> as <code>True</code>?</li>
<li>For my case, it is important that the environment always start with the same seed when it calls for <code>env.reset</code>. How do I tell the data collector that it should begin with a fixed seed?</li>
</ul>
<p>Thank you in advance</p>

vmoens: <aside class="quote no-group" data-username="icarosadero" data-post="1" data-topic="221438">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/icarosadero/48/16461_2.png" class="avatar"> icarosadero:</div>
<blockquote>
<ul>
<li>What is a <code>frame</code>? Is a single frame just the state of the environment?</li>
</ul>
</blockquote>
</aside>
<p>A “frame” is (arguably a poorly chosen term) for a step in the environment</p>
<blockquote>
<p>Assuming that a frame is just the state, what is a <code>batch</code>?</p>
</blockquote>
<p>In the context of collectors, a batch is an ensemble of frames. If you say: <code>frames_per_batch=10</code> you are saying “I want to collect batches that have 10 steps in it”</p>
<ul>
<li>
<blockquote>
<p>Is an <code>iter</code> a set of <code>batches</code>? Would the batches start a fresh new environment if I pass <code>reset_at_each_iter</code> as <code>True</code>?</p>
</blockquote>
</li>
</ul>
<p>An iter of the collector gives you one batch:</p>
<pre><code class="lang-auto">for batch in collector:
    assert batch.numel() == collector.frames_per_batch
</code></pre>
<p>and yes if you set <code>reset_at_each_iter</code> to <code>True</code> you’ll get</p>
<pre><code class="lang-auto">for batch in collector:
    assert batch["is_init"][..., 0].all()
    assert batch.numel() == collector.frames_per_batch
</code></pre>
<p>(assuming your env has an <code>InitTracker</code> transform appended)</p>
<ul>
<li>
<blockquote>
<p>For my case, it is important that the environment always start with the same seed when it calls for env.reset. How do I tell the data collector that it should begin with a fixed seed?</p>
</blockquote>
</li>
</ul>
<p>that should be easy to implement with a very small transform that passes the seed to the env during reset. How does the resetting and seeding look like in your env?</p>
Should we split the trajectories prior to calculating the loss for a DQN?

icarosadero: <p>Consider this minimal example where data is sampled from a collector according to a MLP policy:</p>
<pre data-code-wrap="python"><code class="lang-python">from torchrl.collectors import SyncDataCollector
from torchrl.modules import MLP, QValueModule
from torchrl.collectors.utils import split_trajectories

mlp = Mod(
    MLP(
        out_features = env.action_spec.shape[-1],
        num_cells = [10],
        device = device
    ),
    in_keys = ["observation"],
    out_keys = ["action_value"],
).to(device)
qval = QValueModule(spec=env.action_spec)
test_policy = Seq(mlp, qval)
test_policy(env.reset(seed = 0))

collector = SyncDataCollector(
    env,
    test_policy,
    frames_per_batch=frames_per_batch,
    total_frames=-1
)
loss = DQNLoss(test_policy, action_space=env.action_spec, loss_function='smooth_l1')

data = next(iter(collector))
print("Without split", loss(data.to(device))["loss"])
print("With split", loss(split_trajectories(data).to(device))["loss"])
</code></pre>
<p>The outputs are:</p>
<pre><code class="lang-auto">Without split tensor(91.7286, device='cuda:0', grad_fn=&lt;MeanBackward0&gt;)
With split tensor(33.9880, device='cuda:0', grad_fn=&lt;MeanBackward0&gt;)
</code></pre>
<p>Which of these two cases should I use for training a model?</p>

vmoens: <p>I don’t have access to the env and other params in this script but my take is that your second example (with split) has padded data as input and the loss is computed over all the data, padded and non-padded (I don’t think DQN checks for a mask key when being executed). If that’s a desired feature feel free to open an issue!</p>
<p>But TLDR the first (non-split) is the right one - no need to split the data, everything is designed to work with concatenations of trajectories.</p>
Dynamic dataloader with accelerator `.prepare()`

pbb: <p>I need to design training algorithms that will dynamically add data to the dataset. I wonder what the best practice is for integrating such a dynamic dataloader with <code>accelerator</code>’s <code>.prepare()</code> function?</p>
<p>It seems like some “uncommon” training dataloader iteration will lead to unexpected behavior:</p>
<blockquote>
<p>E.g., manually iterating dataloader with <code>iter</code> prepared by accelerator accelerator’s step will be disturbed, and gradient accumulation will behave abnormally.</p>
</blockquote>
<p>Hence, I’m worried that these uncommon dataloader operations during the training will disturb the accelerator’s internal state. What is the best practice of using <code>accelerator</code> with a dynamic dataloader?</p>

ptrblck: <p>Since your question seems to be specific to the <code>accelerator</code> class you might want to cross-post it in a HF discussion board. For plain PyTorch <code>DataLoader</code>s you could check if a streaming dataset would help and would fit your use case.</p>