akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loading model weights that was trained on multiple GPUs using nn.DataParallel

username: Yangmin
cooked: <p>Hello, I have an issue with using ResNet50 trained model weights that were trained on multi-GPUs.<br>
My goal is using only 1GPU and ResNet to train Generator.</p>
<p>First, I saved the ResNet model weights that were trained with nn.DataParallel().<br>
Then, I loaded the weights and initialized like the code below.</p>
<p>model = nn.DataParallel(model)<br>
checkpoint = torch.load(fpath)<br>
model.module.load_state_dict(checkpoint[‘state_dict’])</p>
<p>Second, I freezed the ResNet model and only trained the Generator with the output of ResNet model features.</p>
<p>The problem is the generator is not properly trained at all!<br>
But weird thing is that when I used ResNet50 model that was <strong>only trained on 1 GPU, and load the weights, the Generator is properly trained!.</strong> I also, checked the ResNet50 model performance after loading through torch.load(),  the performance is still the same.</p>
<p>I really cannot figure out what the problem is here. I checked the model weights properly loaded that was trained with nn.DataParallel.</p>
<ol>
<li>Is it not possible to use model weights trained with nn.DataParallel??</li>
<li>Do I need to do something more rather than loading ResNet50 model?</li>
</ol>
<p>What am I doing wrong?? Can someone tell me.</p>

username: ptrblck
cooked: <ol>
<li>No, it’s possible to load model parameters trained with <code>DataParallel</code>.</li>
<li>Usually, you would load the <code>state_dict</code> before wrapping the model into <code>nn.DataParallel</code>, but I don’t think that’s the issue here.</li>
</ol>
<aside class="quote no-group" data-username="Yangmin" data-post="1" data-topic="143352">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yangmin/48/43502_2.png" class="avatar"> Yangmin:</div>
<blockquote>
<p>Second, I freezed the ResNet model and only trained the Generator with the output of ResNet model features.</p>
</blockquote>
</aside>
<p>Based on your code snippet the <code>Generator</code> is not defined and is another model, which gets the outputs of the pretrained ResNet which is wrapped into <code>nn.DataParallel</code>?<br>
If so, I’m unsure I understand the question completely, as the issue seems to come from the <code>Generator</code>, not the ResNet.</p>
Foward autodiff for function taking single input

username: timudk
cooked: <p>Hi there,</p>
<p>I have a function <code>f : R -&gt; R^d</code>, <code>d &gt;&gt; 1</code>, for which I would like to compute a batched-derivative. Since <code>d</code> is large, I assume that ForwardAD is better suited than BackwardAD.</p>
<p>My attempt is the following:</p>
<pre><code class="lang-auto">tangent = torch.ones_like(x_in).to('cuda')
        with fwAD.dual_level():
            dual_input = fwAD.make_dual(x_in, tangent)
            dual_output = model(dual_input)
            y, jvp_fwAD = fwAD.unpack_dual(dual_output)
</code></pre>
<p>However, it seems that this approach is not significantly faster/memory efficient than simply using</p>
<pre><code class="lang-auto">v = torch.ones_like(t_in).to('cuda')
y, jvp_jvp = jvp(model, x_in, v)
</code></pre>
<p>which does two backward passes?</p>
<p>Any idea what’s the best way to take a derivative w.r.t. such a function?</p>
How to get indices of items in "iters" created using BucketIterator?

username: Gaurav_Sharma
cooked: <p>I have created iters(train, test, and validation) using BucketIterator.splits. I want to know if there is any way I can get the list of indices of items landed in the “test_iter” and further in batches? Thank you in advance!</p>
<pre><code class="lang-auto">train_iter, test_iter, val_iter = BucketIterator.splits(
                                    (train_data, test_data, val_data),
                                    device = device,
                                    batch_size = 250,
                                    sort_within_batch = True,
                                  sort_key = lambda x: len(x.mml))
</code></pre>