akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to efficient use dataset and dataloader to get data from the inference of pre-trained model

Max.T: <p>Hi, I am training two models in GAN fashion, training one during the other being frozen. Besides, model A’s input is model B’s output. Both model A and B are auto-regressive model - the forward method is a little bit different from the inference, because I use teach forcing during forward.</p>
<p>Now when I train model A, I need to do inference of B. I try to develop an efficient data pipeline relying on dataloader and dataset. Should I use the approach below ?:</p>
<pre><code class="lang-auto">dev myDataset():
    def __init__(self):
        ...
    def __len__(self):
        ...
    def __getitem__(self,batch,model_B):
        return model_B.inference(batch)

train_loader = DataLoader(myDataset, ...)
</code></pre>
<p>For every couple epochs, I will update model_B, so the data cannot be prepared in advance. I measured the inference consuming time, it will take about 2 seconds per batch. I have about 10,000 batches for training. Could anyone tell me the aforementioned way is efficient? - By efficient, I mean it can do parallel processing and prefetching (model_B can keep inferencing while model_A is being trained)?</p>
<p>Thank you very much!</p>
Pack_padded_sequence and pad_packed_sequence with variable length input that starts at different time steps

sf23: <p>Hi, based on my understanding of the pack_padded_sequence and pad_packed_sequence methods for RNNs in PyTorch, it seems like sequences of variable lengths are expected to be padded at the end.</p>
<p>For example (where the padding value is 0):</p>
<pre><code class="lang-auto">[1, 2, 3, 4, 0]
[1, 2, 0, 0, 0]
[1, 0, 0, 0, 0]
</code></pre>
<p>However, I am wondering if there is any way I can workaround this / if these methods can handle sequences of variable length that start at varying “time steps”. For example if I have sequences (expressed with padding_value = 0):</p>
<pre><code class="lang-auto">[0, 0, 1, 2, 3]
[0, 1, 2, 3, 0]
[1, 0, 0, 0, 0]
</code></pre>
<p>Is there any way I can handle that in PyTorch / with pack_padded_sequence and pad_packed_sequence.</p>
<p>Thanks in advance! And apologies if the answer is somehow obvious and I’m missing it.</p>

vdw: <p>As I far as I know, I don’t think that’s possible. Not 100% sure, though.</p>
<p>However, what would be the use case? Or w.r.t. your example, what’s the difference between <code>[0, 0, 1, 2, 3]</code> and <code>[0, 1, 2, 3, 0]</code>? Both are effectively the same sequences. So you could convert your inputs to:</p>
<pre><code>[1, 2, 3]
[1, 2, 3]
[1, 0, 0]</code></pre>

sf23: <p>The idea I want to capture is that these different sequences occur at different points in a lifetime. So for an example if I have data that represents info on a given person through the span of their lifetime, but one sequence starts at age 50 while another starts at age 35, I’d like to be able to capture these different ‘start times’  by varying their position within a fixed length sequence.</p>
<p>I suspected as much with pack_padded_sequence / pad_packed_sequence but any other ideas for how to represent these different start times are also much appreciated.</p>

ebarsoum: <p>Why this differences aren’t in the data itself?</p>

vdw: <p>Then I would argue that it’s not an issue of padding. Padding is used to “fill up” sequences to enable efficient batch processing. Ideally, a network learns that padding (e.g., index 0) means nothing and can/should be ignored.</p>
<p>For you, there is a semantic meaning that a sequences hasn’t started yet. So your batch would look like:</p>
<pre><code>[4, 4, 1, 2, 3]
[4, 1, 2, 3, 0]
[1, 0, 0, 0, 0]
</code></pre>
<p>Where index 4 represents “nothing happened yet” or something.</p>
<p>I’m not really sure if this makes semantically sense, but I would still argue that it’s not the same as padding.</p>

sf23: <p>Fair enough—this is something I’m thinking about as well. Thank you!</p>

sf23: <p>Ah, I see what you’re saying. Using a different token to represent “unstated” sequences could be a good idea, I’ll look into this as well. Thanks for your help!</p>

vdw: <p>By the way, this is less a PyTorch question but more a conceptual one. No harm asking here, of course, but you might also want to try machine learning forums with focus on theory (and less on a specific framework).</p>

abediee: <p>Did you find a solution to this problem?</p>
Inconsistency between traced and true model outputs

SandSt0rm161: <p>I am trying to serialize my model to run in C++, but even before C++ I’m testing in python both the original and the trace model outputs on the same input and get different results, here is my python code:</p>
<pre><code class="lang-auto">import torch
from torchvision import models
import torch.nn as nn

model_file = r'D:\workspace\....\model.pt'
model_ft = models.resnet50()
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 6)
model_ft.load_state_dict(torch.load(model_file))

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model_ft, example)

# Test both outputs of original and traced model to compare
model_ft.eval()
with torch.no_grad():
    output_model = model_ft(torch.zeros(1, 3, 224, 224))

output_traced_model = traced_script_module(torch.zeros(1, 3, 224, 224))

print('output_model = ' + str(output_model))
print('output_traced_model = ' + str(output_traced_model))
# save traced model
traced_script_module.save("traced_resnet_model.pt")
</code></pre>
<p>and variables 'output_model ’ and 'output_traced_model ’ are completely different:</p>
<pre><code class="lang-auto">output_model = tensor([[-0.0805,  0.2096,  0.0873, -0.0468, -0.1598, -0.0375]])
output_traced_model = tensor([[-0.4763,  1.5731,  0.2112,  0.3496, -1.6906,  0.0191]],
       grad_fn=&lt;DifferentiableGraphBackward&gt;)
</code></pre>

ptrblck: <p>The difference is created, because you are running the eager model in <code>eval()</code>, while the traced model was in <code>train()</code> mode.<br>
From the <a href="https://pytorch.org/docs/master/generated/torch.jit.trace.html#torch.jit.trace">docs</a>:</p>
<blockquote>
<p>In the returned <a href="https://pytorch.org/docs/master/generated/torch.jit.ScriptModule.html#torch.jit.ScriptModule"><code>ScriptModule</code></a>, operations that have different behaviors in  <code>training</code>  and  <code>eval</code>  modes will always behave as if it is in the mode it was in during tracing, no matter which mode the ScriptModule is in.</p>
</blockquote>

SandSt0rm161: <p>Thanks for the reply.<br>
I confirm that switching the original model to eval() mode before creating the traced model solves the issue.</p>