akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Are the parameters of nn.Module "leaf" nodes of the Graph?

crissallan: <p>Hello,</p>
<p>when I read the docs of torch.Tensor I found that " Only leaf Tensors will have their <a href="https://pytorch.org/docs/stable/autograd.html?highlight=grad#torch.Tensor.grad" rel="noopener nofollow ugc"><code>grad</code></a> populated during a call to <a href="https://pytorch.org/docs/stable/autograd.html?highlight=grad#torch.Tensor.backward" rel="noopener nofollow ugc"><code>backward()</code></a>. To get <a href="https://pytorch.org/docs/stable/autograd.html?highlight=grad#torch.Tensor.grad" rel="noopener nofollow ugc"><code>grad</code></a> populated for non-leaf Tensors, you can use <a href="https://pytorch.org/docs/stable/autograd.html?highlight=grad#torch.Tensor.retain_grad" rel="noopener nofollow ugc"><code>retain_grad()</code></a>."</p>
<p>So does that mean the parameters of nn.Module are leaf nodes too? Because, they has to hold the grad to execute the .step() method to update themselves.</p>
<p>Many thanks!</p>

albanD: <p>Hi,</p>
<p>Yes by design, all the <code>Parameters</code> are leaf Tensors that require gradients. So they will always have their .grad populated (unless you manually set the requires_grad field to False).</p>

crissallan: <p>Thanks a lot! I also had a question about the Autograd of torch::jit::script::Module of C++ API.<br>
I try to load a trained pytorch model into C++(let’s call this model in C++ “M”), first I do the forward process by:<br>
out = M(input);<br>
loss = loss_function(out, ground-truth)<br>
then I tried to use it do backward propagation to calculate the grad of it’s input tensor “input”. However I found that the .grad of “input” is always zero. And the .grad_fn of “output” is a “nullptr”.</p>
<p>Any solutions for this?</p>
<p>Many thanks!</p>

albanD: <p>Hi,</p>
<p>I am not as familiar with the jit c++ API.<br>
But did you make sure that the <code>input</code> require gradients? And if it is not a leaf, you can call <code>.retain_grad()</code> on it to ensure its <code>.grad</code> field is properly populated.<br>
Also make sure that you don’t disable the autograd with a NoGradGuard.</p>

crissallan: <p>Thanks for the suggestions! I’ll try them!</p>
Dataloader StopIteration takes a long time

WinKawaks: <p>After a dataloader has iterated all data, it can move on to the next iteration without reinitialising. I’ve seen code like this.</p>
<pre><code class="lang-auto">    dataloader = DataLoader()
    for epoch in range(10):
        for data, label in dataloader:
            train(data, label)
</code></pre>
<p>I observed that after each round of dataloader finished loading data, a new round took a very long time to start. So I wrote a program to verify this process.</p>
<pre><code class="lang-auto">    dataloader = getDataProvider()    # length = 100  batch_size = 20
    iter = dataloader.__iter__()
    for i in range(10):
        start = time.time()
        try:
            data, label = iter.next()
        except StopIteration:
            print('StopIteration')
            iter = data.__iter__()
            data, label = iter.next()
        end = time.time()
        print(i, '  takes time  ', end - start)
</code></pre>
<p>The cycle i=5 in which the StopIteration exception occurred took 27 seconds. All other cycles take no more than 1 second. Even reinitialising a dataloader takes less than 1 second.</p>
<p>I know that dataloader does some reset function after a StopIteration exception, but why is it more time consuming than reinitialising a dataloader? What is the correct way to reuse a dataloader?</p>

WinKawaks: <p>I changed the parameter ‘num_workers’ of dataloader. Then I found some interesting thing. <strong>_SingleProcessDataLoaderIter</strong> doesn’t hava this problem. <strong>_MultiProcessingDataLoaderIter</strong> takes a long time when the StopIteration exception occurs.</p>

ptrblck: <p>You could try to use <code>persistent_workers=True</code> while creating the <code>DataLoader</code> to avoid shutting down and recreating the processes.</p>

WinKawaks: <p>Thank you. It solves my problem.</p>
Min and Max step size in arange

akernel: <p>I am trying to test a function that I am writing that uses <code>torch.arange</code>, and am running into some floating-point issues. Namely, I am trying to figure out the (reasonable) minimum and maximum <code>step</code> size for which I can guarantee that <code>torch.arange</code> will work. I believe the minimum value is <code>torch.finfo().eps</code> - i.e. the machine epsilon, and the maximum value is <code>torch.finfo().max</code> (ignoring, for the moment, that the step size might be greater than the total length of the sequenc). Is this sound reasoning?</p>
How often is transformations done (internally) when loading a dataset?

Superklez: <p>Say for example I have</p>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.Resize(256),
    transforms.RandomCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(
        mean = [0.485, 0.456, 0.406],
        std = [0.229, 0.224, 0.225]
    )
])
</code></pre>
<p>and I use it to augment a dataset <code>CustomDataset(transform=transform)</code>. Is <code>transform</code> being run <em>for every batch</em>, <em>for every epoch</em>, or is it run only one? I’m curious to know because what if I were to use <a href="https://albumentations.ai/docs/api_reference/core/composition/" rel="noopener nofollow ugc">albumentations</a>’ <code>A.OneOf</code>, how often would it have to randomly select among provided transformations?</p>

ruotianluo: <p>transform is run each time you retrieve a sample from the Dataset object.<br>
So for batch size N, it will run N times every iteration (every sample has a different transformation). (not every batch, every epoch, but every sample)</p>
Changed quantized weights not reflecting in state_dict()

gakadam: <p>PyTorch: 1.7.0</p>
<p>I modified the quantized weights of a net post-quantization as follows:</p>
<pre><code class="lang-auto"># instantiate the quantized net (not shown here).

# get one of the conv layers
tmp = model_int8.state_dict()['features.0.weight']
scales = tmp.q_per_channel_scales()
zero_pts = tmp.q_per_channel_zero_points()
axis = tmp.q_per_channel_axis()

# get int repr
tmp_int8 = tmp.int_repr()

# change value (value change is dependent on the int8 value)
tmp_int8[0][0][0][0] = new_value

# how to convert tmp_int8 to torch.qint8 type?
new_tmp = torch._make_per_channel_quantized_tensor(tmp_int8, scales, zero_pts, axis)

# based on the above step:
model_int8.features[0].weight = new_tmp
</code></pre>
<p>However, <code>model_int8.features[0].weight</code> shows <strong>updated</strong> values, but <code>model_int8.state_dict()['features.0.weight']</code> shows <strong>old</strong> vales.</p>
<p>I also tried saving the modified model and reloading, but the problem persists.</p>
<p>Question is which weight values are being used for inference? I do not see change in the classification results.</p>

Vasiliy_Kuznetsov: <p>You need to use <code>conv_obj._weight_bias()</code> to get the weight, and <code>conv_obj.set_weight_bias(w, b)</code> to set the weight.  Here is an example: <a href="https://gist.github.com/vkuzo/8ad0e472e8743e142a1e72cb0f9efb1d" class="inline-onebox" rel="noopener nofollow ugc">gist:8ad0e472e8743e142a1e72cb0f9efb1d · GitHub</a></p>
How to save a complete model?

Glerk: <p>There are examples showing how to load the saved parameters of a model which is defined some place else.</p>
<p>How can I save an entire model in a single file and then load it later without a separate definition file? Is there a simple way to do this?</p>

ptrblck: <p>You could script/trace the model and export it. This wouldn’t need separate source files, but you would also be unable to manipulate the model easily.<br>
I would not recommend to save the complete eager model via <code>torch.save</code>, as it would still need to import the source files while reloading and can break easily.</p>

my3bikaht: <p>You can save model via torch.save but there are issues like pickling gpu-located models and if you decide to tinker with some functions or classes model is based on you’ll encounter that this is not possible.</p>
<p>Saving through model.state_dict() is a much more stable approach</p>
Different results when loading Triplet model

my3bikaht: <p>Having trouble to replicate model output after save&gt;load.<br>
It’s a triplet model with ResNet50 as an inner model.</p>
<p>Went both ways at the same time: saved state_dict of both models + optimizer vs saving model.cpu() itself.<br>
After calculating output tensor on single input right before saving model and after loading, I have different tensors before and after, yet same if I load state_dict vs model as a whole.</p>
<p>Visualization (TSNE) shows that model is completely lost any sense of understanding what’s going on<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f01fc5ac12f9e8f62de8169b46742f78ecf060b4.png" alt="epoch19" data-base62-sha1="ygeFc6Ei6XTAh6o90MLLV6zgxak" width="640" height="480"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ef71454001ca1bf05a7242148a4ae829235e316.png" alt="embeddings" data-base62-sha1="koJai4nnnYS2OIkm18Bj5SLJ02i" width="640" height="480"></p>
<pre><code class="lang-auto">modelFilename = "modelResNet.mdl"
model = torch.load(modelFilename)
model.cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9)
model.eval()
</code></pre>
<p>Am I missing something? Is it something related to zero grad?</p>

my3bikaht: <p>My bad. Added zero centering to normalizing function and forgot to enable it to data after model load.<br>
Topic closed.</p>