akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
If you save a dataparallel model do I have to again run nn.DataParallel after loading it back?

username: Shubhankar
cooked: <p>If you create a dataparallel model as shown <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models" rel="nofollow noopener">here</a> and then save it as shown <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models" rel="nofollow noopener">here</a>. Are you required to do <code>nn.DataParallel(model)</code> again to load the model and continue training?</p>

username: ptrblck
cooked: <p>No. If you store the <code>state_dict</code> from the model directly or <code>model.modules</code> in case you are using <code>nn.DataParallel</code>, you should be able to reload it into the model again.</p>

username: Shubhankar
cooked: <p>I think you will have to do <code>nn.DataParallel</code> since you will do <code>model.module.load_state_dict(checkpoint['model_state_dict'])</code> and for the model to have module you need <code>model = nn.DataParallel(model)</code></p>
Efficient Per-Example Gradient Computations

username: Quan_Vuong
cooked: <p>Assume we have a batch of data. Given each data point in the batch, I would like to get the norm of the gradient of the output of the network w.r.t the network parameters.</p>
<p>A naive solution is to perform multiple forward and backward passes (one pass for one data point), but this is pretty slow. Is there a better solution?</p>
<p>A better solution seems possible, but I don’t see any mention of it in Pytorch documentation.</p>
<p><a href="https://arxiv.org/abs/1510.01799" class="onebox" target="_blank" rel="nofollow noopener">https://arxiv.org/abs/1510.01799</a></p>

username: Irene_Li
cooked: <p>Hi,</p>
<blockquote>
<p>A naive solution is to perform multiple forward and backward passes (one pass for one data point), but this is pretty slow.</p>
</blockquote>
<p>Do you mean to feed the data points within a for-loop one by one? I don’t think this will be right because PyTorch is doing accumulation on the gradients.</p>
<p>Thanks.</p>

username: Yaroslav_Bulatov
cooked: <p>If you just have conv2d + linear layers, you could do this using single backward pass using something like this – <a href="https://github.com/cybertronai/autograd-hacks/blob/master/autograd_hacks.py#L167" rel="nofollow noopener">https://github.com/cybertronai/autograd-hacks/blob/master/autograd_hacks.py#L167</a></p>
<p>For norms squared, replace “grad1=einsum” line with  <code>torch.sum(B*B, dim=1)*torch.sum(A*A, dim=1)</code></p>

username: ezyang
cooked: <p>You can do per example gradients using torch.autograd.grad.  The key trick is to manually broadcast the weights yourself, and then differentiate with respect to the broadcasted weight. Here’s a very simple example on a model that is just a Linear:</p>
<pre><code class="lang-auto">import torch

torch.manual_seed(0)

B = 4
N = 2
M = 3

input = torch.randn(B, 1, N)
weights = torch.randn(N, M, requires_grad=True)

broadcasted_weights = weights.expand(B, N, M)
output = torch.bmm(input, broadcasted_weights)
loss = output.sum()

print(torch.autograd.grad(loss, broadcasted_weights))
</code></pre>
<p>I don’t know if you can conveniently do this with torch.nn modules, might be worth thinking about how to expose this in the API.</p>

username: ezyang
cooked: <p>Note this doesn’t work if you have operations in your network that implicitly broadcast, but don’t support taking broadcasted input. Convolution is an example of this.</p>

username: Yaroslav_Bulatov
cooked: <p>Simple API would be a helper function that takes an nn.Module and returns a version that implements the trick above. IE, model = batch_expand(model, N)</p>
<p>Autograd memory usage in new model has to be reasonable, otherwise there’s no benefit over original model and batch-size=1.</p>
<p>This seems to be the case in the example above (colab measurement <a href="https://colab.research.google.com/drive/1Lp1Y_ZGhi7FsPQajavpmqHWHICoRYctA#scrollTo=LsNjgyy7kVFj" rel="nofollow noopener">forum - replication memory - simplified</a>)</p>
Regarding Conv1d

username: renpenghit
cooked: <p>Hi I couldn’t grammatically quite understand the following example</p>
<blockquote>
<blockquote>
<blockquote>
<p>m = nn.Conv1d(16, 33, 3, stride=2)<br>
input = torch.randn(20, 16, 50)<br>
output = m(input)</p>
</blockquote>
</blockquote>
</blockquote>
<p>I think the type of nn.Conv1d is class, and accordingly m should be an instance of the class. Why in the third line, m(input) is used like a function?</p>
<p>Actually, it seems that &gt;&gt;&gt; output = m(input)  executes the function forward(self, input) defined in the class nn.Conv1d. So, to be grammatically correct,  I think &gt;&gt;&gt; output = m(input) should be changed into  &gt;&gt;&gt; output = m.forward(input) . But &gt;&gt;&gt; output = m(input)  indeed works.</p>
<p>Could anybody explain a bit about this? What grammatical rule does it follow?</p>
<p>The source code for Conv1d is at the following URL<br>
<a href="https://pytorch.org/docs/master/_modules/torch/nn/modules/conv.html#Conv1d" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/_modules/torch/nn/modules/conv.html#Conv1d</a></p>
<p>Thank you very much!</p>

username: driazati
cooked: <p><code>nn.Module</code> (which <code>nn.Conv1d</code> and all the <code>nn.</code> modules extend), implements the <code>__call__</code> magic method so that it can be used like a function as in your example. <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L530">The implementation can be found here</a>. Using <code>__call__</code> has some extra processing for forward/backward hooks, but, as you have found, its main job is to call <code>forward()</code></p>

username: renpenghit
cooked: <p>Thank you so much for your explanation! Truly helpful!</p>
Element 0 of tensors does not require grad and does not have a grad_fn costume loss

username: netaglazer
cooked: <p>costume loss:</p>
<pre><code class="lang-auto">def loss_function(p, y, b):
    losses = 0
    for i in range(b):
        k = len(y[i])
        loss1 = -(torch.ones(k).cuda()-y[i])@((torch.ones(k).cuda()-p[i]).log())
        l = y[i]*(p[i].log())
        prod = (torch.ones(k).cuda()-y[i]) - l
        loss2 = loss1 + torch.max(torch.ones(k).cuda()-prod)
        losses = loss1 + loss2
    return Variable(losses/(b))
</code></pre>
<p>dont know how to correct this error</p>

username: JuanFMontesinos
cooked: <p>Remove Variable(), it resets the tensor graph and makes it a leaf tensor.<br>
Variable is deprecated since pytorch 0.4. Soo unless you use pytorch 0.3 forget it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: netaglazer
cooked: <p>Still the same error…</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
Could you provide a reproducible snippet?<br>
Right now I think the issue is that neither p or y requires grad.</p>

username: netaglazer
cooked: <p>you want to see what the p and the y are?</p>

username: JuanFMontesinos
cooked: <p>Check before calling the function that p.requires_grad == True nor the same for y</p>
DistributedDataParallel with MPI and torch.distributed with MPI+multiprocessing

username: bapi
cooked: <p>DDP starts fine with gloo, however, with MPI it breaks throwing the following error:<br>
<code>RuntimeError: MPI process group does not support multi-GPU collectives</code><br>
This is when it is used as distributed without multiprocessing in a node. The error occurs at <code>python3.8/site-packages/torch/autograd/__init__.py", line 97, in backward</code>. However, with gloo it breaks after a couple of epochs.</p>
<p>With multiprocessing, on specifying the gpu-id in each process, again with gloo it starts fine and breaks after a couple of epochs, and with MPI it just does not do a dist.init_process_group; throws <code>Local abort before MPI_INIT completed completed successfully</code>. This is the issue that occurs even without torch.nn.parallel.DistributedDataParallel as the process group init has to happen before defining the model etc.</p>
<p>The only option that seems to be available is NCCL.<br>
Does it really not work with any other backend?</p>
<p>Thanks</p>
Gradients from hook triggered by backward and autograd are different

username: kaladin
cooked: <p>Hi, recently I noticed that when catching the gradients from a hook (through the grad_out parameter) that is triggered by a backward() call do not have requires_grad = True nor do they have a grad_fn attribute. However, when I trigger the same hook with autograd, then the values of grad_out caught from the hook are the same, yet they do have requires_grad = True and have a grad_fn attribute. I’m curious towards why this is happening? Thank you so much.</p>
<p>Here is my toy example for reference:</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(0)

# define embedding and linear layers
embedding_layer = torch.nn.Embedding(10, 5, padding_idx=0)
fc = torch.nn.Linear(5, 6)

random_ix = torch.randint(high=10, size=(5,))

embedding_list = []
def hook(module, grad_in, grad_out):
    print("grad out", grad_out[0]) # if triggered through autograd, this will have a grad_fn and requires_grad equal to True, otherwise not
    embedding_list.append(grad_out[0])

# register hook on embedding layer
embedding_layer.register_backward_hook(hook)

# do forward pass 
embeds = embedding_layer(random_ix)
output = fc(embeds)
print("output", output)
merged = torch.sum(output, dim=1)
summed = merged.sum()
print(summed)

# trigger hook through autograd
grad_auto = torch.autograd.grad(summed, embedding_layer.weight, create_graph=True)
print("grad auto", grad_auto[0][random_ix])

# trigger hook through backward() call 
summed.backward()
</code></pre>

username: vainaijr
cooked: <p>maybe if we use</p>
<pre><code class="lang-auto">summed.backward(create_graph=True)
</code></pre>
<p>then we will get same results</p>

username: albanD
cooked: <p>The requires_grad=True field is set if you are <code>create_graph=True</code>.<br>
In the grad call, if you just want to be able to backward again, you should use <code>retain_graph=True</code>.</p>

username: kaladin
cooked: <p>This worked. Thanks!</p>

username: kaladin
cooked: <p>That makes sense, thanks!</p>
Inconsistent gradient shapes for Conv1d and Conv2d

username: gbaasch
cooked: <p>Hello. This is my first post on the PyTorch forum so forgive me if there is not enough detail.</p>
<p>I am trying to use register_backward_hook to get the gradient from a 1d convolutional layer. I found that the gradient shape is not what I expected and it is inconsistent with Conv2d.</p>
<p>For Conv2d the shape of the gradient for input (1,1,28,28) is (1,1,28,28).<br>
I expect that the shape for Conv1d for input of shape (1,1,28) should be (1,1,28).<br>
Instead the shape of the gradient for Conv1d is (1, 2, 1, 28)</p>
<p>I have included code that shows a test case below.</p>
<pre><code class="lang-auto">def fun(module,grad_in,grad_out):
    print('grad_in', grad_in[0].shape)

net1d = nn.Sequential(nn.Conv1d(1,2,1,))
net2d = nn.Sequential(nn.Conv2d(1,2,1,))

x1d = torch.randn(1,1,28,requires_grad=True)
x2d = torch.randn(1,1,28,28,requires_grad=True)

net1d[0].register_backward_hook(fun)
net2d[0].register_backward_hook(fun)

print('Conv1d Gradient Shape')
l = net1d(x1d)
l.backward(torch.ones_like(l))
print('Conv2d Gradient Shape')
l = net2d(x2d)
l.backward(torch.ones_like(l))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Conv1d Gradient Shape
grad_in torch.Size([1, 2, 1, 28])
Conv2d Gradient Shape
grad_in torch.Size([1, 1, 28, 28])
</code></pre>
<p>Could someone please help me to understand the shape of the gradient for Conv1d? Am I misunderstanding something or does this output seem incorrect?</p>

username: albanD
cooked: <p>Hi,</p>
<p>As mentionned in the doc, the backward hooks on nn.Module are not working properly at the moment <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Hopefully we’ll manage to fix them soon.</p>
<p>You should ignore these results. If you want proper hooking, you should use hooks on Tensors directly as <code>l.register_hook(you_fn)</code> for example.</p>

username: gbaasch
cooked: <p>Hello,</p>
<p>Thanks for the quick response. Do you have an estimate on when the backward hooks should be fixed by?</p>

username: albanD
cooked: <p>It has been on the roadmap for a very long time. But it was blocked by required autograd improvements.<br>
I am currently making these improvements so the hook will be fixable afterwards.</p>
Module torch.onnx export create a graph containing an operator named "ATen"

username: Jeremy_Cochoy
cooked: <p>Hello folks,</p>
<p>I have been using the ONNX exporter recently, and I have a question for peoples who had some experience with the exporter and ONNX contributors.</p>
<p>In my model, one of my layers have:</p>
<pre><code class="lang-auto">std = x.std(dim=(2, 3))[:, :, None, None]
std[std == 0] = 1
x /= std
</code></pre>
<p>I know that their is some in place operations, but that shouldn’t cause any trouble in this specific case. (or maybe it does <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"> )<br>
What surprised me is the exported graph I obtain. It contain a non ONNX node named “ATen” (if you are familiar with pytorch.onnx, ATen are a category of operators. An operator is ATen if you can find his definition in torch/csrc/autograd/generated/VariableType.h)</p>
<p>Here is a picture of the graph:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/11123fa48a79f2fedb7032fe20875b870b5cf95b.png" data-download-href="https://discuss.pytorch.org/uploads/default/11123fa48a79f2fedb7032fe20875b870b5cf95b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11123fa48a79f2fedb7032fe20875b870b5cf95b_2_361x500.png" alt="image" data-base62-sha1="2r1dilZRGem9Cm6UhgHmkdGoOlZ" width="361" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11123fa48a79f2fedb7032fe20875b870b5cf95b_2_361x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11123fa48a79f2fedb7032fe20875b870b5cf95b_2_541x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/1/11123fa48a79f2fedb7032fe20875b870b5cf95b.png 2x" data-dominant-color="E8E9E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">712×986 37.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I can probably fix this by rewriting my code as follow:</p>
<pre><code class="lang-auto">std = std + (std == 0).double()  # Replace 0 values by 1
x /= std
</code></pre>
<p>But I am curious how a node that is not an <a href="https://github.com/onnx/onnx/blob/master/docs/Operators.md" rel="noopener nofollow ugc">ONNX operator</a> have appeared during the export. Seems to me like a little nasty bug. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
Export Torchscript module and call other than forward

username: PCerles
cooked: <p>Hi, I have an exported torch script module that looks loosely like:</p>
<pre><code class="lang-auto">class Model(torch.nn.Module):
    def forward(self, tensor):
        // do forward pass

    @torch.jit.export
    def reinitialize_hidden_buffers(self, tensor):
        // reinitializes model parameters
</code></pre>
<p>When I load the torchscript module in Python, I can call reinitialize_hidden_buffers, but when the model is exported and loaded into objective c (for deployment on iOS),  I get<br>
<code>no member named 'reinitialize_hidden_buffers' in 'torch::jit::script::Module</code><br>
Is there a way to call this correctly? Or should I just put control flow logic in the forward pass to do it.</p>

username: PCerles
cooked: <p>This ended up being really straightforward.</p>
<pre><code class="lang-auto">torch::autograd::AutoGradMode guard(false);
at::AutoNonVariableTypeMode non_var_type_mode(true);
_impl.run_method("reinitialize");
</code></pre>
How to create dataloader for multi-size images

username: Jiajun_Zha
cooked: <p>I wrote this toy dataset example for MPII dataset, but the dataset actually has different image size. So the Dataloader malfunctions when concatenate images together into one batch. I’m sure that my mode is able to handle different input size since I’m using deeplabv3_resnet. The problem is just how to concatenate different size of images into one batch.</p>
<pre><code class="lang-auto">class ToyDataset(data.Dataset):
    def __init__(self, root_dir ):
        self.datadir = root_dir
        self.poses = torch.load(os.path.join(self.datadir, 'poses.pt'))
        self.ids = torch.load(os.path.join(self.datadir, 'ids.pt'))
        return
    def __len__(self):
        return len(self.poses)
    def __getitem__(self, idx):
        img = torch.from_numpy(skio.imread(os.path.join(self.datadir, str(idx) + '.png'))).permute(2,0,1).float()/255
        f_img = torch.from_numpy(skio.imread(os.path.join(self.datadir, str(idx) + '_f.png'))).permute(2,0,1).float()/255
        pose = self.poses[idx]
        id = self.ids[idx]
        return {'img':img, 'f_img':f_img, 'pose':pose, 'id':id}
</code></pre>

username: ptrblck
cooked: <p>If you want to create a batch containing data with different shapes, you could use a custom <code>collate_fn</code> as described <a href="https://discuss.pytorch.org/t/torchvision-and-dataloader-different-images-shapes/41026/3">here</a>.</p>
<p>However, <code>deeplabv3_resnet101</code> is be a segmentation model, so your keypoint prediction might not work out of the box, but that’s just a side note and you might already have a plan how to use the model for your use case. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Jiajun_Zha
cooked: <p>Thank you! I’m a green-hand.</p>
Creating input for the model from the raw text

username: al314
cooked: <p>Hi all,</p>
<p>In the official documentation tutorial on text classification</p>
<p><a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html</a></p>
<p>the next data set is used</p>
<pre><code class="lang-auto">train_dataset, test_dataset = 
text_classification.DATASETS['AG_NEWS'](root='/.pytorch_datasets', ngrams=2, vocab=None)
</code></pre>
<p>which contains elements which are already numerical tensors and perfectly suitable for feeding into neural net</p>
<pre><code class="lang-auto">train_dataset[0]

(2,
 tensor([572, 564, 2, 2326,   ...]))

</code></pre>
<p><strong>How do I prepare the same input from the raw text?</strong></p>
<p>So far I have managed to convert only single words into vectors (thanks for the help <a href="https://discuss.pytorch.org/t/is-it-a-correct-way-to-build-vocab-torchtext/68094/3">here</a> )</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torchtext.data import Dataset, Example, Field
from torchtext.data import Iterator, BucketIterator
# https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html

TEXT  = Field(sequential=True, tokenize=lambda x: x.split(), lower=True, use_vocab=True)
LABEL = Field(sequential=False, use_vocab=False)

data = [("shop street mountain is hight", "a"), 
         ("work is interesting", "b")]

FIELDS = [('text', TEXT), ('category', LABEL)]

examples = list(map(lambda x: Example.fromlist(list(x), fields=FIELDS), data))

dt = Dataset(examples, fields=FIELDS)

TEXT.build_vocab(dt, vectors="glove.6B.100d")
LABEL.build_vocab(dt, vectors="glove.6B.100d")

print(TEXT.vocab.stoi["is"])

data_iter = Iterator(dt, batch_size=4, sort_key=lambda x: len(x))

VOCAB_SIZE = len(TEXT.vocab)
embedding = nn.Embedding(VOCAB_SIZE, 32)

print(embedding(torch.tensor(TEXT.vocab.stoi["is"])))
</code></pre>
<p>But this is only for single words while in the raw text I have several sentences per label.</p>
<p><strong>Can you please advise me on how to convert variable-length pieces of text into same dimensional tensors like we have in built-in data set <code>text_classification.DATASETS['AG_NEWS']</code>?</strong></p>

username: sai_m
cooked: <p>Hi Alex,<br>
Please follow this tutorial : <a href="https://mlexplained.com/2018/02/08/a-comprehensive-tutorial-to-torchtext/" rel="nofollow noopener">https://mlexplained.com/2018/02/08/a-comprehensive-tutorial-to-torchtext/</a></p>

username: zhangguanheng66
cooked: <p>Or, you could load your data with a new torchtext abstraction. Text classification datasets, mentioned by you, follow the same new abstraction. It should be very straightforward to copy/paste and write your own pipeline <a href="https://github.com/pytorch/text/blob/master/torchtext/datasets/text_classification.py" rel="nofollow noopener">link</a>.</p>
How to plot real time loss curves

username: Jordan_Howell
cooked: <p>Hello,</p>
<p>I’m trying to plot real time loss curves as my model runs.  The model runs but does not print out the loss.  Could someone take a gander at the code below and see what mistake I’m making?</p>
<pre><code class="lang-auto">def Average(lst): 
    return sum(lst) / len(lst) 

epochs = 50
aggregated_losses = []

max_trn_batch =  4

for i in range(epochs):
    epoch_loss = []
    for b, (image, label, policy, categorical_data) in enumerate(train_loader): #, numerical_data
        image = image.cuda()
        label = label.cuda()
        #numerical_data = numerical_data.cuda()
        categorical_data = categorical_data.cuda()
        
        #count batches
        b += 1
        
        #throttle teh batches
        if b == max_trn_batch:
            break
        
        y_pred = combined_model(image, categorical_data) #, numerical_data
        single_loss = criterion(y_pred, label)
        
        # statistics
        print(f'epoch: {i:3}, batch: {b:3}, loss: {single_loss.item():10.8f}')

        optimizer.zero_grad()
        single_loss.backward()
        optimizer.step()
        epoch_loss.append(single_loss.cpu().data.numpy())
    
    print(Average(epoch_loss))
    aggregated_losses.append(Average(epoch_loss))
    loss_history = pd.DataFrame(aggregated_losses).reset_index()
    loss_history.rename(columns = {'index':'epoch', 0:'average_epoch_loss'},inplace = True)
    x = sns.lineplot(x = 'epoch', y = 'average_epoch_loss', data = loss_history)
    
    
    scheduler.step(Average(epoch_loss))
</code></pre>

username: Jordan_Howell
cooked: <p>Disregard. I found it here:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/stared/livelossplot/blob/master/examples/pytorch.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/stared/livelossplot/blob/master/examples/pytorch.ipynb" target="_blank" rel="nofollow noopener">stared/livelossplot/blob/master/examples/pytorch.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch import optim\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "from livelossplot import PlotLosses"
   ]
  },
  {
</code></pre>

  This file has been truncated. <a href="https://github.com/stared/livelossplot/blob/master/examples/pytorch.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Time per epoch increase when running experiments on multiple GPUs concurrently

username: oli4100
cooked: <p>I am using multiple GPUs to train several models individually (e.g. hyperparameter grid search on a model). I am using Pytorch’s DataLoader with num_workers=4 on a 4GB dataset that is completely read to RAM, and every batch is sent to GPU using [to(device)] in a normal way (i.e. within loop after reading the batch). I’ve noticed the following happening:</p>
<ul>
<li>Time per epoch increases by ~30-40% when concurrently running two experiments on two separate GPUs instead of running only a single experiment on a single GPU. Similarly, when going from two to three GPUs time per epoch increases again by some 30% (over all the GPUs). I’m certain that the models are executed on the GPU, because there is normal and expected CUDA activity showing on each GPU (consistently between 50-90% depending on model variant).</li>
<li>CPU usage when using a single GPU is ~60-70% (6-core/12 thread CPU), when firing up the second GPU it never gets below 100%.</li>
</ul>
<p>My conclusion is that it seems as though my CPU acts as a bottleneck when training on multiple GPUs. Before splashing money on a better CPU, I’d like to ascertain myself that the CPU is indeed the bottleneck and there are no further steps I can take to alleviate the issue.</p>
<p>What I have yet tried:</p>
<ul>
<li>Using both less and more num_workers in the dataloader, but this has limited effect other than slowing down training even further when reducing the num_workers below 4.</li>
<li>Setting os.environ[‘omp_num_threads’] to 1 or 2 to ‘force’ using less threads. (see: <a href="https://discuss.pytorch.org/t/threading-issue-in-pytorch/11229">here</a>, or <a href="https://discuss.pytorch.org/t/cpu-usage-extremely-high/52172">here</a> ) Does not work.</li>
<li>Preloading entire dataset to GPU memory: does not work as dataset is too large to work with all variants of my models.</li>
<li>Setting pin_memory=False or True in DataLoader, does not really matter.</li>
</ul>
<p>My questions:</p>
<ul>
<li>Is this normal behavior? What could be the reason of such performance drop?</li>
<li>What kind of steps could I further take to try to fix this or to ascertain myself that indeed the CPU is acting as the bottleneck?</li>
<li>Maybe this is related to <a href="https://github.com/pytorch/pytorch/issues/25010" rel="nofollow noopener">this</a> bug, although that seems solved.</li>
</ul>
<p>Setup:</p>
<ul>
<li>Windows 10 Pro</li>
<li>Python 3.7</li>
<li>Pytorch 1.3.1 installed via conda</li>
<li>Cuda Toolkit 10.1</li>
<li>CuDNN 7.0</li>
<li>64 GB RAM</li>
<li>2x 1080Ti, 1x 2080Ti</li>
<li>I am using Spyder 3.3.6 with iPython using three separate kernels to run experiments on each GPU. Hence, I’m assuming the processes should run independently.</li>
</ul>
How to move multiple joints with PyTorch

username: DLBOXER
cooked: <p>Hey Guys.</p>
<p>I control the arm on the simulator by reinforcement learning.<br>
The arm has three joints, and I want to move the each joints -1 °, 0 °, or 1 ° each step.<br>
What kind of neural network can?</p>
<pre><code class="lang-auto">class Net(nn.Module):

    def __init__(self, n_in, n_mid, n_out):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(n_in, n_mid) 
        self.fc2 = nn.Linear(n_mid, n_mid)
        self.fc3 = nn.Linear(n_mid, n_out)

    def forward(self, x):
        h1 = F.relu(self.fc1(x))
        h2 = F.relu(self.fc2(h1))
        output = self.fc3(h2)
        return output
</code></pre>
Why is MKLDNN slower than normal cpu mode

username: Shisho_Sama
cooked: <p>Hello Everyone, Hope you are having a great day.<br>
For the past several weeks, I have been trying to get our models to run faster either using the MKLDNN or the Quantization. However, both of them seems to be failing for some reasons.<br>
Today, I found out MKLDNN also doesnt work! it seems not to have any effect on runtime performance. Its even some times, slower than normal cpu mode!<br>
The issue exists both on 1.3.1 and 1.4.0.<br>
You may try the following snippet and see how it performs for you.<br>
Running this on my local machine and also Google Colab, resulted the same and they were both slow as snail , in some cases slower than the normal (default) cpu mode!</p>
<p>Run this on a supporting platform such as Linux (e.g. ubuntu 18.04)</p>
<pre><code class="lang-python">import torch
print(f'Pytorch version : {torch.__version__}')
print(*torch.__config__.show().split("\n"), sep="\n")

from torchvision import models
from torch.utils import mkldnn as mkldnn_utils
import time 

def forward(net, use_mkldnn=False, iteration=1, batch_size=10):
  net.eval()
  batch = torch.rand(batch_size, 3,224,224)
  if use_mkldnn:
    net = mkldnn_utils.to_mkldnn(net)
    batch = batch.to_mkldnn()

  start_time = time.time()
  for i in range(iteration):
      net(batch)
  return time.time() - start_time

net = models.resnet18(False)
iter_cnt = 100
batch_size = 1

no_mkldnn   = forward(net, False, iter_cnt, batch_size)
with_mkldnn = forward(net, True,  iter_cnt, batch_size)

print(f"time-normal: {no_mkldnn:.4f}s")
print(f"time-mkldnn: {with_mkldnn:.4f}s")
print(f"mkldnn is {with_mkldnn/no_mkldnn:.2f}x slower!")

</code></pre>
<p>Any help is greatly appreciated.</p>

username: ptrblck
cooked: <p>It might depend on your architecture, but I see a speedup:</p>
<pre><code class="lang-python">time-normal: 2.7962s
time-mkldnn: 1.5869s
mkldnn is 0.57x slower!
</code></pre>
<p>for an <code>Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz</code>.</p>

username: Shisho_Sama
cooked: <p>Compared to results from <a href="https://discuss.pytorch.org/t/use-mkldnn-in-pytorch/54943/15">here</a> : its really nothing ! I ran this some time before and got around 600x faster inference on mkldnn version!<br>
I remember running it couple of times as well!</p>
Unknown error using torch.symeig()

username: gebrahimi
cooked: <p>I see this error, when I run the following code. I can not see even one example of the error on internet.</p>
<pre><code>eigval1, eigvec1 = torch.symeig(tensor1, eigenvectors = True)

RuntimeError: symeig_cuda: Argument 113 has illegal value
</code></pre>
<p>Any idea what it says?</p>

username: ptrblck
cooked: <p>Could you post an example tensor to reproduce this issue, please?</p>
Basic functions in Pytorch

username: aks407
cooked: <p>I have been able to do so much so far, would really appreciate helps from others to do remaining tasks.<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>from torch.utils.data import Dataset</p>
<p>import torch<br>
import torchvision<br>
import pandas as pd<br>
from <a href="http://skimage.io" rel="nofollow noopener">skimage.io</a> import imread<br>
from skimage.color import gray2rgb</p>
<p>class ChallengeDataset(Dataset):<br>
def <strong>init</strong>(self, dataset,csv_path,train_split=0.8 ):<br>
self.data=pd.read_csv(csv_path)</p>
<p>def <em>getitem</em> (self,index):<br>
pixel_sequence = self.data[‘pixels’][index]<br>
face = [int(pixel) for pixel in pixel_sequence.split(’ ')]<br>
label = self.labels[index]</p>
<pre><code>    return face, label</code></pre>

username: ptrblck
cooked: <p>Is this some kind of assignment you should be working on? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
If that’s the case, I would recommend to use available resources (the docs, this forum etc.) to solve the remaining tasks.</p>
Where is the source code for conv_transpose2d

username: Rahul_Deshmukh
cooked: <p>In torch/nn/Module/conv.py we have a class definition for ConvTranspose2d() which calls a function F.conv_transpose2d. Where F is imported from the parent directory from the file functional.py</p>
<p>However, I could not find any definition of conv_transpose2d any where inside ./torch/</p>
<p>I want to look at the source code for conv_transpose2d. Can anyone point me in the right direction</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>What are you looking for in particular?<br>
The main entry point is <a href="https://github.com/pytorch/pytorch/blob/94ddc2c462bd7c79426057d67b03d4a397db0dd1/aten/src/ATen/native/Convolution.cpp#L524">here</a> but then depending on the device you run it on, if you have cudnn or not, mkldnn or not etc, it goes to different places.</p>
FLOPS of static quantized CNN by pthflops

username: Majid_Shirazi
cooked: <p>Hi everone,</p>
<p>I am trying to calculated the FLOPS of my Static Quantized model by pthflops library of Pytorch but it fails, do U have any idea how to calculate it ? (it works for normal model)</p>
<pre><code class="lang-auto">from pthflops import count_ops
rand_image = torch.rand(1, 3, 96, 96)
count_ops(quantized_net, rand_image)

error :

StopIteration                             Traceback (most recent call last)
&lt;ipython-input-43-9cbbf57752cb&gt; in &lt;module&gt;
----&gt; 1 count_ops(q_net, rand_image)

~/anaconda3/envs/py36/lib/python3.6/site-packages/pthflops/ops.py in count_ops(model, input, custom_ops, ignore_layers, print_readable, verbose, *args)
    203     """
    204     # Make sure that the input is on the same device as the model
--&gt; 205     if next(model.parameters()).device != input.device:
    206         input.to(next(model.parameters()).device)
    207 

StopIteration: 
</code></pre>
What's the most efficient way of computing AA'?

username: deyan
cooked: <p>If I’m only interested in the upper triangle of <code>B</code> (<code>B</code> will be symmetric), what’s the most efficient way of computing it in PyTorch?</p>
<p><code>B = torch.bmm(A, torch.transpose(A))</code></p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think we have anything to do this.<br>
In particular, we have no way to represent a triangular matrix without having the full matrix.<br>
And we have almost no function that takes triangular matrices as input either.</p>
How is GPU memory allocated and released? Procedure to release GPU memory?

username: johann-petrak
cooked: <p>I would really want to understand this properly. There are many many threads about GPU memory problems, but it is hard to get a proper understanding of the details behind how Pytorch manages the memory one the GPU.</p>
<p>I have a problem where I run out of memory when doing crossvalidation, where for each fold, I load and train a new model and then evaluate the model before the same Python variable is used to hold a new model, train that etc.</p>
<p>Initially I thought that if a Python variable references a tensor on the GPU and that Python variable then gets re-assigned, that the memory on the GPU would get released because no variable is pointing to it any more. But this does not seem to be the case: the GPU memory needed by the model seems to accumulate over each fold even though the python variable that holds the model gets re-assigned.</p>
<p>However, if I move the model to the cpu and do <code>torch.cuda.empty_cache()</code> before reassigning the variable, the memory does not seem to accumulate or at least not as much.</p>
<p>Is this intended behaviour? Should GPU memory deallocation happen automatically or is a specific procedure needed? What is going on?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The idea is that we have a caching allocator.<br>
So when you need to put a Tensor on the GPU, we ask the caching allocator. If it already have enough space, it just returns it. Otherwise, it asks the GPU driver for new memory.<br>
When the Tensor is destroyed, the memory is kept around by the allocator.<br>
The memory is only released to the GPU driver if you are about to OOM or if the user cal <code>empty_cache()</code>.</p>
<p>Does that answer your question?</p>

username: johann-petrak
cooked: <p>So when a python variable that refers to a tensor on the GPU directly or indirectly (e.g. by referring to a module that in turn refers to one or more tensors on the GPU) gets reassigned, the GPU memory of all tensors that have been referenced directly or indirectly should be available to get reused?</p>
<p>My problem is that I expected this to happen but it does not appear to happen when I just reassign a variable that points to a torch module on the GPU. Instead I have to first move the module back to the cpu (and manually do `empty_cache() but not sure if this is really required).</p>

username: albanD
cooked: <blockquote>
<p>the GPU memory of all tensors that have been referenced directly or indirectly should be available to get reused?</p>
</blockquote>
<p>Yes exactly.</p>
<blockquote>
<p>My problem is that I expected this to happen but it does not appear to happen</p>
</blockquote>
<p>Could you do a small code sample (30-40 lines) that reproduces this please?</p>
How to set the gradient to some value and do back-propagation?

username: BarCodeReader
cooked: <p>so here I have a very simple model</p>
<pre><code class="lang-auto">def Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(...)
        self.conv2 = nn.Conv2d(...)
        ...
        self.output =nn.Conv2d(...)
        #some more code for relu and BN

    def forward(self, x):
        out = self.conv1(x)
        #some more code for relu and BN
        out = self.conv2(out)
        #some more code for relu and BN
        ...
        out  = self.output(out)

        return out
</code></pre>
<p>I know that I can pass in some random data like:</p>
<pre><code class="lang-auto">model = Net()

fake_input = torch.randn((1,1,32,32),requires_grad=True)
output = model(fake_input)
</code></pre>
<p>and if i want to see the gradient of the input i can use:</p>
<pre><code class="lang-auto">shape = torch.ones_like(output)
output.backward(shape)

#and the gradient for input data will be 
print(fake_input.grad.data)
</code></pre>
<p>so here I want to know, is it possible that we can fix the gradient of ouput layer to be some matrix M, and we back-propagate from that layer, then we study the gradient response of the input?</p>
<p>something like this:</p>
<pre><code class="lang-auto">#change the gradient of output layer to matrix_M
net.output.weight.grad = matrix_M

#do back-propagation, here the network gradient should already be changed to matrix_M
out.backward(shape)

#and study the gradient response of input
print(fake_input.grad.data)
</code></pre>
<p>I am not sure if I did this in correct way…can anyone give me some hints?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Setting the <code>.grad</code> field won’t have any effect.</p>
<p>If you want to backward by setting the gradient for the output’s gradient to a given value <code>M</code>, you can simply do: <code>out.backward(M)</code></p>
<p>I don’t know what you expect to be doing in your example. The gradient of <code>net.output.weight</code> do not influence the gradients in the rest of the network (contrary to the gradients of out).</p>

username: BarCodeReader
cooked: <p>Ok Thanks!</p>
<p>actually what i am trying to do is to see the effective receptive field…as mentioned by <a href="https://arxiv.org/pdf/1701.04128.pdf" rel="nofollow noopener">this paper</a></p>
<p>please correct me if my approach is not right. thanks again</p>

username: albanD
cooked: <p>I’m not familiar with that paper, but it does look like you want to change the activation (out) and not the weights.</p>
How to compute the Jacobian matrix of the network's parameters?

username: S_M
cooked: <p>I am required to compute my loss function as below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df9130bb0fce062bcf42e1a03d0e5daf1ef05395.png" data-download-href="https://discuss.pytorch.org/uploads/default/df9130bb0fce062bcf42e1a03d0e5daf1ef05395" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df9130bb0fce062bcf42e1a03d0e5daf1ef05395_2_690x111.png" alt="image" data-base62-sha1="vTLxMfoIr7HblZOpiMDSe0DsddH" width="690" height="111" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df9130bb0fce062bcf42e1a03d0e5daf1ef05395_2_690x111.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df9130bb0fce062bcf42e1a03d0e5daf1ef05395_2_1035x166.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/f/df9130bb0fce062bcf42e1a03d0e5daf1ef05395_2_1380x222.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1395×225 18.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>where “Theta” is the parameters of the network (i.e. weights) and “f(Theta)” is the output network and “y” is a real label and “x” is the input sample.</p>
<p>would you please give me a tip to do that, by sample code?<br>
how can I compute this in Pytorch during the train of my network?<br>
Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can find in <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7">this</a> gist a function that shows how to compute the hessian.</p>
<p>But you can compute the dot product with f-y much more efficiently using Rop from <a href="https://gist.github.com/apaszke/c7257ac04cb8debb82221764f6d117ad">this</a> gist. In particular it will take <code>Rop(loss, theta, f(theta) - y)</code>.</p>

username: S_M
cooked: <p>Hi albanD,<br>
How about the first gist, Do you mean to use the following function?<br>
where “y” is “loss” and “x” is “theta” in my example. Is it right?</p>
<p>import torch</p>
<p>def jacobian(y, x, create_graph=False):<br>
jac = []<br>
flat_y = y.reshape(-1)<br>
grad_y = torch.zeros_like(flat_y)<br>
for i in range(len(flat_y)):<br>
grad_y[i] = 1.<br>
grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)<br>
jac.append(grad_x.reshape(x.shape))<br>
grad_y[i] = 0.<br>
return torch.stack(jac).reshape(y.shape + x.shape)</p>
<p>Thank you.</p>

username: albanD
cooked: <p>Yes you want the gradient of the loss wrt theta.</p>
Help with pytorch starter code

username: chandrak1907
cooked: <p>I am new to pytorch and trying to learn from online examples. I tried to create RNN for classifying imdb reviews. My code is in gist here: <a href="https://gist.github.com/Chandrak1907/6b91dc1c19d613f6d1a2b22cf30c0ebf" rel="nofollow noopener">https://gist.github.com/Chandrak1907/6b91dc1c19d613f6d1a2b22cf30c0ebf</a></p>
<p>I see that RNN accuracy is around 50%  and parameters are not changing much during training.<br>
Can you pls help me in identifying bug in my code?</p>
<p>Thank you,</p>

username: shahid
cooked: <p>You’re giving a very high learning rate(50). This would cause divergence in your loss surface and that’s why your parameters are messing up. Change the lr to something very small maybe 1e-2 or 1e-3 and try training again</p>

username: chandrak1907
cooked: <p>hi <a class="mention" href="/u/shahid">@shahid</a>, Thank you for reply. I tried various learning rates but no change in final accuracy.<br>
Updated notebook is here - <a href="https://gist.github.com/Chandrak1907/24810970561db246458056d265d504be" rel="nofollow noopener">https://gist.github.com/Chandrak1907/24810970561db246458056d265d504be</a></p>
16-bit integer values to float

username: hash-ir
cooked: <p>I have a dataset consisting of gray-scale images of 16-bit pixel values. PyTorch requires the input to be a float before feeding into the model. I want to know which floating-point precision should I use to not lose any information from the images. So far, I have been using <code>inputs.float()</code> which casts the inputs to <code>torch.float32</code>. Further, does it have any effect on the training time using a higher precision float like <code>torch.double()</code>.</p>

username: colesbury
cooked: <p>You won’t lose any precision converting to float. The 32-bit floating point type can precisely represent all integers all 16-bit integers (actually up to all 24-bit integers too).</p>
<p>You should probably use float instead of double. Most PyTorch operations are faster on float32 and on GPUs float operations tend to be <em>way</em> faster than double.</p>
<p>You probably also want to normalize your inputs. If you’re using a pre-trained model, it’s important that you same normalization as was used to train the model. If you’re training from scratch, normalizing to [-1.0, 1.0] is reasonable or zero-mean unit standard deviation.</p>
Convert array to tensor

username: Sam3
cooked: <p>Hi,</p>
<p>how can i convert array to tensor in pytorch?</p>

username: ekilic
cooked: <p>see <a href="https://discuss.pytorch.org/t/how-to-convert-array-to-tensor/28809">28809</a></p>

username: sai_m
cooked: <p>using : torch.from_numpy(numpy_array), you can convert a numpy array into tensor.<br>
if you are using a list, use torch,Tensor(my_list)</p>
Handling missing data when working with sequences with RNNs: pack and padding

username: ekingedik
cooked: <p>Hello,</p>
<p>I work with time-series sequence data. In a hypothetical way, I can frame my problem as follows:</p>
<ul>
<li>List item I will have N temporally aligned sequences in each forward pass.</li>
<li>List item Each of these sequences will be fed into an LSTM and the last outputs of the LSTMs will be concatenated to form a tensor of size batch_size*(N*hidden_dimension)</li>
<li>List item The resulting tensor will be fed into a linear layer for prediction.</li>
</ul>
<p>The problem I have is, randomly, some of these N sequences might be missing, resulting in the last tensor having a different dimension (for example, in case of one sequence missing, it will be batch_size*((N-1)*hidden_dimension)).</p>
<p>My first attempt at fixing this was to use pad_sequences and pack_padded_sequence. Basically, when there is a missing sequence, I created an empty sensor and padded it to match the rest of the sequence lengths, guaranteeing N sequences in each pass, albeit some being empty. This way, I expected empty sequences to do not contribute to the loss. However, pack_padded_sequence results in the following error when a 0 length is passed to it:<br>
“Length of all samples has to be greater than 0, but found an element in ‘lengths’ that is &lt;= 0”</p>
<p>Is there a way to modify pack_padded_sequences so it will work with zero lengths? Other possibilities are as follows:</p>
<ul>
<li>List item Masking hidden units when there is no data: Is there a way to mask parts of the Linear Layer at the end, where the missing sequence will normally fill, as similar to dropout?</li>
<li>List item Padding the empty streams with a value that would not affect the training procedure: I’m not sure if filling the empty streams with zero will work since my valid sequences already have zeros, so it is a valid value. I’m also not sure if I fill these sequences with a value that does not occur in my dataset will guarantee that it won’t affect the training.</li>
</ul>
<p>What will be the best way to handle this? Thanks in advance!</p>
Speeding up dataloading with DALI tutorial

username: yaysummeriscoming
cooked: <p>Hi everyone, I’d like to share a tutorial I wrote recently about using Nvidia DALI to speed up the Pytorch dataloader.  It contains a few tips I found for getting the most out of DALI, which allow for a completely CPU pipeline &amp; ~50% larger max batch sizes than the reference examples.</p>
<p>DALI gives really impressive results, on small models its ~4X faster than the Pytorch dataloader, whilst the completely CPU pipeline is ~2X faster.  This means nearly 4000 images/s on a Tesla V100 &amp; single GPU ImageNet training in only a few hours!</p>
<p>Article is <a href="https://towardsdatascience.com/nvidia-dali-speeding-up-pytorch-876c80182440" rel="nofollow noopener">here</a> and codebase is <a href="https://github.com/yaysummeriscoming/DALI_pytorch_demo" rel="nofollow noopener">here</a>.  Hope you find it useful!</p>