akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LSTM training problems

wpron: <p>I have problems training a simple nn.LSTM with</p>
<pre><code>rnn = nn.LSTM(1, args.n_hidden, 1)
net = Mynet()
crit = nn.MSELoss()
optim_rnn = optim.Adam(rnn.parameters(), lr=1e-3)
optim_net = optim.Adam(net.parameters(), lr=1e-3)
</code></pre>
<p>One epoch of my training procedure is</p>
<pre><code>c = torch.randn(1, args.n_batch, args.n_hidden)   
h = torch.randn(1, args.n_batch, args.n_hidden)
for k in range(N):
  rnn.zero_grad()
  net.zero_grad()
  i = torch.tensor(y_train[k]).detach().view(1, 1, 1)
  o, (h, c) = rnn(i, (h.detach(), c.detach()))
  p = net(o.view(1, -1))
  err = crit(p.view(-1), torch.tensor(f_train[k]))
  err.backward()
  optim_rnn.step()
  optim_net.step()
</code></pre>
<p>What am I doing wrong?<br>
Is there a problem with how I use the hidden and cell states h and c?</p>
Runtime error, seems like graph is being destroyed

IzzyPutterman: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/887dde9b4388d34d86b22457bf03bb0312927e12.png" data-download-href="https://discuss.pytorch.org/uploads/default/887dde9b4388d34d86b22457bf03bb0312927e12" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/887dde9b4388d34d86b22457bf03bb0312927e12_2_690x270.png" alt="image" data-base62-sha1="jtsBPQCU8uaJJX1CfAOqA4Fk3DA" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/887dde9b4388d34d86b22457bf03bb0312927e12_2_690x270.png, https://discuss.pytorch.org/uploads/default/optimized/2X/8/887dde9b4388d34d86b22457bf03bb0312927e12_2_1035x405.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/8/887dde9b4388d34d86b22457bf03bb0312927e12_2_1380x540.png 2x" data-dominant-color="EBECEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2031×797 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
When I run this I keep getting the error <em><strong>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</strong></em> when computing dfdxdth2 towards the bottom.  If I swap the order of dfdxdth and dfdxdth2, then the error appears on dfdxdth.  So I am assuming somehow my graph is getting screwed up during the backward pass of which ever (dfdxdth or dfdxdth2) comes first.  Also in the last loop in the screenshot, if I were to remove either dfdxdth or dfdxdth2, the loop would fail on the second pass.  This makes me think no matter what the first autograd computed in the loop causes the error.<br>
Sorry about my bad coding etiquette as well</p>
Loading state_dict on Raspberry Pi

Leo-Lautsprecher: <p>Hello,<br>
I’m trying to load a State Dict that I saved on a Windows PC with a Raspberry Pi. I’m getting an error message that delivers almost no results on google:<br>
<code>ValueError: whence value 1460 unsupported</code> (full traceback below)</p>
<p>Here some info about the two systems:<br>
-Windows machine:<br>
-Windows 7, 64bit<br>
-torch._ <em>version</em> _ prints “0.4.1”<br>
-only training on cpu<br>
-num_workers=0 when creating the DataLoader Class.</p>
<p>-Raspberry Pi:<br>
-Raspberry Pi 3 Model b+<br>
-torch._ <em>version</em> _ prints “1.2.0a0+7c40576”<br>
-The file containing the network structure is the same<br>
-I’m loading the state_dict with “self.net.load_state_dict(torch.load(path))”<br>
-When I’m not trying to load the state_dict, everything works out fine (basicly only the forward pass of the model), but I obviously get the wrong values.</p>
<p>If needed, I will try to reproduce the issue with the minimal amount of code, but I thought I give it a try here first. Maybe I can solve the issue with additional parameters (like telling the model that it now runs on another processor)?</p>
<p>Thanks in advance an best regards,<br>
Leo</p>
<p>Full traceback:</p>
<pre><code class="lang-auto">Exception in Tkinter callback
Traceback (most recent call last):
  File "/usr/lib/python3.5/tkinter/__init__.py", line 1562, in __call__
    return self.func(*args)
  File "/home/pi/Desktop/Daten_Aufnehmen/Gui.py", line 547, in clickBtnOk
    self.network.createNets()
  File "/home/pi/Desktop/Daten_Aufnehmen/Neural_network.py", line 161, in createNets
    self.cam_nets.append(NeuralNetwork(file,netDef, cam_sen="camera", n_settings=len(self.settings)))
  File "/home/pi/Desktop/Daten_Aufnehmen/Neural_network.py", line 177, in __init__
    self.net.load_state_dict(torch.load(path))
  File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 386, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.5/dist-packages/torch/serialization.py", line 580, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
ValueError: whence value 1460 unsupported
</code></pre>

Leo-Lautsprecher: <p>Here the update with minimum code:<br>
This works on my Windows PC but not on the Raspberry Pi.</p>
<pre><code class="lang-auto">from Networks import ImageNN_1
import torch

path="State_Dict_Camera1_ImageNN_1.pt"
net=ImageNN_1(3)
state_dict=torch.load(path)
print ("Worked")
</code></pre>

alex.veuthey: <p>I think the first problem is the version mismatch between your two machines (0.4.1 and 1.2.0a). Could you try installing 0.4.1 on the raspberry and checking again?</p>

Leo-Lautsprecher: <p>Thank you, downgrading did the Trick.</p>
Modify Inception_v3 to get multiple outputs

AmardeepGanguly: <p>This is  my code</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, num_classes1, num_classes2):
        super(MyModel, self).__init__()
        self.model_inception = models.inception_v3(pretrained=True)
        num_ftrs = self.model_inception.AuxLogits.fc.in_features
        self.model_inception.fc = nn.Sequential()
        self.fc1 = nn.Linear(num_ftrs, num_classes1)
        self.fc2 = nn.Linear(num_ftrs, num_classes2)

    def forward(self, x):
        x = self.model_inception(x)
        out1 = self.fc1(x)
        out2 = self.fc2(x)
        return out1, out2

</code></pre>
<p>But i am getting the following error<br>
AttributeError: ‘tuple’ object has no attribute ‘dim’</p>

rpalsaxena: <p>Can you share the line of code on which you are getting that error? (or the complete error message)</p>

ptrblck: <p>The <code>inception_v3</code> model returns two outputs in the default setting: the output from the last layer and an auxiliary output.<br>
If you don’t want to use the <code>aux_output</code>, just pass the last output to <code>self.fc1</code>:</p>
<pre><code class="lang-python">x = self.model_inception(x)
out1 = self.fc1(x[0])
out2 = self.fc2(x[0])
</code></pre>

AmardeepGanguly: <p>Thanks peter again!!</p>
How to extract different one from a picture with many repetitive symbol

xh_c: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/573638e18ff98ca1b46bd40e87873b61cc9eb14c.png" alt="image" data-base62-sha1="crvI2Vz0mWIPJqcPv7HQBNm3fvC" width="359" height="117"><br>
I can’t just use image tool to seperate symbol becourse there is some cutted symbol on the edge of image(like the right one). So I wonder is there a deep learning way to solve it.</p>
<p>ps: the wrong symbol is 5% of total in a image</p>

mattdeitke: <p>Not entirely sure what you’re trying to do, but if you’re trying to break the image up like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/4/49b737baf9a150afb949168a554688b1af124608.jpeg" alt="break-up" data-base62-sha1="aw7oDMUWCxCRWYlvyCI5x7wnUoM" width="359" height="117"><br>
I would use a k-means clustering algorithm to segment the data by color. Then, inside of each color, you could break non-neighboring pixels into different groups. I don’t necessarily know if deep learning would be the best approach here.</p>
How to rotate images from npy files?

1414b35e42c77e0a57dd: <p>Hi, all.<br>
I’m tried to load images from npy files with pytorch dataloader. but I failed. So I googling some codes like this.</p>
<pre><code class="lang-auto">x_train = np.load('datasets/train.npy', mmap_mode='r')
batches = int(float(x_train.shape[0]) / batch_size)
for b in range(batches):
     X = torch.from_numpy(x_train[b * batch_size:(1 + b) * batch_size])
...
from torchvision.utils import save_image
save_image(X, 'image.png')
</code></pre>
<p>Now I can load images, but they are rotated 90 degrees.<br>
How can I rotate this images ?<br>
Can I rotate this from save_image function?</p>

mattdeitke: <p>You cannot rotate an image inside of <code>save_image()</code> <a href="https://pytorch.org/docs/stable/torchvision/utils.html#torchvision.utils.save_image" rel="nofollow noopener">[1]</a>, but you can use the method</p>
<pre><code class="lang-auto">torchvision.transforms.functional.roate(img, angle, resample=False, expand=False, center=None)
</code></pre>
<p>to apply transformations to PIL images, while using PyTorch <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.functional.rotate" rel="nofollow noopener">[2]</a>.</p>
Multiple submodule instances without merging gradients?

yonkshi: <p>I am working a model and it contains multiple instances of a sub network. I am having trouble making each instance of the sub-module learn independently. The gradients are averaged during back prop, making the resulting gradient useless. Below is simplified version of my model:</p>
<pre><code class="lang-auto">net_a = ModuleA() 
net_b = ModuleB()

for batch_x, label in dataset: # iterate through dataset
    
    out_b = torch.zeros(10)
    output_arr = []
    for sub_sample in some_sampler(batch): # sampling data along feature dimensions
        out_a = net_a(sub_sample) # Gradient should update `net_a` per slice
        out_b = net_b(out_a, out_b)
        product = out_a.bmm(out_b)
        output_arr.append(product)
        
    output = torch.cat(output_arr)
    loss = F.binary_cross_entropy(output, label, reduction='mean')
    loss.backward()
</code></pre>
<p>This model is a bit similar to RNN but I cannot unfold it like a RNN. When back-propagating through the graph, <code>net_a</code> should be receiving a gradient for each sub sample. I believe it has to do with model instances sharing parameters, but I cannot quite grasp how to separate the gradients?</p>
RuntimeError: size mismatch, m1: [10 x 100], m2: [10 x 100]

NOP: <p>This small code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(10,100)        
        
    
    def forward(self, inp): 
        out = self.l1(inp)
        return out
        

m = M()
inp = torch.empty(10, 100)
out = m(inp)
print(out)
</code></pre>
<h1>RuntimeError: size mismatch, m1: [10 x 100], m2: [10 x 100] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:961</h1>

ptrblck: <p>The number of input features for the linear layer is defined as <code>10</code>, while you pass an input of <code>[batch_size=10, features=100]</code>.<br>
Also note, that <code>torch.empty</code> returns an uninitialized tensor, so you might have random values, Infs, NaNs, etc. in the tensor.</p>

NOP: <p>Still, the error message is little confusing.</p>
How can I wrap a parameter/tensor into a embedding matrix for look up

KylinA1: <p>Hi, I have a learnable parameter/tensor, which is in the computational graph.And is there any way to wrap it into an <code>nn.Embedding</code>  for look up.<br>
Thanks</p>

smth: <p>you can do a lookup by just indexing into it.</p>
<pre><code class="lang-auto">x = torch.randn(100000, 128) # parameter Tensor
indices = torch.tensor([3, 44, 193, 2], dtype=torch.int64)

output = x[indices]
</code></pre>

KylinA1: <p>Thanks for your reply.<br>
This is exactly what I do right now.<br>
I am wondering is it efficient as  <code>nn.Embedding</code>  which support GPU look up.</p>
ONNX ConstantOfShape Operator

iiSeymour: <p>I have a model that results in some <code>ConstantOfShape</code> operators when exporting to ONNX. This operator isn’t widely supported in a lot of the conversion tools (CoreML, TensorFlow, TensorRT, ect) however it doesn’t appear to be a common problem looking at the lack of issues when Googling.</p>
<p>Creating a tensor of float 0 seems like a pretty common operation to do, what am I missing?</p>
Different optimization process when put parameters in different param groups?

sunshineatnoon: <p>Here is a minimal example to explain my question, suppose we have an optimizer:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters(), lr = args.lr)
</code></pre>
<p>and another one that put parameters in different parameter groups but using same learning rate:</p>
<pre><code class="lang-auto">optimizer = optim.Adam([{'params': model.aaa},
                        {'params': param for name, param in model.named_parameters() if 'aaaa' not in name}],
                        lr = args.lr)
</code></pre>
<p>In my experiments, these two optimizers give different results. Specifically, the <code>aaa</code> parameter in the first case gets a small gradient (doesn’t change much during training) while it gets larger gradients in the second case. But if I understand right, these two optimizers don’t have any difference except putting parameters into different param groups. So why are they behaving differently here?</p>

smth: <p>that sounds like a bug, but i cant fathom why.</p>
<p>Because the implementation of Adam is simple, and it just loops over param groups: <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L60" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L60</a></p>

sunshineatnoon: <p>Thanks for your reply. I found if I manually add all parameters instead of using <code>for name param in model.named_parameters()</code>, then it works fine.</p>

ptrblck: <p>Could there be a typo <code>model.aaa</code> vs <code>'aaaa'</code>?</p>
Absolute value of a n x 2 tensor row-wise

Carsten_Ditzel: <p>What is the most convenient way to calculate the row-wise abs value of two numbers across the first dimension of the tensor, i.e. the result should be a n-1 vector with the entries equal to sqrt(a^2 + b^2) for every row. Thanks in adance</p>

JuanFMontesinos: <p>SOmething like this?</p>
<pre><code class="lang-auto">import torch
a = torch.rand(10,2)
a
Out[4]: 
tensor([[0.9320, 0.5596],
        [0.9421, 0.9769],
        [0.2222, 0.6544],
        [0.6278, 0.9877],
        [0.3971, 0.8167],
        [0.3373, 0.7039],
        [0.4469, 0.9218],
        [0.8113, 0.8739],
        [0.6855, 0.1918],
        [0.5449, 0.7199]])
b=a.pow(2).sum(1).sqrt()
b
Out[6]: 
tensor([1.0871, 1.3572, 0.6911, 1.1704, 0.9081, 0.7805, 1.0244, 1.1925, 0.7118,
        0.9029])
a.norm(2,dim=1)
Out[7]: 
tensor([1.0871, 1.3572, 0.6911, 1.1704, 0.9081, 0.7805, 1.0244, 1.1925, 0.7118,
        0.9029])
</code></pre>
How to calculate the gradients of intermediate layer w.r.t another intermediate layer?

sxw: <p>Hi, everyone! I have a problem with calculating gradients of intermediate layer. In PyTorch, using backward() and register_hook() can only calculate the gradients of target layers w.r.t final output or loss. But according to chain rule, these gradients equal (gradients of target layers w.r.t one intermediate layer) * (gradients of this intermediate layer w.r.t final loss). How can I calculate these gradients of target layers w.r.t another intermediate layer?</p>

Tahir: <p>If I understand you correctly, you want to calculate the gradients of a particular layer or upto a particular layer.</p>
<p>While i do not undertand the reason or benefit behind it, you can probably do that by calculating the loss directly from that particular layer, or make a subset of layers to calculate the gradient.</p>

KylinA1: <p>call <code>targetlayer.backward(retain_graph = True)</code> ,  before the call and after your get the gradient, both time you have to set graph gradient to be zero.</p>
Nesterov's accelerated gradient + Adaptive learning rate with time decay factor

Ambareesh_S_J: <p>Training a DAE with an adaptive learning rate with a time decay factor(0.99) and Nesterov’s accelerated gradient. Is there a direct PyTorch optimizer option or a hack which lets me do this?</p>

blackbirdbarber: <p>What do you mean <code>torch.optim.SGD</code> uses Nestorov momentum already?</p>
Calling a scripting method from libtorch (other than forward)

baboonga: <p>Hello,</p>
<p>I would like to know how can I call other methods of ScriptModule?</p>
<p>For example, I have a python module which has 2 scripting methods.</p>
<blockquote>
<p>class MyScriptModule(torch.jit.ScriptModule):<br>
def <strong>init</strong>(self):<br>
super(MyScriptModule, self).<strong>init</strong>()<br>
# torch.jit.trace produces a ScriptModule’s conv1 and conv2<br>
self.conv1 = torch.jit.trace(nn.Conv2d(1, 20, 5), torch.rand(1, 1, 16, 16))<br>
self.conv2 = torch.jit.trace(nn.Conv2d(20, 20, 5), torch.rand(1, 20, 16, 16))</p>
<pre><code>@torch.jit.script_method
def regressor(self, input):
    input = F.relu(self.conv1(input))
    return input

@torch.jit.script_method
def forward(self, input):
  input = self.regressor(input)
  input = F.relu(self.conv2(input))
  return input
</code></pre>
<p>my_script_module = MyScriptModule()</p>
<p>torch.jit.save(my_script_module, ‘test_cpp.pt’)</p>
</blockquote>
<p>How could I call ‘regressor’ in c++ ?</p>
<p>Best,</p>
Best way to assign initial value to tensor?

blackbirdbarber: <p>In case:</p>
<pre><code class="lang-auto">t=torch.Tensor(64,3,28,28)
</code></pre>
<p>I would like to have all values 32, not 0 by default.</p>

ptrblck: <p><code>torch.Tensor</code> won’t initialize all values with 0s, but will use uninitialized memory, so you should manually initialize it.<br>
To set all values to a constant value, you could use several approaches:</p>
<pre><code class="lang-python">t = torch.empty(64, 3, 28, 28).fill_(32.)
t = torch.ones(64, 3, 28, 28) * 32.
</code></pre>

Deepali: <p><code>torch.full((64,3,28,28),32)</code></p>
Perform preprocessing steps on GPU or CPU

Carsten_Ditzel: <p>Is it generally advisable to perform as many steps on entities which have been previously moved to the GPU instead of doing them on the CPU?</p>
<p>Specifically I am talking about FFT and postprocessing steps like torch.roll (to mimic numpys fftshift) and taking absolute values and general multiplications and stuff.</p>
<p>Or should one first pull them down on the CPU and perform those operations there?</p>
<p>Are there general guidelines to this?</p>
<p>thanks in advance</p>
Quantization layer implementation

AtenaNguyen: <p>Hi there,</p>
<p>I would like to implement a single quantization layer with the forward pass of</p>
<blockquote>
<p>x = floor(y / scale + offset)</p>
</blockquote>
<p>where <code>scale</code> and <code>offset</code> are fixed scalars and given in the initialization. Do I need to care about backpropagation or the autograd will automatically take care of it?</p>
<p>Sorry for the newbie question<br>
Thank you</p>

dnnagy: <p>Hi! I think, this will do the job:</p>
<pre><code class="lang-auto">class Quantization(nn.Module):
  def __init__(self, scale, offset):
    super(Quantization, self).__init__()
    self.scale=scale
    self.offset=offset
    
  def forward(self,x):
    return torch.floor(x/self.scale + self.offset)
</code></pre>
<p>And then You can use it as follows:</p>
<pre><code class="lang-auto">x = torch.rand(4,1, requires_grad=True)
A = torch.rand(1,4, requires_grad=True)
q = Quantization(0.1, 1.0)
y = q(x)
z = torch.matmul(A,y)
z.backward()
</code></pre>
<p>To print gradients:</p>
<pre><code class="lang-auto">print(A.grad)
print(x.grad)
</code></pre>

AtenaNguyen: <p>Thank a lot. I will give it a try</p>
Pytorch CNN - cuda out of memory

ASHUTOSH_CHANDRA: <p>I’m running a kenel that uses pytorch and CNN for mnist digit classification. The model is fairly simple but I’m still getting CUDA OUT  OF MEMORY error.</p>
<p>Here is the link to the kernel :     <a href="https://www.kaggle.com/whizzkid/training-best-cnn-model-pytorch" rel="nofollow noopener">https://www.kaggle.com/whizzkid/training-best-cnn-model-pytorch</a></p>
<p>I think I’m using pytorch is some wrong way. Because I’ve seen more complex model getting trained on kaggle successfully. I tried same model with keras and it worked as well. Even batch size is 16</p>
<p>Kaggle provides 16GB gpu. That’s a lot to train my model. Still i’m getting the error I don’t know why.</p>
<p>Can anyone tell me what am I doing wrong ?</p>

god_sp33d: <p>The link is broken.</p>
<p>Are you getting out of memory right at the start of training or does it train for sometime and then you get OOM ?</p>
<p>Also, can you check with batch_size=1 and see if the model runs ?</p>

ASHUTOSH_CHANDRA: <p>Sorry for the link it’s working now.</p>
<p>I’m getting out of memory after 5-6 epochs or sometimes 9-10 epochs.</p>

god_sp33d: <p>make the following change in your code in both the places</p>
<pre><code class="lang-auto">net_loss += loss.item()
</code></pre>
<p>loss.item() would just sum up the loss value, where as if you add loss it would keep on adding the whole graphs.</p>

ASHUTOSH_CHANDRA: <p>thanks <a class="mention" href="/u/god_sp33d">@god_sp33d</a> , it’s working now <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
<p>Although I understood the problem a bit, could you please explain in detail what was happening.</p>
<p>Because earlier also when I was printing the loss value, it was working the same way as of now. But now I’m not getting out of memory error <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

god_sp33d: <p>when you call .item() it returns the value of the tensor which is just a number. But in your earlier case you are adding loss which is a reference(not a value of tensor). So, you are holding that the reference to that computational graph every time you are doing total_loss += loss, which otherwise would have been destroyed after the update. To see the difference yourself,</p>
<pre><code class="lang-auto">print(loss)
print(loss.item())

print(type(loss))
print(type(loss.item()))

</code></pre>
<p>The first prints a tensor which is the graph(as everything is connected) where as the second is just a float.</p>
<p>PS: If your problem is solved mark the above comment as solution so that others may find it helpful.</p>
How to finetune models trained by distributed data parallel(ddp)

kkjh0723: <p>I want to finetuning model by replacing only the last linear layer.<br>
I could do that when I used DataParallel module as below</p>
<pre><code>model = nn.DataParallel(model)

...
model.load_state_dict(checkpoint['state_dict'])
...    

model.module.fc = torch.nn.Linear(model.module.fc.in_features,
                                    opt.n_finetune_classes)
model.module.fc = model.module.fc.cuda()
</code></pre>
<p>In case of DDP, how can I do that?<br>
according to the warning in the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener">document</a>,  it seems that I cannot change parameters after I load the checkpoint.</p>
<pre><code>This module assumes all parameters are registered in the model by the time it is created. No parameters should be added nor removed later. Same applies to buffers.
</code></pre>
<p>Should I change the linear layer first and load the parameters without the linear layer?<br>
Is that a right way?</p>

pietern: <p>Yes, try modifying the module first, and once you’re done, wrapping it in <code>nn.DataParallel</code>.</p>

kkjh0723: <p>Thanks <a class="mention" href="/u/pietern">@pietern</a>. How can I load some part of parameters after wrapping with <code>nn.parallel.DistributedDataParallel</code>?</p>

pietern: <p>It is easier to load the parameters prior to wrapping with DDP. You can save/load the wrapped model as well, but then you can no longer use it <strong>without</strong> DDP.</p>
GPU is idle while dataloader's subprocesses are running

nkennek: <p>output of <code>top</code> and <code>nvidia-smi</code> suggested that:</p>
<ul>
<li>volatile gpu-util sticked almost zero while subprocesses of my dataloader was taking much CPU usage</li>
<li>once the subprocesses had disappeared, volatile gpu-utils got high, and epoch complete was confirmed next</li>
</ul>
<p>it seems that the dataloader started to provide a batch just after finishing fetching all of the contents in dataset.<br>
was this process really asyncronous or not? how can I do better?</p>
<p>my environment:</p>
<ul>
<li>Ubuntu 18.04</li>
<li>python 3.7.3 installed by conda 4.6.11</li>
<li>pytorch 1.1.0</li>
<li>cuda 10.0.130</li>
<li>data: my own image dataset which give a pair of images(3x256x256) as a single input.<br>
data is placed as bunch of jpeg files and loaded by torchvision’s default loader</li>
</ul>
<p>my dataloader had 18 workers and is set <code>pin_memory = True, shuffle = True</code>.</p>
<p>Thanks in advance,</p>
Convert caffe lstm to pytorch

Ryan95: <p>Hi, I want to convert lstm model from caffe to PyTorch, but the calculation results have a problem. I think I correctly set the parameter information required by PyTorch lstm {weight_ih_l0, weight_hh_l0, bias_ih_l0, bias_hh_l0}. because i got same shape between caffe and pytorch<br>
In caffe, bias_ih and bias_hh merged into bias, so I passed the bias to bias_ih and set bias_hh to 0.</p>
<p>can someone tell me where’s problems? thank you!</p>
Dataloader using pandas.read_csv

john90: <p>I have data files containing 5,184 (18x18x16) entries. Each file contains data from a different patient. My current dataloader setup creates a dataset of length 40 (the number of patients). Instead of treating each patient’s data separately, I want to combine them all so that I can randomize them. There should be just over 207,000 spectra in the final dataset.<br>
If I could save the data in a text file, then I could use readlines(). However, the dicom data goes out 6 decimal places and the only option with Matlab to maintain this precision is to export the data into .csv files. Using the pd.read_csv seems to load the whole batch of data as one input instead of the 5,000 it consists of.<br>
Is there a way to separate the data line by line so that when my network calls “for i, data in enumerate(dataset):” the ‘i’ will be the 200+ thoudand individual spectra instead of the 40 different patients?<br>
Here is what’s called to actually load the data:</p>
<pre><code class="lang-auto">import os.path
from pandas import read_csv
import torch
from data.base_dataset import BaseDataset
from data.image_folder import make_dataset

class AlignedLabeledSpectralDatasetModified(BaseDataset):
    def initialize(self, opt):
        self.opt = opt
        self.root = opt.dataroot
        self.dir_AB = os.path.join(opt.dataroot, opt.phase)
        self.AB_paths = sorted(make_dataset(self.root)) # Returns a list of paths of the files in the dataset

    def __getitem__(self, index):
        AB_path = self.AB_paths[index]
        with open(AB_path,"r") as file:
            AB_spectra = read_csv(file, names=['Spectra','NAA','cho'])

        A = AB_spectra.Spectra
        #B = [AB_spectra.NAA, AB_spectra.cho]

        return {'A_data': A, 'A_paths': AB_path}

    def __len__(self):
        return len(self.AB_paths)

    def name(self):
        return 'SingleSpectralDataset'
</code></pre>

simon_thom: <p>I think your problem statement is a little confusing. Couldn’t you just make a sorting by patient in MATLAB and create a new .csv file from the sorted data with 207k entries?</p>
<p>A word of advise however is to do all pandas-loading related work in the init-method of your Dataset as this will make it run much, much faster from my experience.</p>
How to read the data-set with links?

Sainadh086: <p>I am having a data-set which is in tsv format.  It contains website links. Where each link contain a single image. How can I read the data of more than 20,000 images without downloading each image. Help me out by giving some information that I want load the images directly from the links  and to train them.</p>

Tahir: <p>Are the links refering to the images? if yes, then images will have to be downloaded.</p>
<p>I am not sure what you  mean by link containing images… you mean a URL contains an image? maybe you can post a sample to see what format it actually is.</p>

Sainadh086: <p><a class="mention" href="/u/tahir">@Tahir</a> thats correct i am referring to url<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b9bf2451942f81e9962b6ad86b138115fe271a5a.png" data-download-href="https://discuss.pytorch.org/uploads/default/b9bf2451942f81e9962b6ad86b138115fe271a5a" title="Screenshot%20(151)"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b9bf2451942f81e9962b6ad86b138115fe271a5a_2_690x388.png" alt="Screenshot%20(151)" data-base62-sha1="qvbQNye1uptFjvZmsScAOfDfJJE" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b9bf2451942f81e9962b6ad86b138115fe271a5a_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b9bf2451942f81e9962b6ad86b138115fe271a5a_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b9bf2451942f81e9962b6ad86b138115fe271a5a_2_1380x776.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20(151)</span><span class="informations">1920×1080 215 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Tahir: <p>Then images will have to be downloaded before they can be loaded into tensors. either 1 by 1 or download all of them at once in advance and save them locally(in the later case it will be quicker).</p>

JuanFMontesinos: <p>Imageio allows to “read” from url.</p>
How to estimate model size from number of parameters?

koustuvsinha: <p>I notice sometimes even with less number of model parameters my model size is higher. Is there any pytorch specific way to estimate the required model size in GPU <em>before</em> running? Given I do the required python variable cleaning (<code>del</code>).</p>

StefanDenn3r: <p>Hi,<br>
not sure if it is still relevant but maybe this helps:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/sksq96/pytorch-summary" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/8557728?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/sksq96/pytorch-summary" target="_blank" rel="nofollow noopener">sksq96/pytorch-summary</a></h3>

<p>Model summary in PyTorch similar to `model.summary()` in Keras - sksq96/pytorch-summary</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Export an object detection model into onnx

TheTrveAnthony: <p>Hi everybody,</p>
<p>I’m trying to export the faster cnn model into onnx in order to use it with opencv, and get the following error :</p>
<pre><code class="lang-auto">/lib/python3.6/site-packages/torchvision/ops/poolers.py:101: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  scale = 2 ** torch.tensor(approx_scale).log2().round().item()
(....)
RuntimeError: log2_vml_cpu not implemented for 'Long'
</code></pre>
<p>However it works fine when I try it with an image classification model, here is my code:</p>
<pre><code class="lang-auto">import sys as s
import torch.onnx
import torchvision.models
 
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
dummy_input = torch.randn(1, 3, 800, 800)
nom = s.argv[1] + ".onnx"
torch.onnx.export(model, dummy_input, nom)
</code></pre>
<p>I also tried it with a segmentation model and got the exact same error, so what am I missing ?<br>
thanks to those who will answer me</p>
Custom Dataset Implementation

mhubii: <p>Since there have been some requests on how to implement a dataset with libtorch, <a href="https://github.com/mhubii/libtorch_custom_dataset" rel="nofollow noopener">here</a> is a simple example. The dataset loads images of apples and bananas by their file names with OpenCV and converts them into a tensor. Furthermore, you can find a classifier in there, train it, and evaluate classification results.</p>
<p>Hope it helps</p>
<p>Martin</p>
Interactive debug in distributed.launch

tree33: <p>Hi, is there any interactive way to debugin distributed launch?<br>
In pytorch 0.4.1, I use pdb in dataparallel, however, it seems that distributed launch would split the process into multiple copies. When I type sth in pdb, my input would also be split to different copies of processes, that’s not what I expected.</p>
<p>So I’m wondering if there is any method that can help me do interactive debugging in distributed launch?</p>

pietern: <p>This happens because all processes share the same input file descriptor. When you type a character, the first process who reads it will get it. This makes interactive debugging almost impossible. What you can try, in lieu of a a proper solution, is close the input descriptor by running <code>sys.stdin.close()</code> on the ranks where you don’t want to run pdb.</p>
Loading optimizer in a distributed setting

Konstantin_Solomatov: <p>I want to load snapshot from a file on one of the machines running in a distributed setting. From what I see, optimizers aren’t broadcast among machines in such a case. Is there any easy way to do it?</p>

pietern: <p>There is no common way to expose optimizer state AFAIK. If you know how you can access the state of your optimizer then you’ll be able to synchronize it by using <code>torch.distributed</code> collectives directly, e.g. <code>broadcast</code>.</p>
DistributedDataParallel - Master starts without workers

Moin_Nadeem: <p>Hi!</p>
<p>I’m implementing DistributedDataParallel in my code. However, when I start it, if I use PyTorch’s lanch module, one task will start training before the others have begun. This is different from without using PyTorch’s launch module, when I’ll see the processes wait on each other before starting the next epoch, etc.</p>
<p>I’m using an implementation that mirrors <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="nofollow noopener">this Medium article</a>. I’ve been struggling with this issue for two days now, so any help would be extremely appreciated!</p>
<p>Thanks!</p>

pietern: <p>When <code>torch.nn.parallel.DistributedDataParallel</code> is initialized with the right distributed context then every iteration should happen in lock step between all processes. If a single process starts going by itself, I think there is something missing in initialization.</p>
<p>Can you share a code snippet how you initialize all of them?</p>
Heterogeneous GPUs but same computation time

hpc-unex: <p>Hello,</p>
<p>I’m using a RTX 2080 ti and a GTX 1050 ti in a two node cluster using pytorch. The problem cames when I execute it (distributed) but both of them take the same time solving MNIST. There are no sync points. Can anyone help me?</p>

hpc-unex: <p>My cuda version is 10.0 in RTX and 9.2 in GTX. Im using pytorch 1.2 with mpi 3.1</p>

ptrblck: <p>Are you using DDP?<br>
If so, the slower card might sync the faster one.</p>
<p>Or are you profiling the cards separately? If so, your code might have other bottlenecks (e.g. data loading). Have you profiled it?</p>

pietern: <p>We don’t guarantee compatibility between different versions of PyTorch. You say you have one version compiled against CUDA 10 and another against CUDA 9.2. This <em>might</em> work, but YMMV.</p>

hpc-unex: <p>Yes, im using DPP, but im using asynchronous all_reduce to average gradients, so theres no synchronization if I dont make explicit a.wait() (which im not doing just to test), right? In that case, training times still the same which makes no sense for me. Am I losing anything?</p>
VGG16 'fc' error when performing Transfer learning

Rajamannar_Krishnamo: <p>when performing transfer learning I was using the optimizer SGD and the pre-trained network is VGG16 I got this error the code which I used was</p>
<p><code>optimizer_transfer = optim.SGD(model_transfer.fc.parameters(),lr= 0.001)</code><br>
<code>AttributeError: 'VGG' object has no attribute 'fc'</code></p>
<p>when I removed the fc from the code I get the following error</p>
<p><code>optimizer_transfer = optim.SGD(model_transfer.parameters(),lr= 0.001)</code><br>
<code>ValueError: optimizing a parameter that doesn't require gradients</code></p>
<p>These error occurs only when the I use the VGG16</p>

god_sp33d: <p>That is because VGG16 model doesn’t have model.fc instead it has model.classifier. Change your code to</p>
<pre><code class="lang-auto">optim.SGD(model_transfer.classifier.parameters(), lr=0.001)
</code></pre>
<p>Also, it is always better to first print the model and see the way they are named.</p>
<pre><code class="lang-auto">print(model_transfer)
</code></pre>
<p>The final error is because I think you are trying to freeze some part of the network and I guess you frozen everything(I am not sure though).  In optim.SGD change the code from</p>
<pre><code class="lang-auto">model_transfer.parameters() to filter(lambda p: p.requires_grad, model_transfer.parameters())
</code></pre>
<p>to avoid the error but check thoroughly through your network to see the parameters and frozen layers.</p>

Rajamannar_Krishnamo: <p>thanks for your reply <a class="mention" href="/u/god_sp33d">@god_sp33d</a><br>
but I got this for the first solution<br>
<code>ValueError: optimizing a parameter that doesn't require gradients</code></p>
<p>but for the second solution<br>
initially I set my requires_grad is set to False and the model did not train but when I set it to True and run it the batch loss I get it so high <code>24.034479</code></p>
<p>my code is<br>
<code>for param in model_transfer.parameters(): param.requires_grad = True criterion_transfer = nn.CrossEntropyLoss() optimizer_transfer=optim.SGD(filter(lambdap:p.requires_grad,model_transfer.parameters()),lr=0.001)</code></p>

god_sp33d: <p>The first change I suggested was for your error: model.fc, I am assuming you fixed that by changing to model.classifier.</p>
<p>coming to my second suggestion. you are getting the following error because you are trying to include the parameters in the optimizer where requires_grad = False. So, I suggested you to filter out those parameters by applying filter function.</p>
<pre><code class="lang-auto">ValueError: optimizing a parameter that doesn't require gradients
</code></pre>
<p>I think these were your concerns in the original post. As you are getting a very high batch loss. Can you please add more details like batch size ? Also, I see that you are using a pretrained model and finetuning. S0, why don’t you start with a lower learning rate like 0.0001 ?</p>
<p>Also, how does the loss change as you progress ? Does it decrease or very unstable ? If yes, can you show some values ?</p>
DistributedDataParallel with autograd.grad

wenxx151: <p>Hi,</p>
<p>I am using a loss function that contains the gradient of the output w.r.t. to the input of the network, which I obtained with <code>autograd.grad</code>.</p>
<p>I am interested in training my model in parallel using the <code>DistributedDataParallel</code> container. However, as one WARNING in the doc page mentions, <code>DistributedDataParallel</code> does not support <code>autograd.grad</code>. If I understand correctly, this is because local model parameters (not the one averaged across devices) will be used if I use <code>autograd.grad</code> after the <code>forward</code> call. Of course, this is incorrect.</p>
<p>Looking into the implementation of <code>DistributedDataParallel</code>, I found the method <a href="https://github.com/pytorch/pytorch/blob/de582e2f89d154ef7579caf8fe8aac80c2d8a569/torch/nn/parallel/distributed.py#L412" rel="nofollow noopener">_sync_params</a> is called at the beginning of the <code>forward</code> method to sync params across devices. My question is:</p>
<p>Is it OK for me to call <a href="https://github.com/pytorch/pytorch/blob/de582e2f89d154ef7579caf8fe8aac80c2d8a569/torch/nn/parallel/distributed.py#L412" rel="nofollow noopener">_sync_params</a> once more before I use <code>autograd.grad</code> to compute the gradient of the output w.r.t. to the input and then use it in my loss function? In such, the gradient computation will use the averaged parameters. Is there any caveats?</p>

pietern: <p>The problem here is that <code>DistributedDataParallel</code> performs gradient averaging across processes by hooking into the <a href="https://github.com/pytorch/pytorch/blob/v1.1.0/torch/csrc/autograd/functions/accumulate_grad.cpp" rel="nofollow noopener"><code>AccumulateGrad</code></a> function. This allows for performing averaging for the last most gradients while autograd is still running.</p>
<p>Would it be possible for you to first compute the initial loss, call <code>autograd.backward</code> instead of <code>autograd.grad</code>, and have it accumulate the first order gradients in the model parameters? Then you could detach those and compute something else before letting the optimizer do its thing. If not, then you’ll have to perform your own averaging, I think.</p>
Problem with cuda

Sainadh086: <p>My program is showing that cuda is available when running the command <strong>torch.cuda.available()</strong><br>
but when I try to run my model on cuda by typing <strong>model.cuda()</strong> it is not running and showing cuda is not found. Can anyone explain me how to run my model on gpu.</p>

Sainadh086: <p>Command is <em>torch.cuda.is_available()</em> I tried reinstalling pytorch on my laptop and the gpu I am using is gtx 1050 ti,os -&gt;windows 10,platform → Anaconda(Jupyter) and I also tried different commands then also my problem is not solved <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/78f22775c3783ff194a9172efa2f769454085331.png" data-download-href="https://discuss.pytorch.org/uploads/default/78f22775c3783ff194a9172efa2f769454085331" title="Screenshot%20(146)"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/78f22775c3783ff194a9172efa2f769454085331_2_690x388.png" alt="Screenshot%20(146)" data-base62-sha1="hfW6Mb37w9MVNdEpCdBE4V9MccF" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/78f22775c3783ff194a9172efa2f769454085331_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/78f22775c3783ff194a9172efa2f769454085331_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/78f22775c3783ff194a9172efa2f769454085331_2_1380x776.png 2x" data-dominant-color="E1E1E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20(146)</span><span class="informations">1920×1080 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Which driver and CUDA version are you using?<br>
Did CUDA work before?<br>
If you’ve just installed all drivers etc., did you restart your machine?</p>

Sainadh086: <p>I solved this problem by reinstalling with cuda version 10.0<br>
Before I installed pytorch with cuda version 9.0. It not worked properly and my model not utilized gpu.<br>
Previous version detected my gpu by it is not able to utilize it.<br>
So I suggest to be in latest version of cuda<br>
Thanks for helping me.</p>

Sainadh086: <p>Note that You have to manually turn on the GPU for the first in your nvidia control panel.<br>
In control panel you have an option to auto select when needed, Nvidia gpu and integrated gpu.<br>
select Nvidia GPU.<br>
Because  cuda is not automatically turn on the GPU.</p>
Gradients update

hpc-unex: <p>Hello,</p>
<p>As I see in the code, there is a Queue used by background threads in order to communicate the parameters with Allreduce. My question is how this parameters are updated. Allreduce is a blocking collective so the background threads will be waiting until all of them enqueue their parameters. So, maybe there is the possibility that parameters arent updated so a process isnt taking into account sometimes the parameters of the others process which have different data. Am I right? Does this affect to the precision ? How the parameter update work then? Whats the point of using a Queue?</p>

pietern: <p>Which code were you looking at?</p>
<p>In version 1.0 the distributed backend has been updated such that all collectives run asynchronously w.r.t. the main thread, even if they are blocking. For MPI collectives this means they are run on a single background thread. The queue approach was taken in PyTorch before version 1.0. In 1.1 we introduced a new C++ based gradient reduction mechanism (see <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/reducer.cpp" rel="nofollow noopener">reducer.cpp</a>) that concatenates multiple gradients into a large bucket and performs <code>allreduce</code> against those buckets instead of individual parameters.</p>