akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why is model divided onto two device?

username: KouWeiBin
cooked: <p>I used pytorch to train a faster-rcnn model. After converting it into torchscript model, I loaded the model in c++ on GPU. but via the qt debug table, I find that the model is divided onto cup and cuda:0. so can any one tell me why it is and how to fix it?  Thanks very much!</p>

username: KouWeiBin
cooked: <p>I have loaded the TorchScript model on cuda:0 and I use cmd print(model.forward.code) to print the forward function. I found that there are still some variables on cpu. so how can I deal with it?</p>
<p>def forward(self,<br>
argument_1: Tensor) → Tuple[Tensor, Tensor, Tensor]:<br>
_0 = self.model<br>
_1 = _0.roi_heads<br>
_2 = _0.rpn<br>
_3 = _0.backbone<br>
img = torch.select(argument_1, 0, 0)<br>
s = ops.prim.NumToTensor(torch.size(img, 1))<br>
s0 = ops.prim.NumToTensor(torch.size(img, 2))<br>
image = torch.select(argument_1, 0, 0)<br>
mean = torch.to(CONSTANTS.c0, torch.device(“cpu”), 6, False, False, None)<br>
std = torch.to(CONSTANTS.c1, torch.device(“cpu”), 6, False, False, None)<br>
_4 = torch.slice(mean, 0, 0, 9223372036854775807, 1)<br>
_5 = torch.unsqueeze(torch.unsqueeze(_4, 1), 2)<br>
_6 = torch.sub(image, _5, alpha=1)<br>
_7 = torch.slice(std, 0, 0, 9223372036854775807, 1)<br>
_8 = torch.unsqueeze(torch.unsqueeze(_7, 1), 2)</p>
Pruning with weight_norm

username: XYZCODE123
cooked: <p>Hi, when using <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.weight_norm.html" rel="noopener nofollow ugc">weight_norm</a> we replace weight tensor with weight_g and weight_v tensors. While pruning which tensor should be masked? Should mask just weight tensor or weight_g and weight_v?</p>
Use class weight with Binary Cross Entropy Loss

username: John_Doe
cooked: <p>Hello,</p>
<p>I am doing a segmentation project with a Unet. I have an unbalanced dataset with 2 class and I want to apply, as a first step, a weight for each class. I use the loss <code>torch.nn.BCELoss()</code>. After looking on internet, it seems that people that had a similar problem were advised to switch to <code>BCEWithLogitsLoss()</code> which has a <code>pos_weight</code> argument to choose class weight.</p>
<ol>
<li>I would prefer if possible to keep using <code>torch.nn.BCELoss()</code> because, from what I understood, if I use <code>BCEWithLogitsLoss()</code> I would have to remove the sigmoid layer at the end of my network which means that in every single place where I make a prediction with my network with code like <code>pred = model(data)</code> I would then need to add a sigmoid to my prediction to get the same output as before. This solution doesn’t seem great as there are many places in my codes where I use <code>pred = model(data)</code>.</li>
</ol>
<p>Have I understood my problem clearly ? Is there any way to keep using <code>BCELoss()</code> ?</p>
<p>Thank you for reading.</p>

username: ptrblck
cooked: <p>I believe you should be able to manually weight the unreduced loss, if you are using binary targets. If that’s not the case, you would need to use <code>nn.BCEWithLogitsLoss</code> with the <code>pos_weight</code> argument. <a href="https://github.com/pytorch/pytorch/issues/5660">This issue</a> explains the use case a bit more and this code snippet shows the results:</p>
<pre><code class="lang-python">for i in range(100):
    pos_weight = torch.randint(1, 100, (1,)).float()
    
    criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
    x = torch.randn(10, 1, requires_grad=True)
    y = torch.randint(0, 2, (10, 1)).float()
    
    loss = criterion(x, y)
    
    criterion_raw = nn.BCEWithLogitsLoss(reduction='none')
    loss_raw = criterion_raw(x, y)
    weight = torch.ones_like(loss_raw)
    weight[y==1.] = pos_weight
    loss_raw = (loss_raw * weight).mean()
    
    criterion_raw_sig = nn.BCELoss(reduction='none')
    loss_raw_sig = criterion_raw_sig(torch.sigmoid(x), y)
    loss_raw_sig = ((loss_raw_sig) * weight).mean()

    print(loss - loss_raw)    
    print(loss_raw_sig - loss)
</code></pre>
<p>Note that <code>nn.BCEWithLogitsLoss</code> increases the numerical stability, so it might still be valuable to switch to this criterion.</p>

username: John_Doe
cooked: <p>Thank you for your help, it is really appreciated.</p>
<p>I have switched to using <code>nn.BCEWithLogitsLoss(pos_weight=pos_weight)</code> as it seems to be the easiest solution.</p>
<p>Just to be sure, if I have a dataset repartition of 80%  of negative example and 20% of positive example, then <code>pos_weight</code> should be 4 right ?</p>
<p>Have a nice day !</p>

username: ptrblck
cooked: <p>Yes, it would be calculated as <code>nb_neg/nb_pos = 80/20 = 4</code>.</p>
Simple code example with NVLink support

username: Florin_Andrei
cooked: <p>I need to test a system that has multiple GPUs, linked with NVLink. What is the minimal code I could run to show NVLink actually works?</p>
<p>See this simple code example - how would you change it to take advantage of NVLink?</p>
<p><a href="https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html</a></p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html"><code>DistributedDataParallel</code></a> via NCCL would use NVLink, if available.<br>
For a quick performance test, I would recommend to run the <a href="https://github.com/NVIDIA/nccl-tests">nccl-tests</a> and also verify the connections between the GPUs via <code>nvidia-smi topo -m</code>.</p>

username: Florin_Andrei
cooked: <p>I have to actually demo PyTorch, so I’ll see if I can get DistributedDataParallel to work.</p>
<p>nccl-tests looks like a useful hardware / config check, I’ll give it a try.</p>
Architecture of deeplabv3_resnet50

username: BjornSing
cooked: <p>How does the architecture of deeplabv3_resnet50 look like?</p>
<p>I am using this command to get the model:<br>
model = torchvision.models.segmentation.<code>deeplabv3_resnet50</code> (<em>pretrained=False</em> , <em>progress=True</em> , <em>num_classes=21</em> , <em>aux_loss=None</em> , **<em>kwargs</em> )</p>
<p>I can then get some impression of how the model looks like just by printing model, but this doesn’t give me a good intuitive way of visualizing the model.</p>

username: ptrblck
cooked: <p>Printing out the model wouldn’t show the computation graph and would only print the child modules, so I agree that this would not be sufficient to “see” the structure.<br>
You could check out e.g. <a href="https://github.com/szagoruyko/pytorchviz"><code>PyTorchViz</code></a> to visualize the computation graph in case that’s helpful.</p>
<p>PS: Often I also take a look at the source code, but for segmentation/detection models this is unfortunately also not trivial.</p>

username: BjornSing
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> ! <a href="https://github.com/szagoruyko/pytorchviz" rel="noopener nofollow ugc"><code>PyTorchViz</code></a> was exactly what I was looking for <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Pytorch tabular+ precision recall curve

username: cekik
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c550ce8945c036bfccdb021bd9a415cc8b5a911.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c550ce8945c036bfccdb021bd9a415cc8b5a911" title="pytorch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c550ce8945c036bfccdb021bd9a415cc8b5a911.png" alt="pytorch" data-base62-sha1="daO96Y2yWfknCatXOmQEYQoI1CV" width="598" height="500" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch</span><span class="informations">852×712 32.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can ı implement precision-recall curve to above script, classification examples with pytorch tabular are not  enough</p>

username: ptrblck
cooked: <p>You could use e.g. <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html"><code>sklearn.metrics.precision_recall_curve</code></a> and provide the expected inputs as numpy arrays via <code>tensor.numpy()</code>.</p>
Efficient data loading

username: Jay_Super
cooked: <p>Hello, I have a dataset that has a shape similar to an image (i.e. [N_sample, N_feature_dim, Height, Width]. I saved the dataset as a chunk which results in a single file “train.pt” with a size of more than 50 G.  Because of its size, it is taking too long when I load them in the first place.</p>
<p>So here is my question. Is it better to save the dataset sample-wise (e.g 1.pt, 2.pt, 3.pt… N.pt) or save them into a different format other than pt? I would appreciate any form of advice!</p>
Appending Torch arrays having one similar dimension

username: mollasimir
cooked: <p>Hi There! I would like to know how it can be possible to get a Tensor T of size [x,y,z] , given two tensors T1 [x,y] and T2 [x,z]. Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> !</p>

username: ptrblck
cooked: <p>I don’t think it would be possible to concatenate or stack these tensors to the desired shape, since the number of elements differ.<br>
E.g.:</p>
<pre><code class="lang-python">x, y, z = 2, 3, 4
print(x*y*z)
&gt; 24
print((x*y) + (x*z))
&gt; 14
</code></pre>
<p>You could transform each tensor to <code>T</code> by adding the missing dimension (via <code>unsqueeze</code>) and then e.g. calling <code>expand</code> or <code>repeat</code> on it.</p>
PyTorch 1.0 loading VGGFace2 weights in Python3.7

username: Alpkant
cooked: <p>I am using Python3.7 and PyTorch 1.0 to develop a face recognition system. I want to use VGGFace2 Resnet50 pretrained model as described <a href="https://github.com/ox-vgg/vgg_face2" rel="noopener nofollow ugc">here</a> as a feature extractor. I have downloaded the model and weights.<br>
I run the following codes as project readme says:</p>
<pre><code>MainModel = imp.load_source('MainModel', 'resnet50_128_pytorch.py') 
model = torch.load('resnet50_128_pytorch.pth')
</code></pre>
<p>First line executed as expected but in the second line i got</p>
<blockquote>
<p>‘ascii’ codec can’t decode byte 0xc3 in position 1124: ordinal not in<br>
range(128)</p>
</blockquote>
<p>I searched in the Stackoverflow and Google and i saw that it may be about this model saved with Python2 and loading from Python3 creates a problem. Is there any way that i can solve this?</p>
<p>Thank you.</p>

username: Kushaj
cooked: <p>Have you tried this <a href="https://www.dlology.com/blog/how-to-load-python-2-pytorch-checkpoint-in-python-3-1/" rel="nofollow noopener">link</a></p>

username: Alpkant
cooked: <p>I did but while i save the state dict from Python2 i got KitModel doesn’t have <strong>get_item</strong> function error appeared. Unfortunately it didn’t work. I think it’s about the custom model</p>

username: Kushaj
cooked: <p>When you try to save the state_dict, did you have the model definition also.</p>

username: Alpkant
cooked: <p>With the model in my case if you mean this</p>
<pre><code class="lang-auto">MainModel = imp.load_source('MainModel', 'resnet50_128_pytorch.py') 
</code></pre>
<p>yes i also had the model definition before save</p>

username: pavan_kalyan
cooked: <p>Have you successfully implemented it…I am also trying for the same GitHub links seems to be broken hope you could help …</p>
What does dataset.loadnext() do?

username: Dylan_Yung
cooked: <p>I see some research code [<a href="https://github.com/KarlXing/LUSR/blob/main/train_lusr.py" class="inline-onebox" rel="noopener nofollow ugc">LUSR/train_lusr.py at main · KarlXing/LUSR · GitHub</a>] calling .loadnext() on their ExpDataset object. What does this do? I can’t seem to find documentation on this function.<br>
(<a href="https://github.com/KarlXing/LUSR/blob/main/train_lusr.py" class="inline-onebox" rel="noopener nofollow ugc">LUSR/train_lusr.py at main · KarlXing/LUSR · GitHub</a>)</p>

username: Epoching
cooked: <p>Looks like it’s just a function they wrote for their custom <code>Dataset</code> class:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4c29756f73d7f6630bb36342ee6a3777efe3b11.png" data-download-href="https://discuss.pytorch.org/uploads/default/e4c29756f73d7f6630bb36342ee6a3777efe3b11" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4c29756f73d7f6630bb36342ee6a3777efe3b11.png" alt="image" data-base62-sha1="wDHLV7aXHBVwkde5pcYWkls2POp" width="565" height="500" data-dominant-color="13161C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">667×590 19.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Source: utils.py via <a href="https://github.com/KarlXing/LUSR/blob/main/utils.py" rel="noopener nofollow ugc">LUSR/utils.py at main · KarlXing/LUSR · GitHub</a></p>
How do I store a graph to use autograd.grad() on it again later?

username: Ethan_R
cooked: <p>Hello Everyone,</p>
<p>I am attempting to make my code more efficient by storing a graph, say x, somewhere and then calling autograd.grad(x) multiple times rather than re-computing x each time I want to do it. Let me give a more concrete example.</p>
<pre><code class="lang-auto">def hvp(y, w, v):
    if len(w) != len(v):
        raise(ValueError("w and v must have the same length."))

    #First backprop
    first_grads = grad(y, w, retain_graph=True, create_graph=True)

    # Elementwise products
    elemwise_products = 0
    for grad_elem, v_elem in zip(first_grads, v):
        elemwise_products += torch.sum(grad_elem * v_elem)

    # Second backprop
    return_grads = grad(elemwise_products, w, create_graph=False)

    return_grads = To_List(return_grads).detach()

    return return_grads
</code></pre>
<p>Here first_grads will be the same each time I call the function (as y and w will also be the same), so it makes little sense to calculate y, w, and first_grads each time I want to call the function (y is the model loss, w contains the model parameters). Is there a way to somehow retain first_grads so I don’t have to re-compute it? I have already tried creating a class that stores y, w, first_grads and calls this hvp as one of its functions. I also tried storing the return_grads.detach() in a different variable in case detach() deleted the whole graph.</p>
<p>Thank you,<br>
Ethan</p>
Collection of popular CNN model architectures and benchmark datasets for PyTorch framework

username: jason-deeplite
cooked: <p>Deeplite Torch Zoo - <a href="https://github.com/Deeplite/deeplite-torch-zoo" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Deeplite/deeplite-torch-zoo</a></p>
<p>A recently created Github repository - The <code>deeplite-torch-zoo</code> package is a collection of popular CNN model architectures and benchmark datasets for PyTorch framework. The models are grouped under different datasets and different task types such as classification, object detection, and segmentation.</p>
My torch program runs very slowly，please help me

username: Richard_Gan
cooked: <p>The training data is floating point. The epoches is set to 10000,then jump to custom function. In the function, the batch_num is 95. I set a batch of data to be 29 rows. Each row of data is 20 dimensions. I try to run on 3080. but it is too slow. How can I solve it?</p>
<p>the main func:<br>
for epo in range(epoch):<br>
print(“In iteration: %d” % (epo + 1))<br>
mse_all = model.SAE_GRU_Network(X, Y, U, time_steps, batch_size, hidden_size)<br>
loss = mse_all.cuda()<br>
opt.zero_grad()<br>
loss.backward()<br>
opt.step()</p>
<pre><code>    print(epo+1, sqrt(mse_all/(batch_size*time_steps*data_dim)))
</code></pre>
<p>the SAE_GRU_Network:<br>
def SAE_GRU_Network(self, X, Y, U, time_steps, batch_size, hidden_size):<br>
mse_batch = 0<br>
h0 = torch.zeros(1, hidden_size, device=device)<br>
for i in range(batch_size):<br>
x_input = X[i][0]<br>
mse_steps = 0<br>
h = h0<br>
for j in range(time_steps):<br>
y_target = Y[i][j]<br>
u_input = U[i][j]<br>
h, y_output = self.one_step(h, x_input, u_input)<br>
x_input = y_output<br>
y_true = torch.reshape(y_target, [-1])<br>
y_pred = torch.reshape(y_output, [-1])<br>
mse_steps += torch.sum(torch.square(y_true - y_pred))<br>
mse_batch += mse_steps<br>
return mse_batch</p>

username: Richard_Gan
cooked: <p>The data type uses float32…To be precise,batch_size is batch_num</p>

username: Richard_Gan
cooked: <p>I translated it from Chinese. Some parts may be inaccurate, please forgive me</p>

username: ptrblck
cooked: <p>You could profile your code using the <a href="https://pytorch.org/blog/introducing-pytorch-profiler-the-new-and-improved-performance-tool/">profiler</a> or e.g. <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">Nsight Systems</a> to check for bottlenecks and depending on these findings try to speed them up (e.g. the data loading in case your GPU would be waiting for the next data batches to arrive).</p>

username: Richard_Gan
cooked: <p>Maybe the net I designed is not suitable for GPU. Thank you very much for your kind attention personal response</p>
Torch.empty - how is it implemented

username: bottaio
cooked: <p>Hello,<br>
I am starting to getting more accustomed with pytorch source code but I’ve recently hit a wall. I don’t quite understand C &lt;-&gt; CUDA &lt;-&gt; Python bindings yet I guess and that’s my problem here.</p>
<p>I just couldn’t find the source code of <code>torch.empty</code> function. Probably the easiest way to find it out is to ask <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Thanks.</p>

username: SimonW
cooked: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/TensorFactories.cpp#L32-L34" target="_blank">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/TensorFactories.cpp#L32-L34" target="_blank">pytorch/pytorch/blob/master/aten/src/ATen/native/TensorFactories.cpp#L32-L34</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="32" style="counter-reset: li-counter 31 ;">
<li>Tensor empty(const Type&amp; dtype, IntList size) {</li>
<li>return dtype.tensor(size);</li>
<li>}</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: libtorch_coder
cooked: <p>For PyTorch 1.17 or 1.18, do you know how the ‘empty(…)’ implemented?</p>
<p>I just found :</p>
<pre><code class="lang-auto">Tensor empty(
    IntArrayRef size,
    at::optional&lt;DimnameList&gt; names,
    const TensorOptions&amp; options,
    optional&lt;MemoryFormat&gt; optional_memory_format) {
  if (!names.has_value()) {
    return at::empty(size, options, optional_memory_format);
  }
  TORCH_CHECK(options.layout() == Layout::Strided,
      "NYI: named tensors only support strided layout");
  TORCH_CHECK(options.device().is_cpu() || options.device().is_cuda(),
      "NYI: named tensors only support CPU and CUDA tensors");
  auto result = at::empty(size, options, optional_memory_format);
  internal_set_names_inplace(result, names);
  return result;
}
</code></pre>

username: Luoshang_Lowson_Pan
cooked: <p>same here. Wasn’t able to find the implementation for <code>at::empty</code></p>
About if gradient is computed correctly

username: Justin_Chen
cooked: <p>Hi there,</p>
<p>I met this error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.DoubleTensor [16, 4, 4]], which is output 0 of SliceBackward, is at version 401; expected versio
n 399 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>Then after fixing the inplace operation, there is no warning of this kind, does this mean that the gradient is calculated correctly?</p>
<p>I reused the output of the model several times before compute the loss which looks like:</p>
<pre><code class="lang-auto">param = model(something)
pred = new tensor
for xx iteration:
    new_pred = pred
    new_pred[certain slice] += param * pred[certain slice]
    pred = new_pred
loss(pred, target)
</code></pre>
<p>Will this cause any problem for deciding on what value loss should dependent on? (or should I keep the update history of pred in separate address rather than directly modify it?)</p>
<p>Thanks a lot!</p>
Accessing the Backward of a Function (gelu) without Autograd

username: Seledorn
cooked: <p>Hi,</p>
<p>I’m having a checkpointed model where the parts between the checkpoints can be run in place, but, as well all know, in place operations do not play well with the autograd.</p>
<p>However, the first forward of the checkpointed model could be run in place, but as far as I know there’s no way of telling the checkpointing to run different things during the first forward and the backward-forward.</p>
<p>In order to work around this, I’m thinking of just creating a large custom autograd function which does the manual gradient checkpointing in the backward. Though I’m not particularly keen on reimplementing the backward of built-in functions such as gelus my self.</p>
<p>Is there a way of accessing for example gelu_backward directly without having to do a forward pass?</p>
<p>Seemingly I could create a c++ extension and access gelu_backward form Aten directly but that is a bit inelegant.</p>
<p>Thanks</p>
Recalculate gradients

username: CodyBrody
cooked: <p>Hi All,</p>
<p>I have calculated some weights per label for my dataset. Something like this:</p>
<p><code> label_weights =  {0:2.0, 1:3.5, 2:0.7, 3:1.5}</code></p>
<p>Next, I want to multiply these weights with the gradient of the loss function according to the label of the input image. Will something like this work:</p>
<pre><code class="lang-auto">for x , t in train_loader:
     optimizer.zero_grad ( )
     z = network (x.to (device))
     J = loss (z , t.to (device))
     J.backward ( )
      for p in network.parameters():
           p.grad *= label_weights 
      optimizer.step ( )
</code></pre>
<p>Or is there another way to do it?</p>
<p>Thanks for your help in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>No, I don’t think your approach would work, since the <code>.grad</code> attributes do not contain any label information, so you won’t be able to multiply the <code>dict</code> with it.<br>
Depending on the used loss function you could use the <code>weight</code> argument (e.g. in <code>nn.CrossEntropyLoss</code>) or you could use an unreduced loss and multiply it before applying the reduction and calling <code>backward()</code>.</p>

username: CodyBrody
cooked: <p>Thank you for your reply. I found an answer that elaborates on the first method (<a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731" class="inline-onebox">Passing the weights to CrossEntropyLoss correctly</a>). On that answer, is it enough to pass the weights list without the class labels, even when the labels are one-hot vectors?</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> doesn’t work with one-hot encoded targets, but needs a target containing class indices. The passed <code>weights</code> correspond to these class indices, too.</p>
Cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:450

username: Ritwik_Jain
cooked: <p>I was running a python script when I got this error: “RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:450”.</p>
<p>How can I solve this problem?</p>
<p>Pytorch version: 1.1.0<br>
Cuda version: 9.0</p>
<p>Please let me know if any other details are required.</p>

username: ptrblck
cooked: <p>Could you update PyTorch to the latest release and check, if you are still seeing the issue, please?</p>

username: Ritwik_Jain
cooked: <p>Actually the script only works on 1.1.0. If I run it on the latest release I get the following error: “AttributeError: ‘AvgPool3d’ object has no attribute ‘divisor_override’”</p>

username: ptrblck
cooked: <p>The <code>divisor_override</code> argument isn’t available in <a href="https://pytorch.org/docs/1.1.0/nn.html#torch.nn.AvgPool3d"><code>nn.AvgPool3d</code> in PyTorch 1.1.0</a> and was added in <a href="https://pytorch.org/docs/1.2.0/nn.html#torch.nn.AvgPool3d">1.2.0</a>, so I’m unsure why your code snippet would try to call it after you’ve updated the PyTorch version.<br>
Could you check what’s causing the <code>AttributeError</code>?</p>
CUDA out of memory with multiprocessing

username: UCDuan
cooked: <p>Hi all,</p>
<p>I’m using multiprocessing to do inference and it shows CUDA out of memory error. The code is like the following:</p>
<pre><code class="lang-auto">from torch.multiprocessing import Pool, set_start_method

if __name__ == '__main__':

    try:
        set_start_method('spawn')
    except RuntimeError:
        pass

    with Pool(3) as pool:
        pool.map(predict_lat, range(839))
</code></pre>
<p>And the predict_lat function is the following:</p>
<pre><code class="lang-auto">def predict_lat(lat):
    ds = COWholeDataset(ppt=ppt, tmax=tmax, tmin=tmin, tmean=tmean, attributions=attributions, topo_file=topo,
                        lat_id=lat, window_size=180, snow_cover=None)
    dataloader = DataLoader(ds, batch_size=512, shuffle=False)
    pred = []
    print('Sub begin: ', lat)
    for data in dataloader:
        x_d_new, x_attr = data
        x_d_new, x_attr = x_d_new.to(device), x_attr.to(device)
        y_hat = model(x_d_new, x_attr)[0]
        y_hat_sub = y_hat[:, -1:, :]
        pred.append(y_hat_sub.cpu().data.numpy())
        del x_d_new, x_attr, y_hat, y_hat_sub
    pred = np.concatenate(pred).flatten()
    np.save(path + str(lat) + '_' + str(e), pred)
</code></pre>
<p>I’m thinking that the GPU memory should be released when one subprocess finished, and if my GPU can handle three models at one time, it should be fine.<br>
But every time when the first subprocess finished and the fourth begins, it will show the CUDA out of memory error. Does that imply the GPU memory occupied by the first subprocess is not released properly?</p>
<p>Thanks for any ideas and help!</p>
Loss stays nan after end of training

username: cekik
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afde2ee2e95f32d60926d9d0af3668321d90209a.png" alt="pytorch" data-base62-sha1="p5NA7fE5uucr3FctXcaVzNqfWL8" width="617" height="213"></p>

username: ptrblck
cooked: <p>Could you describe the issue and use case a bit more?<br>
I.e. in particular it would be interesting so see, if the loss value was invalid before finishing the training, what loss function you are using etc.</p>

username: cekik
cooked: <p>i figured out . The reason was keeping null values, therefore learning rate was set to zero</p>
How to create a shuffle dataloader from 2 datasets and then shuffling the batches

username: ammaarahmad1999
cooked: <p>I want to create a dataloader with batches containing data from 1 dataset. I have 2 datasets</p>
<p>train_data = TensorDataset(train_sn, train_sm, train_tt, train_y)<br>
scaffold_data = TensorDataset(sca_txt, sca_att, sca_y)</p>
<p>I want to create a single dataloader for these 2 dataset such that a batch contains data from either 1 dataset but not both. Also I want to shuffle the batches among themselves. That is it shouldn’t be like all batches from 1 dataset followed by batches from other dataset while training.</p>
<p>I tried this<br>
dataset_3 = torch.utils.data.ConcatDataset((train_data,scaffold_data))<br>
dataloader = DataLoader(dataset_3, batch_size=16)</p>
<p>But here batches are not shuffled. So it defeats my purpose. My main goal is to train a Multitask NLP Network with both these dataset</p>
Is batch norm accumulation possible in resnet?

username: FaultyBagnose
cooked: <p>Hi,<br>
I would like to train a resnetxxx model on very large images (~3000x3000) in a domain where downscaling loses relevant information. My GPU only has memory enough for a single image at a time, so I look into gradient accumulation, discussed e.g.  <a href="https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient-in-pytorch-i-e-iter-size-in-caffe-prototxt/2522">here</a>. This seems to work fine for the gradient itself, but is it possible to distribute the batch norm calculations over several (micro-)batches also?</p>

username: ptrblck
cooked: <p>No, I don’t think batchnorm layers have this functionality added (in case you have multiple GPUs, you could try <code>SyncBatchNorm</code>). For a single GPU, you could try to change the <code>momentum</code> and see, if you could smooth the running estimates.</p>

username: FaultyBagnose
cooked: <p>Thanks!<br>
I will try to vary the normalization momentum.</p>
<p>Ideally, I would like to reduce the batch size to 1, and then the minibatch standard deviations cannot even be estimated for a single batch, so neither momentum adjustment nor syncing over GPUs is meaningful. This is why I would like to estimate the normalization parameters (mu and sd) over ‘batches of minibatches’.</p>
Is BatchNorm momentum backwards in PyTorch?

username: Alexander_Soare
cooked: <p>I just need to confirm I’m not confusing myself. I’m used to momentum (in an exponential moving average) referring to the weight that is placed on historical values in the time series. Higher momentum means more weight is placed on what has happened, rather than what is happening now. In math-speak if we have a time series S, and we want to produce an exponential moving average series V with momentum \beta</p>
<p><a href="https://i.stack.imgur.com/9BTEA.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/9BTEA.png" alt="enter image description here" width="219" height="32"></a></p>
<p>This is how <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization" rel="noopener nofollow ugc">Keras</a> defines it.</p>
<p>But PyTorch seems to define it the other way around as shown in the note <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="noopener nofollow ugc">here</a>. So in the PyTorch variation, higher “momentum” is the opposite of what we would intuitively think it to be. A momentum of 1 would just reproduce S in V. Right?</p>
<p>If so, I get that the note is there in plain sight. But even then, it should probably say something like “Yes, we know that this seems backwards and is counter to the common meaning of the word “momentum”.”</p>

username: ptrblck
cooked: <p>Yes, I think you are right and I also think the note mentions exactly this, doesn’t it?</p>
<blockquote>
<p>This <code>momentum</code> argument is different from […] the conventional notion of momentum.</p>
</blockquote>

username: Alexander_Soare
cooked: <p>Thanks for confirming! Yes I know it says that and it’s probably enough for most people. Even with that I still went on a hunt because I wasn’t sure that the “different” was what I thought it was.</p>
NLP from Scratch with PyTorch, fastai, and HuggingFace

username: Epoching
cooked: <p>Hey guys! I made a small technical tutorial/blog post on using different libraries for NLP related pipelines! If you’re curious, check it out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p><a href="https://amarsaini.github.io/Epoching-Blog/jupyter/nlp/pytorch/fastai/huggingface/2021/06/27/NLP-from-Scratch-with-PyTorch-FastAI-and-HuggingFace.html" rel="noopener nofollow ugc">NLP from Scratch with PyTorch, fastai, and HuggingFace | Epoching’s Blog (amarsaini.github.io)</a></p>
<p>The post was made using <a href="https://github.com/fastai/fastpages" rel="noopener nofollow ugc">fastpages</a>. It converts your Jupyter Notebooks into blog posts!</p>
<p>Carry on learning,</p>
<p>Epoching</p>

username: ptrblck
cooked: <p>Thanks for sharing! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Install packages

username: Achrouf
cooked: <p>ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Accès refusé: ‘C:\ProgramData\Anaconda3\Lib\site-packages\numpy\conftest.py’<br>
Consider using the <code>--user</code> option or check the permissions.<br>
hello i am trying to run a package i have the following error on often<br>
thank you in advance</p>

username: ptrblck
cooked: <p>It seems you are running into a permission issue. Have you tried to add the suggested <code>--user</code> argument?</p>

username: Achrouf
cooked: <p>Yes it works thank you very much, but now I have the following error:<br>
’<code>load_weights</code> requires h5py when loading weights from HDF5.’)<br>
ImportError: <code>load_weights</code> requires h5py when loading weights from HDF5.<br>
I tried to install h5py but it still doesn’t work,</p>

username: ptrblck
cooked: <p>I don’t know which library depends on <code>h5py</code> and would assume it should work after installing it.<br>
Did you make sure to restart the kernel after installing the missing package?</p>

username: Achrouf
cooked: <p>yes but at the same time I am performing in the terminal</p>
Can data_parallel take multiple input tensors and can it be used during backward() (libtorch)

username: 26sramirez6
cooked: <p>Can torch::nn::parallel::data_parallel support a Module that takes multiple input tensors for its forward() ?</p>
<p>Also does it work for backwards()?</p>

username: 26sramirez6
cooked: <p>bump. Anyone know the answer?</p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/pytorch/pytorch/blob/e3d75b8475a1668a02ac4a23c160b7aee8ebb3d3/torch/csrc/api/include/torch/nn/parallel/data_parallel.h#L246-L285">this code</a> it seems that a single <code>Tensor</code> is required.</p>
Many unknown tiny GPU processes are started with DDP

username: xiaobaishu0097
cooked: <p>I am using DDP for multi-GPU training. However, as demonstrated in the below figure, too many unknown GPU processes are started if I use more than one GPU.<br>
Could anyone provide me with advice on this matter?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/1/71a2426835059192fd5d63b2b5ce978587505fb3.png" data-download-href="https://discuss.pytorch.org/uploads/default/71a2426835059192fd5d63b2b5ce978587505fb3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71a2426835059192fd5d63b2b5ce978587505fb3_2_227x500.png" alt="image" data-base62-sha1="gdfAMKxXTYGXUWm6x8iykGZydZV" width="227" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71a2426835059192fd5d63b2b5ce978587505fb3_2_227x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71a2426835059192fd5d63b2b5ce978587505fb3_2_340x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/1/71a2426835059192fd5d63b2b5ce978587505fb3_2_454x1000.png 2x" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">569×1253 60.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Are you using <code>torch.multiprocessing</code> or any other multiprocessing library in your code?<br>
In a standard use case one process would be used for each device.</p>
PyTorch Installation Problems

username: JadF123
cooked: <p>Hi,</p>
<p>So, I have been trying to install PyTorch using the links given on the website getting started section… i have nvidia and cuda toolkit installed yet i keep getting problems with the installation… i will post the screenshots here<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb7b6aabf97be82b1325767b857bb3b898c1fb5b.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb7b6aabf97be82b1325767b857bb3b898c1fb5b" title="torch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb7b6aabf97be82b1325767b857bb3b898c1fb5b_2_690x68.png" alt="torch" data-base62-sha1="qKxHLSTBErmRjFr5RAHBIhiqVMD" width="690" height="68" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb7b6aabf97be82b1325767b857bb3b898c1fb5b_2_690x68.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb7b6aabf97be82b1325767b857bb3b898c1fb5b_2_1035x102.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb7b6aabf97be82b1325767b857bb3b898c1fb5b_2_1380x136.png 2x" data-dominant-color="807B7B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch</span><span class="informations">1864×186 72.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I also tried installing pytorch audio to find where the problem actually is but this keeps showing up</p>
<p>On another note… can someone explain to me how to dataload the data installed from a kaggle competition… im not quite sure how to do it especially that its like 119 gb and i got 16gb ram… i was told by someone to load the data from different indexes or something and i tried to do some research but havent gotten a clear answer… the kaggle competition im trying to enter is the SIIM Covid 19 competition</p>
<p>Thanks for any kind of help,<br>
Jad</p>

username: ptrblck
cooked: <p>It seems that <code>pip</code> might have trouble finding these packages. Could you try to update <code>pip</code> itself and rerun the install command?</p>
I want to extract features(activations) from the 4 layers of resnet 18. Is my implementation correct?

username: Alankar_Mahajan
cooked: <p>Hi, I want to extract activations from the 4 resnet blocks/layers and use them. Is my code correct?</p>
<p>Resnet modules:</p>
<pre><code class="lang-auto">for num, layer in enumerate(resnet_18._modules.keys()):
    print(num, layer)
</code></pre>
<p>The above code outputs:</p>
<pre><code class="lang-auto">0 conv1
1 bn1
2 relu
3 maxpool
4 layer1
5 layer2
6 layer3
7 layer4
8 avgpool
9 fc
</code></pre>
<p>Custom Model:</p>
<pre><code class="lang-auto">class ResNet18(nn.Module):
    
    def __init__(self, layer_numbers):
        super().__init__()
        self.layer_numbers = layer_numbers # indices of layers whose activations are needed
        self.layer_acts = OrderedDict()
        self.resnet_18 = models.resnet18(pretrained=True)
        self.forward_hooks = []
        
        #register hooks
        for layer_num, layer_name in enumerate(self.resnet_18._modules.keys()):
            if layer_num in layer_numbers:
                self.forward_hooks.append(getattr(self.resnet_18, layer_name).register_forward_hook(self.get_activations(layer_name)))
    
    #Defining hook to get intermediate features     
    def get_activations(self, layer_name):
        def hook(module, input, output):
            self.layer_acts[layer_name] = output
        return hook
    
    #forward pass
    def forward(self, x):
        out = self.resnet_18(x)
        #don't need output values
        return self.layer_acts
</code></pre>
<p>Model Usage:</p>
<pre><code class="lang-auto">required_layers = [4, 5, 6, 7]
model = ResNet18(layer_numbers=required_layers)
model.eval()
</code></pre>
<p>Now, if I pass my image to the model, will I be getting correct layer activations for the 4 layers?</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>The code looks alright and you can check the results by accessing the <code>dict</code> after a forward pass where the activations would be stored.<br>
To compare your approach you would also directly access the desired layers and register the hooks e.g. via <code>model.layer4.register_forward_hook(...)</code>.</p>
How to keep the tensor _version not changed?

username: amyxwgu
cooked: <p>Hi all,<br>
I am updating my model in 2 steps: 1) variable optimization; 2) projection to the optimization constraints. I found the _version of the tensor changed after the projection. So the step 1) optimization on the wrong tensor version. The codes are shown below. Is it possible to keep the tensor _version not changed? Thanks~</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD([x], lr=0.3)
For i in range(iteration):
    optimizer.zero_grad()
    probs = myfunc(x)
    probs.backward()
    optimizer.step() # optimization on x._version = 0

    x = projection(x) # _version number will be increased here
</code></pre>

username: amyxwgu
cooked: <p>A not that elegant way is to move the definition of <code>optimizer</code> inside the For loop. Will this generate thousands of versions of the tensor <code>x</code> in the memory?</p>
<pre><code class="lang-auto">For i in range(iteration):
    optimizer = torch.optim.SGD([x], lr=0.3)
    
    optimizer.zero_grad()
    probs = myfunc(x)
    probs.backward()
    optimizer.step() 

    x = projection(x) # _version number will be increased here
</code></pre>

username: tom
cooked: <aside class="quote no-group" data-username="amyxwgu" data-post="2" data-topic="125135">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/4da419/48.png" class="avatar"> amyxwgu:</div>
<blockquote>
<p>Will this generate thousands of versions of the tensor <code>x</code> in the memory?</p>
</blockquote>
</aside>
<p>Not all at the same time. But with this, the optimizer.step is literally <code>x -= lr * x.grad</code> and the trick will break as soon as you have momentum etc.</p>
<p>I would have to ask: why do you want <code>_version</code> to be unchanged? To my mind, it should be updated – you are wanting to modify <code>x</code> after all. <code>_version</code> exists in order to let autograd detect when it needs something for the backward that has been changed after it has been recorded during the backward (i.e. is invalid for autograd purposes now).</p>
<p>The generic idiomatic thing to do IMO is</p>
<pre><code class="lang-auto">    # your stuff here
    optimizer.step()
    with torch.no_grad():
        x.copy_(projection(x))
</code></pre>
<p>Of course if you change projection to update x in-place, you can skip the <code>copy_</code>.</p>
<p>Best regards</p>
<p>Thomas</p>
Static runtime uses

username: yipyerp
cooked: <p>I’ve seen there is an experimental static runtime ( <a href="https://github.com/pytorch/pytorch/tree/master/torch/csrc/jit/runtime/static" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/csrc/jit/runtime/static at master · pytorch/pytorch · GitHub</a> ), but documentation is somewhat limited. It seems like an optimized runtime for models with particular graph restrictions. What kind of performance gains can be expected? Are there other advantages of using this runtime?</p>