akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why convlution is even faster than torch.sort?

Maxwell_Albert: <p>As far as I concern, sort is O(Nlog(N)) algorithm but convolution is larger than that. But why is convolution so fast at cpu?<br>
Here is the code:</p>
<pre><code class="lang-auto">import torch
from torchvision.models import mobilenet_v2
import time
input = torch.rand([64,3,224,224])
model = mobilenet_v2(pretrained= True).features[0:1]
time_sort_avg = 0.0
time_conv_avg = 0.0
for i in range(50):
    start =time.time()
    output = model(input)
    time_conv = time.time() -start
    output = output.view(-1)
    start =time.time()
    sorted,index = torch.sort(output)
    time_sort = time.time() -start
    time_sort_avg += time_sort
    time_conv_avg += time_conv
print("sort time avg:",time_sort_avg/50)
print("conv time avg:",time_conv_avg/50)
</code></pre>
<p>And here is the result</p>
<pre><code class="lang-auto">sort time avg: 2.178560104370117
conv time avg: 0.07962306022644043
</code></pre>

eqy: <p>There are are going to be many factors in play here, but many of them will boil down to computer  architecture and optimizations. A typical modern CPU is going to have pretty wide execution units (consider 256/512-bit instructions via AVX-256/512), which can get even wider if you consider multithreaded implementations. A library implementation for conv will often have handcrafted implementations to leverage these wide vector extensions to extract as much parallelism as possible as the CPU will support. On the other hand, I believe <code>torch.sort</code> will dispatch to <code>std::sort</code> under the hood, which may not use all of the fancy vector extensions available on modern CPUs and shouldn’t do multithreading. A lot of engineering time has been spent getting conv to go as fast as possible for DL applications.</p>
<p>Another perspective is that convolution is an operation that can have very high arithmetic intensity: read a weight or input once and it may be multiplied and accumulated hundreds of times (e.g., per output channel AND per sliding window). On the other hand if we think about how often an input might be compared/written in sort, it’s log(N). It turns out that this kind of repeated access per the same element can be very cheap on modern hardware given the cache heirarchy.</p>
<p>There’s even more things that I glossed over but they add up in making the “constant factor” in complexity wildly different between something like <code>torch.sort</code> and a highly specialized convolution implementation that you would find in oneDNN/MKLDNN.</p>

Maxwell_Albert: <p>Wow, really specific! Thank you very much.</p>
Weight sharing within one weight tensor (nn.Parameter)

fmarin: <p>Hello all,<br>
I have a weight matrix (for use in a costum model module). In this weight matrix I would like to be able to share weights in a flexible manner (e.g. I provide a list of indices for weights that should be shared).  During the training the weights should then receive the same gradient updates and remain the same throughout. Is this possible and if so what approach should I take?</p>
Prefix parameter names in saved model if trained by multi-GPU?

ming: <p>Hi,</p>
<p>If I train a model with one GPU (without nn.DataParallel), the parameter names in saved models are something like:<br>
features.0.weight</p>
<p>If the model was initialized with nn.DataParallel, the saved parameter names have a prefix:<br>
model.features.0.weight</p>
<p>While doing inference I only use one GPU so the model failed to load the latter model file because the parameter names are not matching. I am wondering why parameter names are prepend the prefix? Can I trim the prefix and still use the model?</p>
<p>Thanks.<br>
Ming</p>

apaszke: <p>Yes, you can just remove the prefix:</p>
<pre><code class="lang-python">state_dict = {k.partition('model.')[2]: v for k,v in state_dict}
</code></pre>

ming: <p>Thanks for the reply! Removing prefix works.</p>
<p>Curious why the prefix is needed? It creates inconvenience when we want to resume training a single-GPU-trained-model with multi-GPU, or pass a multi-GPU trained model to inference code that only uses one GPU?</p>
<p>Also it looks the pretrained resnet doesn’t have ‘module.’ prefix, does it mean that they were trained on single GPU?</p>

apaszke: <p>It’s needed because that’s how state_dicts work <img src="//discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"> You recursively go over the network, prepending the names. But maybe it’s a good idea to override that for DataParallel.</p>
<p>No, they probably had the prefixes trimmed before serialization.</p>

lunasdejavu: <p>Hi I met the same problem but<br>
after I changed the code like this:</p>
<pre><code class="lang-auto">state_dict =checkpoint['model_dict']
state_dict = {k.partition('model.')[2]: v for k,v in state_dict}
model.load_state_dict(state_dict)
</code></pre>
<p>it showed this error:<br>
<code>ValueError: too many values to unpack (expected 2)</code><br>
why it didn’t work?</p>

Lynx_Commando: <aside class="quote no-group" data-username="apaszke" data-post="2" data-topic="494">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar"> apaszke:</div>
<blockquote>
<pre><code class="lang-auto">state_dict = {k.partition('model.')[2]: v for k,v in state_dict}
</code></pre>
</blockquote>
</aside>
<p>{k.partition(‘module.’)[2]:state_dict[k]  for k in state_dict.keys()}</p>

pepeportolo: <p>I know this thread is a little bit old, but I just run into the same problem. From <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html:" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/saving_loading_models.html:</a></p>
<p>"<code>torch.nn.DataParallel</code> is a model wrapper that enables parallel GPU utilization. To save a <code>DataParallel</code> model generically, save the <code>model.module.state_dict()</code> . This way, you have the flexibility to load the model any way you want to any device you want."</p>
<p>Regards<br>
Pepe</p>
Custom Imagenet Initialization For Group Convolution

ufukpage: <p>Hi everyone!</p>
<p>I have got a question about initialization of 2D group convolutions. I have been trying to initialize custom ResNet encoder having group convolutions with pretrained weights. However, these pretrained weights belong to normal convolution layers, not to group ones. Obviously, there is dimension mismatch between group convolution weights and pretrained weights. I guess there is trick squeezing group weights into single tensor, mentioned in (<a href="https://github.com/pytorch/pytorch/issues/23854" class="inline-onebox" rel="noopener nofollow ugc">Fan out calculation broken for group (depthwise) convolution · Issue #23854 · pytorch/pytorch · GitHub</a>), In my case, I need to access each group kernel to initialize separately. In short, I need to convert pretrained normal convolution layers to group convolution layers with the same weights. How am I supposed to do this kind of initialization?</p>
<p>Thank you.</p>
Casting a model dict to a different dataype

Amur_Saqib_Pal: <p>Hello. I have an exported model in .pth format. It is loaded in an Ordered Dictionary format, from which I would like to separate the numeric tensors (not the key values i.e. strings) and cast them to a different datatype, say int. However, upon calling dict.values(), I get a huge output, an excerpt of which is given below.</p>
<pre><code class="lang-auto">[-7.6345e-03, -4.4007e-03, -7.9214e-03,  ..., -8.5340e-04,
           6.4599e-04, -5.3076e-04]], device='cuda:0')}, 128: {'momentum_buffer': tensor([ 0.0129, -0.0025, -0.0054, -0.0146, -0.0153,  0.0049,  0.0010, -0.0075,
          0.0060, -0.0038,  0.0023,  0.0005, -0.0026,  0.0036, -0.0013,  0.0051,
          0.0073,  0.0064, -0.0050,  0.0048,  0.0024, -0.0002,  0.0051,  0.0052,
         -0.0051,  0.0013, -0.0116,  0.0018,  0.0002,  0.0051, -0.0054,  0.0053,
         -0.0014,  0.0086, -0.0128,  0.0101, -0.0145,  0.0103,  0.0042, -0.0055],
        device='cuda:0')}}
</code></pre>
<p>I want to extract only the numeric values from this. How can I do that?</p>
Could not run 'aten::zero_' with arguments from the 'SparseCsrCPU' backend. 

Piyush_Paliwal: <p>When applying backward method on loss <code>loss.backward()</code>, I get the error:</p>
<pre><code class="lang-auto">--&gt; 363 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)

File ~/miniconda3/lib/python3.9/site-packages/torch/autograd/__init__.py:173, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    168     retain_graph = create_graph
    170 # The reason we repeat same the comment below is that
    171 # some Python versions print out the first line of a multi-line function
    172 # calls in the traceback and some print out the last line
--&gt; 173 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    174     tensors, grad_tensors_, retain_graph, create_graph, inputs,
    175     allow_unreachable=True, accumulate_grad=True)

NotImplementedError: Could not run 'aten::zero_' with arguments from the 'SparseCsrCPU' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::zero_' is only available for these backends: [CPU, Meta, MkldnnCPU, SparseCPU, BackendSelect, Python, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, AutocastCPU, Autocast, Batched, VmapMode, Functionalize].

</code></pre>
<p>I understand that fed input data was not dense numpy array but sparse csr <code>torch.sparse_csr_tensor(X.indptr, X.indices, X.data, X.shape, dtype=torch.float32)</code>. Is this is an issue? Everything works except <code>loss.backward()</code>.</p>

ptrblck: <p>Could you post a minimal, executable code snippet which would reproduce this issue, please?<br>
I’m unsure if it’s a known limitation or if it’s a bug (and e.g. internally the tensor should be dense in the backward pass).</p>
Extract layer output from CNN and use it as input for LSTM

natalie-sch: <p>Hi, I would like to use the output of a CNN model (the output from the layer before softmax, for example), replace the remaining layers with LSTM and train the added LSTM part.<br>
I have read a couple of articles around this topic (such as <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254" class="inline-onebox">How can l load my best model as a feature extractor/evaluator?</a>) but unfortunately still struggle to fully understand it.</p>
<p>The output of the CNN model is loaded</p>
<pre><code class="lang-auto">model = ShallowFBCSPNet(
    in_chans = 21,
    n_classes = 2,
    input_window_samples=1000,
    final_conv_length=25, 
)
model.load_state_dict(torch.load('SavedModels/cnnshallow.pth'))
model.eval()
</code></pre>
<p>returning</p>
<pre><code class="lang-auto">ShallowFBCSPNet(
  (ensuredims): Ensure4d()
  (dimshuffle): Expression(expression=transpose_time_to_spat) 
  (conv_time): Conv2d(1, 40, kernel_size=(25, 1), stride=(1, 1))
  (conv_spat): Conv2d(40, 40, kernel_size=(1, 21), stride=(1, 1), bias=False)
  (bnorm): BatchNorm2d(40, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv_nonlin_exp): Expression(expression=square) 
  (pool): AvgPool2d(kernel_size=(75, 1), stride=(15, 1), padding=0)
  (pool_nonlin_exp): Expression(expression=safe_log) 
  (drop): Dropout(p=0.5, inplace=False)
  (conv_classifier): Conv2d(40, 2, kernel_size=(25, 1), stride=(1, 1))
  (softmax): LogSoftmax(dim=1)
  (squeeze): Expression(expression=squeeze_final_output) 
)
</code></pre>
<p>I registered a forward hook</p>
<pre><code class="lang-auto">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

model.conv_classifier.register_forward_hook(get_activation('conv_classifier'))
x = torch.randn(1, 25)
output = model(x)
print(activation['conv_classifier'])
</code></pre>
<p>I would like to freeze the CNN weights, reshape the tensors and put the following LSTM model on top.</p>
<pre><code class="lang-auto">class LSTMModel(torch.nn.Module):
  def __init__(self):
    super(LSTMModel, self).__init__()
    self.lstm = torch.nn.LSTM(input_size=6000, hidden_size=21, num_layers=1, batch_first=True)
    self.fc = torch.nn.Linear(21, 1)

  def forward(self, x):
    batches = x.size(0)
    h0 = torch.zeros([1, batches, 21])
    c0 = torch.zeros([1, batches, 21])
    (x, _) = self.lstm(x, (h0, c0))
    x = torch.nn.functional.relu(x)
    x = self.fc(x)
    return x

model = LSTMModel()
criterion = torch.nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters())
</code></pre>
<p>Any help would be greatly appreciated!</p>
GPU usate decreases massively during the first epoch

Valerio_Biscione: <p>Hello. I am training a fairly simple supervised model: a conv net that generates an embedding fed into an lstm.<br>
At the beginning of training, for around halfthe first epoch, I manage to do around 40 batches a second, and GPU utilization is at around 20%. However, this number slowly decreases, and after a couple of minutes I am at around 7% of GPU util. and doing around 2-3 batches a second. It sometime goes up again for 30 seconds on so, but inevitably comes back down. The GPU memory does NOT go up during this period.</p>
<p><strong>Does anyone have any idea about that?</strong></p>
<p>-More Info:<br>
Since I suspect it might be due to the nature of the dataset, here some more detail about it. I am using a custom DataLoader. Each training sample is composed of 5 elements, say <code>(seq, array1, array2, label, image)</code>, where seq is a list of 6 PIL images 32x32. I first generate the dataset, and then save it into a pickle file. At train time, I load the pickle object, and put each element into a list.<br>
The <code>__getitem__</code> function is like this:</p>
<pre><code class="lang-auto">def __getitem__(self, idx)
      return torch.cat([self.transform(c).unsqueeze(0) for c in self.seq[idx]]), # (6, 3, 32, 32)
               torch.tensor(self.array1[idx]).float(), \ # (6, 2)
               torch.tensor(self.array2[idx]).float(),  \  # (11)
               torch.tensor(self.label[idx]).long(), \   # (0)
               self.transform(self.images[idx])   #  (3, 128, 128)
</code></pre>
<p>More Info:<br>
maybe this is not really directly related to the training setup. If I stop training and restart it again straight away, the GPU is slow EVEN at the very beginning. Maybe the GPU is overheating? But the <code>nvidia-smi</code> shows a temperature of only 53C which seems acceptable…</p>

ptrblck: <aside class="quote no-group" data-username="Valerio_Biscione" data-post="1" data-topic="150660">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/valerio_biscione/48/22759_2.png" class="avatar"> Valerio_Biscione:</div>
<blockquote>
<p>If I stop training and restart it again straight away, the GPU is slow EVEN at the very beginning.</p>
</blockquote>
</aside>
<p>This sounds indeed as if your setup is running into a potential heating issue and is downclocking the hardware. Try to measure the clocks of the GPU, CPU, etc. and see if these are decreased aver time.</p>

Valerio_Biscione: <p>Thanks, I’ll try that. Just to make sure it isn’t something with my code I tried on a different machine, and there is no slow down.</p>
ObjectDetection model pretrained download fails

markytools: <p>Downloading: “<a href="https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth</a>” to /home/uclpc1/.cache/torch/hub/checkpoints/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth</p>
<p>Pretrained object detection model downloading loading for very long.<br>
I tried</p>
<blockquote>
<p>wget <a href="https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth</a></p>
</blockquote>
<p>and it doesn’t download. Kindly fix this bug pytorch.</p>

ptrblck: <p>I cannot reproduce the issue and get properly download the model in less than a minute.<br>
Maybe the server had some issues when you’ve tried it?</p>

markytools: <p>I don’t know why but for some reason I do,</p>
<p>wget <a href="https://download.pytorch.org/models/fasterrcnn_mobilenet_v3_large_320_fpn-907ea3f9.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/fasterrcnn_mobilenet_v3_large_320_fpn-907ea3f9.pth</a></p>
<p>it doesn’t work, but when I do curl:</p>
<p>curl -O <a href="https://download.pytorch.org/models/fasterrcnn_mobilenet_v3_large_320_fpn-907ea3f9.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/fasterrcnn_mobilenet_v3_large_320_fpn-907ea3f9.pth</a></p>
<p>it works.<br>
Maybe mind replacing pytorch pretrained model downloading with curl? Must be a linux thing.</p>

ptrblck: <p>I’m on Ubuntu 20.04 so I don’t think it’s a “Linux thing” <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Imagenet training__learning curve

nikiguo93: <p>Hi I trained the resnet50 from scratch and used images from Imagenet dataset.Then I got the learning curve as follows. For the val loss, it first reached around 0.004 and finally fluctuate around 0.009. Is there any problem with my learning curve?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6f751fe4f4b839ef9c760bd53df30dda48813ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/b6f751fe4f4b839ef9c760bd53df30dda48813ee" title="1651566891(1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6f751fe4f4b839ef9c760bd53df30dda48813ee_2_690x247.png" alt="1651566891(1)" data-base62-sha1="q6AMpKtKnVQIKhSVaHGnaYVw7QG" width="690" height="247" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6f751fe4f4b839ef9c760bd53df30dda48813ee_2_690x247.png, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6f751fe4f4b839ef9c760bd53df30dda48813ee.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6f751fe4f4b839ef9c760bd53df30dda48813ee.png 2x" data-dominant-color="FAF9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1651566891(1)</span><span class="informations">843×302 22.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Scriptable vs compose transforms

aleemsidra: <p>What are scriptable <code>transofrms</code> in PyTorch? How are they different from <code>transform.Compose</code>? For both of them, multiple transforms can be grouped together. Scriptable ones are for tensors only. Other than that, how are they different?</p>

snehangsude: <p>Been trying to find a better explanation of this. Haven’t been able to find any article of post wrt this. Have you found any answers?</p>
To receive input values of different sizes, and to randomly select layers

Hyeongu_Oh: <p>I hope you understand my lack of English.</p>
<pre><code class="lang-auto">    self.list5 = [1,2,3,4,5]
    self.floor1list = nn.ModuleList([self.fc1, self.fc2, self.fc3, self.fc4, self.fc5])


def forward(self, x, input_num):
    select = random.sample(self.list5, input_num.item())

    x1 = x[:, :2700]
        
    for j,i in zip(range(input_num), select):
        x1 += self.leaky(self.floor1list[i-1](torch.cat((x[:,:2700], x[:,2700+5400*j:2700+5400*(j+1)]))))
    x1 = self.sigmoid(x1)
    return x1

---&gt; 48         x1 = x[:, :2700]
TypeError: unhashable type: 'slice'

---&gt; 45         select = random.sample(self.list5, input_num.item())
ValueError: only one element tensors can be converted to Python scalars

---&gt; 50         for j,i in zip(range(input_num), [1,2,3,4,5]):
TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p>batch size = 8<br>
x size = 8x32400<br>
input_num size = 8x1</p>
<p>Force different sizes of inputs to the same size, So I get input_num.<br>
I’d like to select random layers according to input_num.<br>
Errors occur on almost all lines <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:">  <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Is there a better way?</p>

markfilan: <aside class="quote no-group" data-username="Hyeongu_Oh" data-post="1" data-topic="81881">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hyeongu_oh/48/14475_2.png" class="avatar"> Hyeongu_Oh:</div>
<blockquote>
<p><code>TypeError: unhashable type: 'slice'</code></p>
</blockquote>
</aside>
<p>A slice is a subset of a sequence such as a string, a list , or a tuple . Values in a Python dictionary cannot be sliced like a list. This is because dictionaries can have custom key values. They are not indexed from zero. If you try to <a href="http://net-informations.com/python/pro/slice.htm" rel="noopener nofollow ugc">slice</a> a dictionary as if it were a list, you’ll encounter the “TypeError: unhashable type: ‘slice’” error. You must directly specify what values you want to access from our dictionary. To do this, refer to the appropriate key names in the dictionary. The other option you have is using a for loop with zip , and range functions. The zip() function will help you to zip the dictionary keys with the range iterator. And the range() function will help you to specify the number of elements you want to access from the dictionary.</p>
Weighted sample losses

sesale: <p>I am working on a project where I have to calculate the losses for each individual sample t. These losses are multiplied by an individual sample weight lambda. I would now like to calculate the gradients for each individual loss before multiplying them with the given sample weight. However, I am not sure about the dimensions. For one sample loss, calculating the gradients, in my case, results in 14 tensors with individual length. Should I just multiply each individual scalar in the resulting gradients with the respective weight?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c05ff8dda3b2090489960412ce51a6bf9df71c0f.png" alt="Screenshot from 2022-05-02 15-06-00" data-base62-sha1="rrPicRVENSyGWoqlE5YVH7d2MBV" width="249" height="60"></p>

InnovArul: <p>I think you do not need to deal with gradient calculations.<br>
If you multiply the <code>lambda</code>s  to the respective individual losses and sum/average it up, the gradient calculation will be handled accordingly by PyTorch.</p>
<p>Similar to the below code:</p>
<pre><code class="lang-auto">criterion = torch.nn.MSELoss(reduction='none')
...

# lambda = (b,), pred = (b, d), target = (b, d)
loss = lambda * criterion(pred, target)
loss = loss.mean()  # or sum()
loss.backward()
</code></pre>

sesale: <p>Thanks for your answer! But wouldn’t that result in taking the gradients of the weighted losses? Since the equation asked for the gradients of the losses which are then weighted by lambda. Does it even matter whether I first multiply by the weights and then calculate the gradients or the other way around?</p>

InnovArul: <aside class="quote no-group" data-username="sesale" data-post="3" data-topic="150588">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/cdc98d/48.png" class="avatar"> sesale:</div>
<blockquote>
<p>Does it even matter whether I first multiply by the weights and then calculate the gradients or the other way around?</p>
</blockquote>
</aside>
<p>No. It doesn’t matter whether you multiply the loss by <code>lambda</code>s (weighted loss) or calculate the gradients first &amp; then multiply the <code>lambda</code>s.</p>
<aside class="quote no-group" data-username="sesale" data-post="3" data-topic="150588">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/cdc98d/48.png" class="avatar"> sesale:</div>
<blockquote>
<p>wouldn’t that result in taking the gradients of the weighted losses?</p>
</blockquote>
</aside>
<p>As I understand, <code>lambda</code>s are constant here. Taking the gradient of weighted loss is same as multiplying the <code>lambda</code>s after taking the gradient.</p>
How to compute Cross Entropy Loss for sequences

Litlon: <p>I have a sequence continuation/prediction task (input: a sequence of class indices, output: a sequence of class indices) and I use Pytorch.</p>
<p>My neural network returns a tensor of shape (batch_size, sequence_length, numb_classes) where the entries are a number proportional to the propability that the class with this index is the next class in the sequence. My targets in the training data are of shape (batch_size, sequence_length) (just the sequences of the real predictions).</p>
<p>I want to use the CrossEntropyLoss</p>
<p>My question: How do I use the Cross Entropy Loss function? Which input shapes are required?</p>
<p>Thank you!</p>

Nisan_Aryal: <p>i did this once with the lstm model. The approach is similar to that of any cross entropy loss approach. Give the predicted as (batch_size/sequence length, num_classes) and the target is (sequence length/ gt class for each step in one dimensional)…</p>
Dynamic Inference with Channel indexing is very slow

Scott_Hoang: <p>Hi guys,<br>
I don’t know whether anyone has encountered this before, but in an attempt to boost inference speed, I’m using channel indexing for weight selection for dynamic inference following this example:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/JiahuiYu/slimmable_networks">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/JiahuiYu/slimmable_networks" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/b61a8de6c487559c8555ea78bc724d2d8f1f7d3ae24334481ddcf021f094eca2/JiahuiYu/slimmable_networks" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/JiahuiYu/slimmable_networks" target="_blank" rel="noopener nofollow ugc">GitHub - JiahuiYu/slimmable_networks: Slimmable Networks, AutoSlim, and...</a></h3>

  <p>Slimmable Networks, AutoSlim, and Beyond, ICLR 2019, and ICCV 2019 - GitHub - JiahuiYu/slimmable_networks: Slimmable Networks, AutoSlim, and Beyond, ICLR 2019, and ICCV 2019</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
However, the inference speed of a Resnet18 with only 0.25x width measured on a mobile device is worst than the full vanilla model<br>
Any thought?</p>
Torchaic tensor logical 'and' reduce syntax

eleven: <p>What is the “torchaic” way of reducing a tensor with a logical and?</p>
<pre><code class="lang-auto">t = torch.tensor([True,True,True])
f = torch.tensor([True,False,True])

# if t:
    # this is the ideal syntax I want but it
    # throws RuntimeError: Boolean value of Tensor with more than one value is ambiguous

# if torch.and(t):
    # this could also be the ideal syntax I want but it
    # throws SyntaxError: invalid syntax

# this works, but do I really have to write my own function?        
def is_all_true(t):
    r = torch.tensor([True])
    for i in range(len(t)):
        r = torch.bitwise_and(r, t[i])
    return r

if is_all_true(t):
    print("t is all True") # outputs "t is all True"
if not is_all_true(f):
    print("f is not all True") # outputs "f is not all True"
</code></pre>

eleven: <p>“Torchaic” way:</p>
<pre><code class="lang-auto">torch.all(t) # outputs tensor(True)
torch.all(f) # outputs tensor(False)
</code></pre>
How to efficiently create sparse band matrices?

hasco641: <p>Hi all,</p>
<p>I want to construct the precision matrix for a model that has an AR(1)-like structure. This means that the precision matrix is  a sparse band-like matrix and if you were using R, you would use functionality such as <code>bandSparse</code> from the <code>Matrix</code> library.</p>
<p>The matrix should be of the form  (sorry, can’t find any LaTeX support):<br>
$$<br>
\begin{bmatrix}<br>
1 &amp; -\alpha &amp;  &amp;  &amp;  \<br>
-\alpha &amp; 1+\alpha^2 &amp; -\alpha &amp;  &amp;  \<br>
&amp;  &amp; \ddots &amp;  &amp;  \<br>
&amp;  &amp; -\alpha &amp; 1+\alpha^2 &amp; -\alpha \<br>
&amp;  &amp;  &amp; -\alpha &amp; 1<br>
\end{bmatrix}<br>
$$</p>
<p>Below is working code, but as I am sure that you can tell, won’t be the most direct way of creating this matrix. Any thoughts on how to do it better would be appreciated. Thanks!</p>
<pre><code class="lang-auto">N = 100
alpha = 0.6

dependence = torch.cat([torch.tensor([1., -alpha, 0]).reshape(1, 3),
                        torch.tensor([-alpha, 1+alpha**2, -alpha]).reshape(1, 3).expand(N-2, 3),
                        torch.tensor([0., -alpha, 1.]).reshape(1, 3)], 0)

band_matrix = torch.zeros(N, N)

total_pad = N - 3
band_matrix[0] = torch.cat((dependence[0], torch.zeros(total_pad)), 0)
for i in range(1, N-1):
    left_pad = i - 1
    right_pad = total_pad - left_pad
    band_matrix[i] = torch.cat((torch.zeros(left_pad), 
                                dependence[i], 
                                torch.zeros(right_pad)), 0)
band_matrix[N-1] = torch.cat((torch.zeros(total_pad), dependence[N-1]), 0)

</code></pre>
How to locate UserWarning:

ZhengdiYu: <p>I’m getting this warning:</p>
<p>/usr/local/lib/python3.8/site-packages/torch/nn/functional.py:3679: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.</p>
<p>However, I’ve already confirmed that all of my code is using F.interpolate(input, scale_factor=(x, x)), x is just a number. I don’t know where the warning came from. Could you give me a way to better locate the issue?</p>
Loss becomes 'nan' after some iterations

Hrant_Baloyan: <p>I am trying to build Autoencoder whose encoder,decoder are nested TreeLSTM-s. During training after some iterations loss becomes ‘nan’. I did try to decrease learning rate, do gradient clapping,data normalization but still it becomes ‘nan’. What can be wrong?</p>
<p>RuntimeError: Function ‘MulBackward0’ returned nan values in its 0th output. This is error message.</p>

AbdulsalamBande: <p>I suspect some your labels or training data might contain Nans. If your data is in a csv file.<br>
<strong>Remove Nans before training</strong></p>
<pre><code class="lang-auto">data = data.dropna()
</code></pre>

Hrant_Baloyan: <p>My data is in this format <a href="https://github.com/unbounce/pytorch-tree-lstm/blob/master/example_usage.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-tree-lstm/example_usage.py at master · unbounce/pytorch-tree-lstm · GitHub</a> look tree in this code. How can I check if it contain Nans here?</p>

ksmdanl: <p>Given the following code snippet,</p>
<pre><code class="lang-auto">h, c = model(
            data['features'],
            data['node_order'],
            data['adjacency_list'],
            data['edge_order']
        )
</code></pre>
<p>I suppose you can iterate through the dataset <code>data['features']</code> to inspect which features contribute to NaNs value and print them out.</p>

Hrant_Baloyan: <p>I did but there is no nan element in data[‘features’]</p>

ksmdanl: <p>That’s okay. Did you try for debugging purpose to include a regularisation layer or term?</p>

Hrant_Baloyan: <p>I didn’t. Are there any other reasons for nan output?</p>

ksmdanl: <p>Depending on your model or your training setup it can be few reasons.<br>
If your model is deep enough it might suffer from vanishing gradients, if you don’t apply regularisation, especially if you’re using sigmoid/tanh as activations.<br>
It can also be that you’re learning rate was too high and the gradients became infinity hence your model diverged.<br>
These are typical scenarios, few tricks should help, like choosing a smaller learning rate or apply regularisation.</p>

Hrant_Baloyan: <p>My training set is 3000 style and content tensors. When I make my training set 100  and then  train my model only on that 100 tensors I don’t get ‘nan’ in my loss. But increasing training set I get. Can  there  be other problem going on?</p>
Pytorch3D: Comparing/Evaluting two meshes

aswamy: <p>I have a generated mesh and ground truth mesh and I would like to compare the two meshes for accuracy and completeness. Are there any libraries to compute accuracy and completeness either in pytorch3d or in any 3D libraries?</p>
<p>Thank you!</p>

Laurent1: <p>Check out <a href="https://pytorch3d.org/" rel="noopener nofollow ugc">PyTorch3D</a> that implements the Chamfer distance.</p>
Autograd matrix to matrix function

ejg: <p>Hi!</p>
<p>I hope I’m in the right place to ask this question. I’m new to actually caring about how autograd works, so I’m trying to understand how I can define a new autograd function in the case where I map a matrix to a scalar using intermediate matrix transformations. I’m mainly wondering what mathematically is going on. I’m looking at <a href="https://pytorch.org/tutorials/beginner/examples_autograd/polynomial_custom_function.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch: Defining New autograd Functions — PyTorch Tutorials 1.11.0+cu102 documentation</a>.</p>
<p>I realize I  can using autograd calculate the gradient of something like <code>energy</code> below,</p>
<pre><code class="lang-auto">def mat2mat_function(matrix):
  return(matrix@ matrix)

def energy(matrix):
  return(torch.trace(matrix_sq.T @ matrix_sq)) 
</code></pre>
<p>My intermediate function <code> mat2mat_function</code> in the above example I input a matrix and am returned is square. It is not clear to me what is going on auto-grad wise.  What is meant by applying the chain rule to a matrix-to-matrix function? In this case, I can of course guess that I mean that I multiply the current gradient with 2X, as the function is X^2. But what if my matrix-to-matrix function is much more complicated? I looked into Gateux derivatives, but they give operators between the spaces and not directly gradients. So what is going on, and how would I implement my own custom matrix-to-matrix autograd function?</p>
<p>I can, following the example, define a class</p>
<pre><code class="lang-auto">class matrix_function(torch.autograd.Function):
@staticmethod
    def forward(ctx, input):
      
        ctx.save_for_backward(input)
        return input @ input 

    @staticmethod
    def backward(ctx, grad_output):
        
        input, = ctx.saved_tensors
        return grad_output * 2*input #correct ??????? 

</code></pre>
<p>If the Gateaux derivative of my intermediate matrix function was some much more complicated thing, which only when evalualted would return a matrix, what do I do then? Imagine that I’m for instance solving an equation <code>A(X) = Y</code>, and am inputting <code>X</code> to obtain <code>Y</code>?</p>

tom: <ol>
<li>
<p>Your function only works for square matrices, I don’t know if that is on purpose.</p>
</li>
<li>
<p>Autograd in the usual backward AD mode computes a gradient or more generally a vector jacobian product. The grad_output is the vector and the backward should return this grad output matrix-multiplied with the jacobian of the function. I would expect the gradient to be something like <code>grad_out @ input.T + input.T @ grad_out</code>.</p>
</li>
<li>
<p>You can check with <code>torch.autograd.gradcheck(matrix_function.apply, torch.randn(5, 5, requires_grad=True, dtype=torch.double))</code> or so.</p>
</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>
[DataParallel in Window Machine] PyTorch is not compiled with NCCL support warnings.warn('PyTorch is not compiled with NCCL support')

C_J: <p>I’m using DataParallel in my window 10 machine, with pytorch version of 1.10.<br>
Although the code works, I get this message:</p>
<p>UserWarning: PyTorch is not compiled with NCCL support<br>
warnings.warn(‘PyTorch is not compiled with NCCL support’)</p>
<p>I learned that window doesn’t support, NCCL, but I’m not sure what does it imply.</p>
<p>A. Doesn’t really affect the performance.<br>
B. It works, but it’s not efficient on window machine (maybe slow, or memory inefficient?)<br>
C. It’s not actually doing the task in parallel. (e.g. work in sequential, gpu0-&gt;gpu1-&gt;gpu2…)</p>
<p>Depending on the answer, is there a recommended solution or getaround?</p>
<p>Thanks for reading!</p>

kwen2501: <p>Hi, you may refer to the Note <a href="https://pytorch.org/docs/stable/distributed.html#backends-that-come-with-pytorch" rel="noopener nofollow ugc">here</a>:</p>
<blockquote>
<p>As of PyTorch v1.8, Windows supports all collective communications backend but NCCL.</p>
</blockquote>
<p>Hence I believe you can still have torch.distributed working, just without the performance brought by NCCL.</p>
<p>If in doubt, you can use the <code>torch.distributed.is_available()</code> <a href="https://torch.distributed.is_available()" rel="noopener nofollow ugc">API</a> to see if it works on your platform.</p>

C_J: <p>I’m not using distributed, I’m just using DataParallel and I get this message.<br>
For example in the code, I do not ask to use “nccl”</p>

cbalioglu: <p>Have you tried using v1.11? That warning message won’t have any performance impact on your training, but I can’t find any place in our latest code base where we call <code>nccl.is_available()</code> in DataParallel (which ultimately outputs that warning).</p>
What's the fastest setting for distributed training, and why is it?

chenglu: <p>I’ve seen that for the <code>torch.distributed.init_process_group</code>, we can set up the process communication by <code>tcp</code> or <code>file</code>, by giving right argument to <code>init_method</code>. So I wonder how to set this for the fast performance and why.</p>
<p>Maybe we can discuss on the following cases:</p>
<ul>
<li>for single node, multi process(multi GPUs) . Use <code>tcp://127.0.01:$port</code> vs <code>file:///tmp/somefile</code>
</li>
<li>for multi node. Use <code>tcp://$ip:$port</code> vs <code>file:///share/nfs/somefile</code>
</li>
</ul>
<p>Thanks.</p>

cbalioglu: <p>Unless you have a very specific reason, I would always suggest using TCPStore which is much more reliable than FileStore. Having said that our stores are mostly used for setting up your distributed training environment. Once all ranks have established their communication via a CCL, stores play very little role in the performance of your training. So it is much more important to specify the correct parameters for your underlying CCL (e.g. NCCL, Gloo) than the type of store.</p>
Updating Adam optimizer after changing network structure during training

PeaBrane: <p>I have a use case where I am required to change the network architecture in the middle of training, while using the Adam optimizer.</p>
<p>For example, let’s say I am training a network first for CIFAR-10, where the last layer is an <code>self.fc = nn.Linear(1024, 10)</code>. And after the test accuracy reaches a certain threshold (say 70%), I switch to the CIFAR-100 dataset, and I replace the last layer with <code>self.fc = nn.Linear(1024, 100)</code>.</p>
<p>My question is, if I am using the Adam optimizer where the running mean of the parameter gradients are kept, what is the best way for me to update the optimizer such that the running mean for the previously layers are kept.</p>
<p>In other words, if I simply re-run <code>optim.Adam(model.parameters())</code>, will the running mean of all layers be reset as well?</p>
Torch.backends.quantized.supported_engines is different on macOS virtual env

binoua: <p>Hello everyone,</p>
<p>I am trying to use <code>torch</code> on a GitHub macOS virtual environment (<a href="https://github.com/actions/virtual-environments" class="inline-onebox" rel="noopener nofollow ugc">GitHub - actions/virtual-environments: GitHub Actions virtual environments</a>). However, I see something which is not like my native macOS (11.6, intel).</p>
<p>Basically, I run this <code>https://github.com/bcm-at-zama/gha-tests/runs/6261701379?check_suite_focus=true</code> GitHub action, ie <code>python3 -c "import torch; print(torch.backends.quantized.supported_engines)"</code><br>
`. It says:</p>
<pre><code class="lang-python">`['qnnpack', 'none']`
</code></pre>
<p>If I do the same <code>python3 -c "import torch; print(torch.backends.quantized.supported_engines)"</code> on my macBookPro, I get a:</p>
<pre><code class="lang-auto">['qnnpack', 'none', 'fbgemm']
</code></pre>
<p>Why do I get such a difference please? Notably, I would like to use <code>torch.quantization.quantize_dynamic</code> and I have the impression that it needs <code>fbgemm</code>.</p>
<p>Remark that I have torch  1.11.0 on both machines / virtual env.</p>
<p>Thanks a lot<br>
(and congrats for what you’re doing with torch, this is amazing)</p>
Custom CUDA operator only work well on cuda:0

shzygmyx: <p>I’m learning how to write custom cuda operator by your tutorial at <a href="https://github.com/pytorch/extension-cpp" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/extension-cpp: C++ extensions in PyTorch</a></p>
<p>However, I found the operator only output correct results on “cuda:0”, and output wrong aresults(all_zeros tensor) in other devices like “cuda:1”</p>
<p>Is there anyway to make the cuda operator work well on<br>
“cuda:1”?</p>

eqy: <p>That sounds surprising; could you share some more details about the setup e.g., are <code>cuda:0</code> and <code>cuda:1</code> identical devices? What happens if <code>cuda:1</code> used as <code>cuda:0</code> e.g., with <code>CUDA_VISIBLE_DEVICES=1</code>?</p>

shzygmyx: <p>Interestingly, two methods that set cuda device lead to different result:<br>
method 1) : set <code>CUDA_VISIBLE_DEVICES=1</code> before running code <code>x=torch.LongTensor([1,2,3]) ...</code><br>
method 2): do not set environment variable and run code <code>x=x.to("cuda:1")</code></p>
<p>method 1 runs perfectly, while method 2 fails.</p>

ptrblck: <p>It sound as if your custom extension might be missing the <code>deviceGuard</code> usage?</p>

shzygmyx: <p>Thanks for your suggestion! Could you please provide more details about how to add <code>deviceGuard</code>? Since I did not found anything related to this method in official tutorial, I guess this could be very helpful to point that out in a new version of tutorial <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>In your custom extension you would have to add:</p>
<pre><code class="lang-auto">#include &lt;c10/cuda/CUDAGuard.h&gt;

const at::cuda::OptionalCUDAGuard device_guard(device_of(local_tensor));
/* 
your code
*/
</code></pre>

shzygmyx: <p>It works! Thank you so much!</p>
NVIDIA NVIDIA RTX A5000 with CUDA capability sm_86 is not compatible with the current PyTorch installation

jiansong_zhang1: <p>I set up my PyTorch environment following the ’ [environment_gpu.yml’ file from this Github Project <a href="https://github.com/graphdeeplearning/graphtransformer" rel="noopener nofollow ugc">https://github.com/graphdeeplearning/graphtransformer</a>, which goes like this:<br>
name: graph_transformer<br>
channels:</p>
<ul>
<li>pytorch</li>
<li>dglteam</li>
<li>conda-forge</li>
<li>fragcolor</li>
<li>anaconda</li>
<li>defaults<br>
dependencies:</li>
<li>cudatoolkit=10.2</li>
<li>cudnn=7.6.5</li>
<li>python=3.7.4</li>
<li>python-dateutil=2.8.0</li>
<li>pip=19.2.3</li>
<li>pytorch=1.6.0</li>
<li>torchvision==0.7.0</li>
<li>pillow==6.1</li>
<li>dgl-cuda10.2=0.5.2</li>
<li>numpy=1.16.4</li>
<li>matplotlib=3.1.0</li>
<li>tensorboard=1.14.0</li>
<li>tensorboardx=1.8</li>
<li>future=0.18.2</li>
<li>absl-py</li>
<li>networkx=2.3</li>
<li>scikit-learn=0.21.2</li>
<li>scipy=1.3.0</li>
<li>notebook=6.0.0</li>
<li>h5py=2.9.0</li>
<li>mkl=2019.4</li>
<li>ipykernel=5.1.2</li>
<li>ipython=7.7.0</li>
<li>ipython_genutils=0.2.0</li>
<li>ipywidgets=7.5.1</li>
<li>jupyter=1.0.0</li>
<li>jupyter_client=5.3.1</li>
<li>jupyter_console=6.0.0</li>
<li>jupyter_core=4.5.0</li>
<li>plotly=4.1.1</li>
<li>scikit-image=0.15.0</li>
<li>requests==2.22.0</li>
<li>tqdm==4.43.0</li>
<li>pip:
<ul>
<li>tensorflow-gpu==2.1.0</li>
<li>tensorflow-estimator==2.1.0</li>
<li>tensorboard==2.1.1</li>
</ul>
</li>
</ul>
<p>However, it raised warning ‘NVIDIA RTX A5000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.’</p>
<p>It seems that my environment is unable to work, and the command ‘model.to(device)’ just froze and nothing shows up.</p>
<p>I have read other related posts that suggests I could solve the problem with:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
</code></pre>
<p>But sadly it shows conflicts and does not work as well, can anybody help me with this problem?</p>

ptrblck: <p>Your Ampere GPU needs CUDA11+ while your current installation uses <code>cudatoolkit=10.2</code> and will thus not work. The CUDA10.2 binaries are also not shipping <code>sm_86</code> for the same reason.<br>
Install the CUDA11.3 binary and it should work. If you are hitting conflicts, create a new conda environment and install it again.</p>
Pandas dataframe accumulates RAM memory

iknunyants: <p>I have a large dataframe dataset. There are float columns but also columns with lists of int. I wanted to optimise ram usage by storing it in 10 parquet files and reading them one by one while training. But when my dataset class is loading another parquet file into my “data” variable ram usage seems to slightly increase. This goes on and on, ram usage appears only larger when I view it via htop. Is there any way to fix it? Also, open to advice on how to solve the problem of loading the initial large dataset to ram.</p>

ejguan: <p>IIUC, You can load each parquet file lazily in <code>__getitem__</code> function in your <code>Dataset</code> class.</p>

iknunyants: <p>Perhaps I did not write it clearly. As I understand I’m doing everything lazily and every time I check the current file (partition). If this parquet is not in memory (i.e. I’ve read the previous one), I read it.</p>

iknunyants: <p>Sovled this issue just replacing pandas dataframe with pyarrow table. It takes 60% less ram for me, but not as fast in terms of getting rows values. My workaround is to make a numpy list from columns. Unfortunately, it takes ram as it used to with pandas.<br>
But my main problem with memory accumulation is gone.</p>