akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
datasets.ImageFolder generates the same label for different classes

AndrewK: <p>My code and output:</p>
<pre><code class="lang-auto">import torch
from torch import nn

import matplotlib.pyplot as plt
import numpy as np
import torchvision
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision.transforms as transforms

NUM_OF_CLASSES = 29
batch_size = 32

TestTransform = transforms.Compose(
    [
    transforms.Resize(224),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ]
)

RASBAND_set = torchvision.datasets.ImageFolder('../input/asl-alphabet-test', transform=TestTransform)
loader = torch.utils.data.DataLoader(RASBAND_set, batch_size=batch_size, shuffle=True)

# function to show an image
def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

# get some random test images
dataiter = iter(loader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join(f'{labels[j]}' for j in range(batch_size)))
</code></pre>
<p>Output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/91c674b21268bcc608575f7453ab21d405ac16e1.jpeg" alt="image" data-base62-sha1="kNAqdBXfjNIF6z2TzsPmrSYw4NP" width="376" height="202"></p>
<pre><code class="lang-auto">20 26 15 26 21 28 26 8 26 26 26 26 26 18 2 19 11 3 8 26 26 26 18 26 26 14 1 6 26 26 6 26
</code></pre>
<p>The dataset I used: <a href="https://www.kaggle.com/datasets/danrasband/asl-alphabet-test" class="inline-onebox" rel="noopener nofollow ugc">ASL Alphabet Test | Kaggle</a></p>
<p>For a reason, a lot of images get the label ‘26’ when they are clearly not in the same class. I checked the folder structure, and it should be sufficient for ImageFolder. What could be the problem?</p>

ptrblck: <p>Based on your description it seems you are concerned about the image to class mapping in the `ImageFolder.<br>
If so, you could check this mapping manually via:</p>
<pre><code class="lang-python">print(RASBAND_set.imgs)
</code></pre>
<p>which would return the a list containing image path and their label tuples.</p>
How to make an algorithm to learn some actions more than others in a multi action env

HRZ800: <p>Hi, I have a multi action env and I wanted to my algorithm(ppo) to consider learning some of my actions first, to do so I thought to detach unwanted actions for some epochs and then back to normal again. I implemented this with an if in <code>forward</code> method ( I pass a parameter to point if it must be detached or not?) do you think this implementation is right?<br>
for e.g we have 4 actions and I want to learn action 1 and 2 more like below:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/1/2129b19a25f8abf28295f787680b8d3fa503aea6.png" alt="model" data-base62-sha1="4Jn5gHwSG3WZZecZjy2zTE84wzc" width="401" height="371"></p>
How to get the input and output feature maps of a CNN model?

Amur_Saqib_Pal: <p>Hi, I am trying to find the dimensions of an image as it goes through a convolutional neural network at each layer. So for instance, if there is maxpooling or convolution being applied, I’d like to know the shape of the image at that layer, for all layers. I know I can use the <strong>nOut=image+2p-f / s + 1</strong> formula but it would be too tedious and complex given the size of the model. Is there a simple way to do this, perhaps a visualization tool/script or something? Thanks in advance.</p>

Andrei_Cristea: <p>I’ve never implemented this before, but I think you can iterate through all the Modules in your model and register a hook that prints the shape of the output, using <code>register_forward_hook</code> <a href="https://pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_forward_hook.html" rel="noopener nofollow ugc">like here</a>.</p>
<p>Something roughly like this:</p>
<pre><code class="lang-auto">import torch
model = torch.hub.load('pytorch/vision:v0.10.0', 'resnet18', pretrained=True)

def hook(module, input, output):
    print(output.shape)
    return None

for module in model.children():
    module.register_forward_hook(hook)
    
_ = model(torch.rand(1, 3, 224, 224))

Output:
torch.Size([1, 64, 112, 112])
torch.Size([1, 64, 112, 112])
torch.Size([1, 64, 112, 112])
torch.Size([1, 64, 56, 56])
torch.Size([1, 64, 56, 56])
torch.Size([1, 128, 28, 28])
torch.Size([1, 256, 14, 14])
torch.Size([1, 512, 7, 7])
torch.Size([1, 512, 1, 1])
torch.Size([1, 1000])
</code></pre>
<p>Note that in this case some of the children are Sequential blocks, so you may want to step through those / print their names and such.</p>
RuntimeError: Unknown qengine, how to solve the problem?

Oleg_Gelo: <p>import torch<br>
import sounddevice as sd<br>
import time</p>
<p>language = “ru”<br>
model_id = “ru_v3”<br>
sample_rate = 48000<br>
speaker = “baya”<br>
put_accent = True<br>
put_yo = True<br>
device = torch.device(‘cpu’)<br>
text = “Здраствуй Олег, рад знакомству”</p>
<p>model, _ = torch.hub.load(repo_or_dir=‘snakers4/silero-models’,<br>
model=‘silero_tts’,<br>
language=language,<br>
speaker=model_id<br>
)<br>
model.to(device)</p>
<p>audio = model.apply_tts(text=text,<br>
speaker=speaker,<br>
sample_rate=sample_rate,<br>
put_accent=put_accent,<br>
put_yo=put_yo)</p>
<p>print(text)</p>
<p>sd.play(audio, sample_rate)<br>
time.sleep(len(audio) / sample_rate)<br>
sd.stop()</p>
<p><span class="hashtag">#problems</span><br>
Using cache found in C:\Users\User/.cache\torch\hub\snakers4_silero-models_master<br>
Traceback (most recent call last):<br>
File “D:/Voice_Assistent2/main.py”, line 14, in <br>
model, _ = torch.hub.load(repo_or_dir=‘snakers4/silero-models’,<br>
File “D:\Voice_Assistent2\venv\lib\site-packages\torch\hub.py”, line 399, in load<br>
model = _load_local(repo_or_dir, model, *args, **kwargs)<br>
File “D:\Voice_Assistent2\venv\lib\site-packages\torch\hub.py”, line 428, in _load_local<br>
model = entry(*args, **kwargs)<br>
File “C:\Users\User/.cache\torch\hub\snakers4_silero-models_master\src\silero\silero.py”, line 88, in silero_tts<br>
model = imp.load_pickle(“tts_models”, “model”)<br>
File “D:\Voice_Assistent2\venv\lib\site-packages\torch\package\package_importer.py”, line 249, in load_pickle<br>
result = unpickler.load()<br>
File “C:\Users\User\AppData\Local\Programs\Python\Python38\lib\pickle.py”, line 1210, in load<br>
dispatch<a>key[0]</a><br>
File “C:\Users\User\AppData\Local\Programs\Python\Python38\lib\pickle.py”, line 1251, in load_binpersid<br>
self.append(self.persistent_load(pid))<br>
File “D:\Voice_Assistent2\venv\lib\site-packages\torch\package\package_importer.py”, line 227, in persistent_load<br>
loaded_reduces[reduce_id] = func(self, *args)<br>
File “D:\Voice_Assistent2\venv\lib\site-packages\torch\jit_script.py”, line 344, in unpackage_script_module<br>
cpp_module = torch._C._import_ir_module_from_package(<br>
RuntimeError: Unknown qengine</p>
<p>Process finished with exit code 1</p>

valerieboldarev: <p>Здравствуйте. у меня такая же проблема. Получилось решить?<br>
я нашел такой пост <a href="https://discuss.pytorch.org/t/getting-unknown-qengine-error-when-using-quantized-model-in-a-vm/111283/2" class="inline-onebox">Getting `Unknown qengine` error when using quantized model in a VM - #2 by Shisho_Sama</a><br>
Hello. I have the same problem. Did you manage to solve it?<br>
I found such a post <a href="https://discuss.pytorch.org/t/getting-unknown-qengine-error-when-using-quantized-model-in-a-vm/111283/2" class="inline-onebox">Getting `Unknown qengine` error when using quantized model in a VM - #2 by Shisho_Sama</a></p>
How to fill selected values with index into another embedding?

Albert_HK: <p>Hi,</p>
<p>how could I achieve the following function without for loop ?<br>
Should I use scatter function ?<br>
Thanks!</p>
<pre><code class="lang-auto">position_ids = torch.arange(500).reshape(10, 50)
prompt_pos = torch.arange(50).reshape(10, 5)

for index in range(10):
    position_ids[index][prompt_pos[index][0]:] =  prompt_pos[index][0] 
</code></pre>

Albert_HK: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,  Could I get your help ? , Thanks~~~~~</p>

KFrank: <p>Hi Albert!</p>
<aside class="quote no-group" data-username="Albert_HK" data-post="1" data-topic="152070" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/0ea827/48.png" class="avatar"> Albert_HK:</div>
<blockquote>
<p>how could I achieve the following function without for loop ?<br>
Should I use scatter function ?</p>
</blockquote>
</aside>
<p>I don’t see a way to do this with <code>scatter()</code>.  You can, however, compute<br>
an appropriate boolean mask and then use it to index into <code>position_ids</code><br>
(or, equivalently, use <code>torch.masked_select()</code>):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt;
&gt;&gt;&gt; position_ids = torch.arange(500).reshape(10, 50)
&gt;&gt;&gt; prompt_pos = torch.arange(50).reshape(10, 5)
&gt;&gt;&gt;
&gt;&gt;&gt; position_idsB = position_ids.clone()
&gt;&gt;&gt;
&gt;&gt;&gt; for index in range(10):
...     position_ids[index][prompt_pos[index][0]:] =  prompt_pos[index][0]
...
&gt;&gt;&gt; msk = torch.arange (50).unsqueeze (0).expand (10, 50) &gt;= prompt_pos[:, 0].unsqueeze (1)
&gt;&gt;&gt;
&gt;&gt;&gt; position_idsB[msk] = prompt_pos[:, 0].unsqueeze (1).expand (10, 50)[msk]
&gt;&gt;&gt;
&gt;&gt;&gt; torch.equal (position_ids, position_idsB)
True
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Albert_HK: <p>Thanks！ It works  ~.~</p>
Quantization multiple branches

julien_p: <p>Hi, I want to quantize a CNN with two fully connected branches at the end of the network. I don’t know if I should use one or two instances of DequantStub? Both V1 and V2 seem to work but what is the best practice?</p>
<pre><code class="lang-auto">class QuantizedModelV1(Module):

    def __init__(self, fp32_model):
        super(QuantizedModel, self).__init__()
        self.model_fp32 = fp32_model
        self.quant = QuantStub()
        self.dequant_b1 = DeQuantStub()
        self.dequant_b2 = DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        b1, b2 = self.model_fp32(x)
        b1 = self.dequant_b1(b1)
        b2 = self.dequant_b2(b2)
        return b1, b2
</code></pre>
<pre><code class="lang-auto">class QuantizedModelV2(Module):

    def __init__(self, fp32_model):
        super(QuantizedModel, self).__init__()
        self.model_fp32 = fp32_model
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        b1, b2 = self.model_fp32(x)
        b1 = self.dequant(b1)
        b2 = self.dequant(b2)
        return b1, b2
</code></pre>

jerryzh168: <p>DeQuantStub doesn’t carry any state so both are fine. I’d recommend the first one though, just to be more robust.</p>
Multi-label instance segmentation

AviPeltz: <p>Hi all,<br>
I’m an undergrad working on my senior project which is building a vision-based strawberry inspection system. I used detectron2 to train a model to detect instances of strawberries of various classes in images, prior to that I also trained a model to just identify strawberries in images(which I used to do model-assisted labeling to build the dataset i’m using). But I was wondering if there was a relatively easy way to modify the Mask R-CNN architecture to attach multiple class labels to an given object.<br>
This would be useful for this use-case because I want to identify all the instances of strawberries in the image, but I also want to tag each berry with quality criteria i.e. (green, underripe, overripe, damaged, moldy,… ) there are a lot of berries that have overlapping quality criteria and having the model infer the dominant class seems like a much more noisy problem to solve. Any help would be greatly appreciated!</p>
<p>Thanks!</p>

Andrei_Cristea: <p>Hi Avi, cool project! The easiest approach might be to train a separate image classification model that can classify strawberries into the categories you’re interested in, and in the inference stage just run this second model on your detectron2 output. Is there a particular reason you’re looking to combine both in the same model instead of taking this easier road?</p>
Loading large samples with DataLoader

robertomest: <p>Hello folks,<br>
I’m training a network with very large input samples, something along the lines of a tensor with <code>(48, 10, 448, 448)</code> dimensions, and I’m seeing a big slowdown in throughput when using <code>num_workers &gt; 0</code>.<br>
I’m exactly sure if I’m correct, but I think it’s because the tensors are being copied from the worker processes to the main process and this is a huge bottleneck. I’ve look around online but haven’t found any discussions on this topic. Is there a way to avoid this copy, or a recommendation on how to mitigate this problem in any other way? I appreciate the help.</p>
How to inference asynchronous

Forceless: <p>Is there any support for asynchronous inference?<br>
Since inference on GPU will also block the CPU, I hope I can process some CPU tasks while waiting.</p>

eqy: <p>By default cuda kernels are run asynchronously (you need to call <code>torch.cuda.synchronize()</code>) to block until all launched kernels are done.</p>

Forceless: <p>I know this, but my code don’t works like async</p>
<pre><code class="lang-python">import torch
data = torch.randn(128,16,3,224,224)
data = data.to('cuda')
from torchvision import models
model = models.squeezenet1_1(pretrained=True)
model.to('cuda')
for i in range(100):
    model(data[i])
</code></pre>
<p>What I expect:</p>
<ul>
<li>Immediately stoped, but take a long time</li>
<li>add torch.cuda.synchronize() don’t cost extra time</li>
</ul>

eqy: <p>How many GPUs are there in this setup?</p>
<p>As a followup it might be useful to investigate <a href="https://docs.python.org/3/library/multiprocessing.html" rel="noopener nofollow ugc">multiprocessing</a> in this setup even if it is indeed clunky (as a way to avoid unexpected GIL interactions).</p>
The time cost of torch.distributed.all_reduce across ranks is inconsistent

acplus: <p>I try torch.distributed.all_reduce and observe that the time overhead across machine is inconsistent. Especially rank2 machine achieve the smallest time overhead. My code is as follows:</p>
<pre><code class="lang-auto">def noop(net):
    grad_list = list() 
    for param in net.parameters(): 
        grad_list.append((param + torch.randn(param.shape).to(device)).flatten())
    flatten_grad = torch.cat(grad_list)
    start = time.time() 
    torch.distributed.all_reduce(flatten_grad, op=torch.distributed.ReduceOp.SUM)
    torch.cuda.synchronize()
    print('rank {}, after reduce sum grad {}, time cost {}'.format(adaptdl.env.replica_rank(), flatten_grad.sum(), time.time() - start))

def demo(net):
    for i in range(10): 
        noop(net)
    exit(0)
</code></pre>
<p>The result is as follows:</p>
<pre><code class="lang-auto">
rank 1, after reduce sum grad 19065.580078125, time cost 0.15841436386108398
rank 0, after reduce sum grad 19065.580078125, time cost 0.1609196662902832
rank 2, after reduce sum grad 19065.580078125, time cost 0.04276561737060547
rank 1, after reduce sum grad 19919.873046875, time cost 0.15205001831054688
rank 0, after reduce sum grad 19919.873046875, time cost 0.11464929580688477
rank 2, after reduce sum grad 19919.873046875, time cost 0.04297232627868652
rank 0, after reduce sum grad 24817.13671875, time cost 0.11996173858642578
rank 1, after reduce sum grad 24817.13671875, time cost 0.1443774700164795
rank 2, after reduce sum grad 24817.13671875, time cost 0.04287433624267578
rank 1, after reduce sum grad 30847.12109375, time cost 0.14579391479492188
rank 0, after reduce sum grad 30847.12109375, time cost 0.12775921821594238
rank 2, after reduce sum grad 30847.12109375, time cost 0.042836904525756836
rank 1, after reduce sum grad 2664.037109375, time cost 0.13935065269470215
rank 0, after reduce sum grad 2664.037109375, time cost 0.13114142417907715
rank 2, after reduce sum grad 2664.037109375, time cost 0.04304385185241699
</code></pre>
<p>Is it normal or not?<br>
Any potential ways to address this issue?</p>

pritamdamania87: <p>To measure the time accurately, you probably need another <code>torch.cuda.synchronize</code> before you execute the allreduce to ensure all previous GPU kernels have finished and only then start measuring time for allreduce.</p>
<p>The other way to accurately measure GPU time is using cuda events <code>https://pytorch.org/docs/stable/generated/torch.cuda.Event.html</code></p>
Question on ROCm support for W6800

carperbr: <p>Hey all,</p>
<p>I’m slowly working on building a local compute cluster for working on ml projects but recently realized I may have rushed a little. I was curious if anyone here has experience using a W6800 with PyTorch? My plan is to set up a few nodes over time running Ubuntu for distributed training (somewhat new to Linux so very open to recommendations here), but I have now realized that W6800 is listed as a supported GPU on PyTorch for ROCm.</p>
<p>A second question: how does the W6800 compare to an RTX 3090 Ti? Benchmarks for games look fairly equal, however this is not exactly super relevant. My original goal was to set up a compute cluster with 3090 Tis, but the W6800 having 32GB of VRAM is obviously more alluring to me now and I could easily sell the 3090 Ti I bought in order to partially fund a W6800.</p>
<p>Does anyone have any thoughts on this? My apologies if this is the wrong category, however it seemed there was no great category for this.</p>
How index from one dimension of a multi dimensional tensor using another tensor as indexes?

malfonsoarquimea: <p>Hi! I have two arrays of shape:</p>
<pre><code class="lang-auto">items.shape=(200,100,3)
indexes.shape=(200)
</code></pre>
<p>I would like to pick, for all 200 items in the first dimension, the item designed by the respective item at indexes. I would like to do a more elegant solution than</p>
<pre><code class="lang-auto">torch.stack([items[i,indexes[i]] for i in range(len(indexes))])
</code></pre>
<p>Thanks in advance!</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python"># setup
items = torch.randn(200, 100, 3)
indexes = torch.randint(0, 100, (200,))

# reference
ref = torch.stack([items[i,indexes[i]] for i in range(len(indexes))])

# direct indexing
out = items[torch.arange(items.size(0)), indexes]

# test
print((out == ref).all())
# tensor(True)
</code></pre>
Binary classification using vision transformers

OuisYasser: <p>Hello everyone.<br>
I have a project on a binary classification using vision transformers. Actually the architecture has a lot of other blocks but the one in interest is the encoder (vision transformer). However, in my dataset, in order to classify images into 0/1, each image can be both so i do not really know how to train the global model.</p>
<p>question: why each image can be classified for both 0/1?<br>
answer: in order to classify the image, I am relying on two inputs, the first one is the image and the second one is some vector which I add to the image. So, each image has both vectors of 0 and 1.</p>
<p>in the training loop I am passing an image and vector of 1 and have a class label (same thing for other class). and then I am summing the two losses *0.5 and backpropagate the global loss. is it correct?</p>
How to create a new layer in pytorch

Shivani_Sri_Varshini: <p>To create a new layer do we need to write the backward pass or autograd will take care of backward pass for given forward pass</p>
<pre><code>      class CustomLayer(torch.nn.Module):
          def __init__(self, k,b):
              super(CustomLayer, self).__init__()
      
              self.latent_dim = k
              Q=torch.empty(k,b)
              R=torch.empty(k,b)
          def forward(self, A):
              Q=A+Q
              R=A+R
              return Q, R
</code></pre>
<p>for this i’m getting error<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64]], which is output 0 of SelectBackward, is at version 65; expected version 64 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>How to interpret this error and how to write custom layer</p>

ptrblck: <aside class="quote no-group" data-username="Shivani_Sri_Varshini" data-post="1" data-topic="152192">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shivani_sri_varshini/48/45394_2.png" class="avatar"> Shivani_Sri_Varshini:</div>
<blockquote>
<p>do we need to write the backward pass or autograd will take care of backward pass for given forward pass</p>
</blockquote>
</aside>
<p>If you are using PyTorch operations and tensors then Autograd will take care of the backward pass. On the other hand, if you are using a 3rd party library (such as <code>numpy</code>) or use non-differentiable operations you would need to implement a custom <code>autograd.Function</code> with the <code>backward</code> definition.</p>
<aside class="quote no-group" data-username="Shivani_Sri_Varshini" data-post="1" data-topic="152192">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shivani_sri_varshini/48/45394_2.png" class="avatar"> Shivani_Sri_Varshini:</div>
<blockquote>
<p>for this i’m getting error<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
</aside>
<p>I don’t think this module causes the issue as no inplace operations are used there, so could you post a minimal, executable code snippet reproducing this error?</p>
Is there any method to estmate the time comsuming for torch.tensordot() or einsum() on GPU?

kuangll: <p>As we do some large dimension tensors contraction, we found there are two kinds of kernels are used, <code>unrolled_elementwise_kernel</code> for tensors <strong>reshape</strong>, and <code>cutlass::Kernel</code> for tensors <strong>mm</strong>.</p>
<p>For the first step (<strong>reshape</strong>), how to calculate the data size copied and estimate the data copy (device to device)?</p>
<p>For the second (<strong>mm</strong>), there always several or many cuda kernels generated for larger size tensors, could anybody know how that comes from?</p>
<p>Thanks.</p>

tom: <p>The reshape will copy the tensor, so it likely you will find an approximately linear relationship between <code>.numel()</code> and runtime. There are some subtleties as the input is going to be non-contiguous (otherwise the .reshape would not copy), but maybe those do not change as much between the things you would be comparing.<br>
I think <code>mm</code> on CUDA uses CuBLAS by default. I think what you are seeing are optimizations CuBLAS makes.</p>
<p>Best regards</p>
<p>Thomas</p>

kuangll: <p>Thanks for your reply <a class="mention" href="/u/tom">@tom</a>. I have more test recently. For the first reshape step, we found the seleted dimensions to be contracted is important to this step.</p>
<p>For example, I want to select 6 dimensions from a 31-dim tensor to permute and reshape, if the dimensions (22, 24, 26, 18, 21, 0) cost ~48ms, but (29, 0, 8, 25, 26, 27) cost ~25ms (At the end I paste my test code).</p>
<p>I profile to see the slower case is memory bound and include more uncoalesced global memory accessses. But so far I cannot make it clear how to calculate the memory requirement.</p>
<pre><code class="lang-auto">import torch
import time

a = torch.randn([2]*31, device="cuda", dtype=torch.complex64)

for i in range(20):
    torch.cuda.synchronize()
    t0 = time.time()
    #a.permute((1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 23, 25, 27, 28, 29, 30, 22, 24, 26, 18, 21, 0)).reshape([2**25, 2**6])
    a.permute((1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 28, 30, 29, 8, 25, 26, 27, 0)).reshape([2**25, 2**6])
    torch.cuda.synchronize()
    t1 = time.time()
    print(f'Iter#{i} time/sec: ', t1 - t0 )
</code></pre>

tom: <p>So einsum translates to a series of batch matrix multiplications after permutations and merging the axes that are treated the same (contraction/lhs to result/rhs to result/batch). So you could do this yourself if you wanted to.</p>
Double backwards is not supported for CuDNN RNNs

Tianqi_Li: <p>Good morning everyone,</p>
<p>I am using LSTM to build the model of WGAN with gradient penalty, however, during the loss backward propagation for the critic, I have got the error ‘Double backwards is not supported for CuDNN RNNs’. So I wonder whether it is possible to solve this problem? Due to the fact that this is the model for my Master project and I don’t want to change the model, I am therefore looking forward to help!</p>
<p>Thank you very much for your reponse!</p>

ptrblck: <p>You could either disable cuDNN globally via <code>torch.backends.cudnn.enabled = False</code>, which would most likely slow down your entire code or disable it for the RNN layer only via <code>with torch.backends.cudnn.flags(enabled=False):</code>.</p>

Tianqi_Li: <p>Thank you very much! I will take a try!</p>
Workaround to indexing containing grads

bstan: <p>Hello community,</p>
<p>I am struggling to think of a way to get the behavior I am looking for.</p>
<p>I have a sub-network predicting 3D regions of interest, yielding a tensor of the shape (num_objects, 6). The 6 refers to x1, y1, z1, x2, y2, z2 coordinates of the 3D region of interest. Based on these coordinates I want index a boolean tensor that has the same shape as the input image, namely (120, 120, 120), and set the values of these region of interests to False.<br>
So naturally I first cast the tensor of shape (num_objects, 6) to integers and use the predicted coordinates for the sake of indexing.</p>
<pre><code class="lang-auto">dummy_fmap[roi[0][0]:roi[0][3], roi[0][1]:roi[0][4], roi[0][2]:roi[0][5]] = False
</code></pre>
<p>Since the indexing operation is unfortunately not differentiable, the weights in the sub-network are not optimized. Now I am thinking of alternate approaches and would be glad to get some help on this.<br>
I am aware that ROI pooling of two stage detectors does something similar, but was not able to adopt this to my task.</p>
<p>Thanks in advance!</p>
Loss function which acts on only one class

damar: <p>Hi all, I have a network which first classifies samples. After this, these classifier outputs are fed to a regressor. The network is used to decorrelate a classifier output with a specific feature. I however only want to use the regression loss function for the background class (the signal class should not be decorrelated). Is there a way to define a loss function which is normal for one class and constant/zero for another class?</p>
<p>I tried something like this, since I have a batch size of 50:</p>
<pre><code class="lang-auto">regression_loss_function(reg_outputs, reg_targets)*(torch.ones([50])-classifier_targets) + classifier_loss_function(classifier_outputs, classifier_targets)
</code></pre>
<p>But I realise I understand too little of the Pytorch loss functions to make this work. Any help is appreciated!</p>
Classifier that outputs random distribution

Peter_Featherstone: <p>I have a classifier where the last layer is a <code>torch.nn.Linear</code> (fc) layer with let’s say 1024 output classes.<br>
For genuine inputs which fall into one of the 1024 classes, I want the classifier to output the correct class, i.e. a distribution which can be trained with <code> torch.nn.CrossEntropyLoss</code> such that if I pass the output through <code>torch.softmax()</code> and <code>torch.argmax()</code> I get the correct class.<br>
Now, if I pass bad input, I want the classifier to output a random distribution, for example a gaussian distribution. How do I train that?</p>
<p>So far some candidates are <code>torch.nn.KLDivLoss</code> and <code>torch.nn.GaussianNLLLoss</code>. But I don’t know how to use them in this context. I’ve also thought about calculating the mean, variance, skewness and kurtosis of the output distribution and regressing them to 0, 1, 0 and 3 respectively. But not sure if that’s the right way to do it. There is also the <code>torch.distributions</code> package but not sure what to do with that.</p>
<p>Any ideas?</p>
<p>Thanks</p>
Ensembles with pre-trained classification models for image retrieval

beginner1: <p>Hello guys,<br>
I’m working on a image retrieval project and I want to do an ensemble with pre trained models (as instance resnet 18 and resnet 50). This is the snippet of code I have. Any ideas how I can do that?</p>
<pre><code class="lang-auto">import torch
import logging
import torchvision
from torch import nn

from model.layers import Flatten, L2Norm, GeM


CHANNELS_NUM_IN_LAST_CONV = {
        "resnet18": 512,
        "resnet50": 2048,
        "resnet101": 2048,
        "vgg16": 512,
    }


class GeoLocalizationNet(nn.Module):
    def __init__(self, backbone, fc_output_dim):
        super().__init__()
        self.backbone, features_dim = get_backbone(backbone)
        self.aggregation = nn.Sequential(
                L2Norm(),
                GeM(),
                Flatten(),
                nn.Linear(features_dim, fc_output_dim),
                L2Norm()
            )
    
    def forward(self, x):
        x = self.backbone(x)
        x = self.aggregation(x)
        return x


def get_backbone(backbone_name):
    if backbone_name.startswith("resnet"):
        if backbone_name == "resnet18":
            backbone = torchvision.models.resnet18(pretrained=True)
        elif backbone_name == "resnet50":
            backbone = torchvision.models.resnet50(pretrained=True)
        elif backbone_name == "resnet101":
            backbone = torchvision.models.resnet101(pretrained=True)
        
        for name, child in backbone.named_children():
            if name == "layer3":  # Freeze layers before conv_3
                break
            for params in child.parameters():
                params.requires_grad = False
        logging.debug(f"Train only layer3 and layer4 of the {backbone_name}, freeze the previous ones")
        layers = list(backbone.children())[:-2]  # Remove avg pooling and FC layer
    
    elif backbone_name == "vgg16":
        backbone = torchvision.models.vgg16(pretrained=True)
        layers = list(backbone.features.children())[:-2]  # Remove avg pooling and FC layer
        for l in layers[:-5]:
            for p in l.parameters(): p.requires_grad = False
        logging.debug("Train last layers of the VGG-16, freeze the previous ones")
    
    backbone = torch.nn.Sequential(*layers)
    
    features_dim = CHANNELS_NUM_IN_LAST_CONV[backbone_name]
    
    return backbone, features_dim
</code></pre>
ONNX export using list of nn.Sequential/nn.ModuleList

MrStuhl: <p>Hello together,</p>
<p>I have the problem that, if using a list of nn.Sequential or nn.ModuleList the export of ONNX does not work and just prints out numbers. Spliiting all elements from the list to separate objects instead works as usual. Does someone know why this is actually happening and may explain it to me please? Is this a wanted behaviour?</p>
<p>Maybe one example code snippet to make more clear what my problem is right now.</p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class DoesNotWork(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()

        self.seq = [nn.Sequential(*[
            nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)
        ])]
    
    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        return self.seq[0](x)

class Works(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()

        self.seq = nn.Sequential(*[
            nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)
        ])
    
    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        return self.seq(x)

if __name__ == '__main__':
    ex_mod1 = Works().eval()
    ex_mod2 = DoesNotWork().eval()
    input   = torch.rand(1, 3, 960, 1280)
    y1      = ex_mod1(input) 
    y2      = ex_mod2(input) 

    torch.onnx.export(
        model           = ex_mod1,
        args            = (input),   # input
        f               = r".../mod.onnx",
        export_params   = True,         
        verbose         = False,
        input_names     = ["i_x"],
        output_names    = ["o_y"],
        opset_version   = 11            
    )

    torch.onnx.export(
        model           = ex_mod2,
        args            = (input),   # input
        f               = r".../mod.onnx",
        export_params   = True,          
        verbose         = False,
        input_names     = ["i_x"],
        output_names    = ["o_y"],
        opset_version   = 11            
    )

</code></pre>

MrStuhl: <p>Ok,</p>
<p>I have found an explanation here if someone else is interested. I have overseen the actual error. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p><a href="https://discuss.pytorch.org/t/cannot-insert-a-tensor-that-requires-grad-as-a-constant/46561">Cannot insert a Tensor that requires grad as a constant - jit - PyTorch Forums</a>.</p>