akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why most torchvision.transforms inherit nn.Module?

username: Jingnan_Jia
cooked: <p>From the <a href="https://pytorch.org/vision/master/_modules/torchvision/transforms/transforms.html#Compose" rel="noopener nofollow ugc">source code</a>, I found nearly all of the torchvision.transforms inherit nn.Module.</p>
<p>But in the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">official tutorial on custom transforms</a> we can create the custom transform from scratch (without the inheritance from nn.Module).</p>
<p>I found in the <a href="https://pytorch.org/vision/master/_modules/torchvision/transforms/transforms.html#Compose" rel="noopener nofollow ugc">source code</a>, even those transforms are created from scratch  (without the inheritance from nn.Module), it seems that these transforms will still work well.</p>
<p>So why most torchvision.transforms inherit nn.Module? And do we need to inherit nn.Module if we build our own custom transforms?</p>

username: thomasjo
cooked: <p>They inherit from <code>nn.Module</code> for two key reasons;</p>
<ol>
<li>It makes the transforms runnable on GPUs.</li>
<li>It makes them compatible with TorchScript.</li>
</ol>
<p>Unless either of these are important to your specific application, you can safely implement your custom transforms without inheriting from <code>nn.Module</code>.</p>

username: Jingnan_Jia
cooked: <pre><code class="lang-auto">class MyNormalize:
    def __call__(self, img):
        if type(img) == np.ndarray:
            mean, std = np.mean(img), np.std(img)
        elif type(img) == torch.Tensor:
            mean, std = torch.mean(img), torch.std(img)
        img = img - mean
        img = img / std
        return img
</code></pre>
<p>This is the transform I created for myself. This custom transform can still work on GPUs. So I donnot think the first reason is important.</p>
How to sum over specifi samples in batch

username: gebrahimi
cooked: <p>I have the following batch</p>
<pre><code class="lang-auto">tensor([[[ 0.,  1.],
         [ 2.,  3.],
         [ 4.,  5.],
         [ 6.,  7.]],

        [[ 8.,  9.],
         [10., 11.],
         [12., 13.],
         [14., 15.]],

        [[16., 17.],
         [18., 19.],
         [20., 21.],
         [22., 23.]]])
</code></pre>
<p>with shape [3,4, 2]. how can I get different summation combinations along the batch dimension? for instance I want the first row of output to be summation of row1 and row3 and second row, just row 2, i.e.</p>
<pre><code class="lang-auto">tensor([[[ 16.,  18.],
         [ 20.,  22.],
         [ 24.,  26.],
         [ 28.,  30.]],

        [[ 8.,  9.],
         [10., 11.],
         [12., 13.],
         [14., 15.]],
</code></pre>
<p>I thought of multiplication with tensor tensor([[1., 0., 1.],<br>
[0., 1., 0.]]), but obviously there is dimension mismatch. Thank you.</p>

username: gebrahimi
cooked: <p>I figured out the solution. torch.matmul(m.T, n.T).T, where m is the main tensor and n is the tensor of 0 and 1s. Just in case someone needs it.</p>
How to provide colorful seegment to the labels of pascal voc dataset?

username: Aashish_P
cooked: <p>I have been using the pascal voc dataset directly from the torchvision.datasets as:</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.VOCSegmentation(root='drive/My Drive/VOC/',
                                                 year='2012',
                                                 image_set='trainval',
                                                 download=False,
                                                 transform=transforms.Compose([
                                                                      transforms.Resize(144),
                                                                      transforms.CenterCrop(128),
                                                                      transforms.ToTensor()
                                                 ]),
                                                 target_transform=transforms.Compose([
                                                                             transforms.Resize(144),
                                                                             transforms.CenterCrop(128),
                                                                             transforms.ToTensor()
                                                 ]),
                                                 transforms=None)
</code></pre>
<p>but using this I am unsuccessful to do my task since I need a colorful segment of the images, the images, and their respective layers are<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/be4e5837eadaa31919c034d36d51ee9765fdb526.png" alt="download" data-base62-sha1="r9wzstpwwrBwfIqtP5eryQE9ctU" width="128" height="128"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fae48d5c2fa3a960912828954aeadd6ea94ade3f.png" alt="helloyaar" data-base62-sha1="zNv2nFJ4zGGh8yXHjvyPKZcgAJF" width="128" height="128"><br>
how can I make those labels colorful using palette as</p>
<pre><code class="lang-auto">[(0, 0, 0), 
                           (128, 0, 0), 
                           (0, 128, 0),
                           (128, 128, 0),
                           (0, 0, 128),
                           (128, 0, 128), 
                           (0, 128, 128), 
                           (128, 128, 128),
                           (64, 0, 0), 
                           (192, 0, 0), 
                           (64, 128, 0), 
                           (192, 128, 0), 
                           (64, 0, 128),
                           (192, 0, 128), 
                           (64, 128, 128), 
                           (192, 168, 128), 
                           (0, 64, 0),
                           (128, 64, 0), 
                           (0, 192, 0), 
                           (128, 192, 0), 
                           (0, 64, 128)]
</code></pre>

username: seyeeet
cooked: <p>I have the same problem, did you solve it? also how does the structure of your folder looks like inside VOC?</p>
Conv2d with 4 pos args

username: bverhaar
cooked: <p>LS</p>
<p>I’m trying to understand some code made by someone else.<br>
One of the layers in the pytorch neural network is:</p>
<pre><code class="lang-auto">self.conv1 = torch.nn.Conv2d(4, 32, 8, 4)
</code></pre>
<p>In the code I notice that the input to this layer is<br>
torch.Size([1, 4, 84, 84])<br>
and the output of this layer is<br>
torch.Size([1, 32, 20, 20])</p>
<p>I read the documentation of pytorch conv2d, but this confused me since there are only 3 positional arguments:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html</a></p>
<pre><code class="lang-auto">torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros')
</code></pre>
<p>whereas in the code I have there are 4 positional arguments.</p>
<p>I’d like to understand how Conv2d(4, 32, 8, 4) transforms a torch.Size([1, 4, 84, 84]) to a torch.Size([1, 32, 20, 20])</p>
<p>Kind rgds</p>

username: rad
cooked: <p>So, the parameters are the following:</p>
<ul>
<li>in_channels = 4</li>
<li>out_channels = 32</li>
<li>kernel_size = 8</li>
<li>stride = 4</li>
</ul>
<p>The input is of size (batch_size, n_channels, height, width). Batch dim is 1 in both, so that can be discarded. The second dim of the input is 4, which is the in_channels, and the second dim of the output is 32, which is the out_channels.</p>
<p>Now, how (84, 84) turns into (20, 20) follows the formula described in the conv2d documentation:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca6156f5e512f92fe6707015877d66e469fc2b8f.png" alt="image" data-base62-sha1="sSkZTFEGP67n9XsE1Hq4I8H54oD" width="630" height="182"></p>
<p>h_out = floor(((84 + 2 * 0 - 1 * (8 - 1) - 1))/4 +1) = floor(19 + 1) = 20<br>
w_out is the same.</p>

username: bverhaar
cooked: <p>Excellent, thanks a lot!</p>
<p>So, do I understand correctly that there are in total 4 * 32 * 8 * 8 kernel coefficients? I.e. one 8x8 kernel for each (input layer C_in, output layer C_out) combination?</p>

username: bverhaar
cooked: <p>I just realized that key word arguments can be assigned as a positional argument as well. Just tested it, and indeed it’s the case. Didn’t know that!</p>
Transfer learning on an existing class in object detection with pytorch

username: JoseAntonio7
cooked: <p>I need to perform transfer learning on a class of a pre-trained model using pytorch oriented to object detection.</p>
<p>Basically what I want to do is based on some detections that I have done manually on a series of cars, perform transfer learning to improve the output of the detection model.</p>
<p>To date I have not found information related to this aspect, so I do not know very well how to perform this task.</p>

username: rad
cooked: <p>The transfer learning technique you are looking for in this case is called <em>fine-tuning</em>. Basically you take a network for object detection (a CNN architecture for instance, from <code>torchvision.models</code>), and you “unfreeze” some of its layers, and train only those. You usually want to unfreeze the last layers, as the first ones are used to extract more general features. The more images you have in your own labeled data, the more layers you should “unfreeze”. Probably a good starting point is <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">this</a>, but if you want more just Google “fine tune object detection pytorch”.</p>
Pruning at locally consistent levels

username: Joshua_Clancy
cooked: <p>I am exploring pruning for research (I am not concerned with speed reduction). I am wondering if there is an easy way to assure that each layer gets pruned to the same degree. For example, when I prune with global unstructured (25%) I get this result layer by layer.</p>
<p>Sparsity in conv1.weight: 7.81%<br>
Sparsity in conv2.weight: 18.60%<br>
Sparsity in lin1.weight: 39.92%<br>
Sparsity in lin2.weight: 11.89%<br>
Sparsity in upconv1.weight: 11.33%<br>
Sparsity in upconv2.weight: 8.59%<br>
Global sparsity: 25.00%</p>
<p>The global sparsity is indeed 25%, but I would like this to be true for each layer individually. Does anyone have any ideas of how to best accomplish this?</p>
Model architecture visualization tool (2D or 3D)

username: celineteller
cooked: <p>Hello,</p>
<p>Is there any interesting tool to plot the model architecture in 3D or 2D at least ?</p>
<p>Thank you</p>

username: floriandonhauser
cooked: <p>Tensorboard is one option, have a look at the tutorial for it:<br>
<a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html</a></p>
Custom torch.autograd.Function - backward implementation?

username: tdm
cooked: <p>I have written a custom torch.autograd.Function.</p>
<p>The forward function is complete, and is solely a number of tensor operations that transform the set of parameter inputs (matrix multiplication, argmax, concatenation etc.). It’s not something I could differentiate by hand.</p>
<p>How can I implement the .backward() function in this case? I want to obtain the gradients of all the inputs, and my understanding was that PyTorch could compute the derivatives automatically?</p>
<p>Apologies if I’m missing something very obvious here.</p>
<p>The function is simply as follows right now - I’ve simplified it slightly - the full ‘transformationFn’ exists within the forward function.<br>
input: N x 4 tensor<br>
params1: 1 x 2 tensor<br>
params2: 1 x 2 tensor<br>
transformationFn takes [input, params1, params2] → returns N x 4 tensor</p>
<pre><code class="lang-auto">class model(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, params1, params2): #
        ctx.save_for_backward(input, params1, params2)
        return transformationFn(input, params1, params2)
</code></pre>

username: KFrank
cooked: <p>Hi Tdm!</p>
<aside class="quote no-group quote-modified" data-username="tdm" data-post="1" data-topic="118959" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tdm/48/37395_2.png" class="avatar"> tdm:</div>
<blockquote>
<p>I have written a custom torch.autograd.Function.</p>
<p>The forward function is complete, and is solely a number of tensor operations that transform the set of parameter inputs (matrix multiplication, argmax, concatenation etc.).<br>
…<br>
How can I implement the .backward() function in this case?</p>
</blockquote>
</aside>
<p>If your function consists solely of (usefully-differentiable) pytorch<br>
tensor operations you don’t need to package it as an official<br>
<code>torch.autograd.Function</code> nor write an explicit <code>backward()</code><br>
function; autograd will work “automatically” when you call <code>backward()</code><br>
on your final result (typically your loss).</p>
<p>So (just to make something up):</p>
<pre data-code-wrap="python"><code class="lang-python">def my_function (input, params1, params2):
    return (params1 * input + params2 * input**2).sum (dim = 1)
</code></pre>
<p>autograd can “backward” through <code>my_function()</code> without you needing<br>
to write any further code.</p>
<p>Best.</p>
<p>K. Frank</p>

username: tdm
cooked: <p>Thanks very much KFrank! I had previously tried this, and I was obtaining ‘None’ for all but one of my gradients, so had thought that I had not implemented the function properly.</p>
<p>I will now look at the training loop to see what the issue is.</p>
Build time of release build

username: MartinPerry
cooked: <p>I am using libtorch 1.8.0 with Cuda in my C++ project. My project is build in Visual Studio 2019, C++17. The problem is build time of release build. Debug build is created under a minute, however, release build takes up to 10 minutes. I know that release builds take longer time in generall, but this seems way too much for an app with a single, small convolution network.</p>
<p>Is there any way how to speed up this process? I have tried to use precompiled headers, but took was even longer.</p>
Register hook for all operators, including e.g., simple add

username: kmaeng
cooked: <p>I want to call a function on each operator of the forward pass of my NN.<br>
So naturally, I am registering a forward hook.<br>
However, I am currently doing something like this:</p>
<pre><code class="lang-auto">for name, module in net.named_modules():
    module.register_forward_hook(hook_fn)
</code></pre>
<p>and it only registers a hook for named module, e.g., Conv2d, but not for simple operators, e.g., + or torch.cat().<br>
I want this hook to be called on every operator node, including simple ones like + or torch.cat().<br>
Is there a way to do so?</p>
<p>Thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I am afraid these functions are at a much lower level than torch.nn and you won’t be able to use a direct hooking system.<br>
One thing you can try to do is wrap your input in a Tensor-like object and write a custom implementation that calls your hook and then delegate to the original pytorch implementation: <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-with-a-tensor-like-type" class="inline-onebox">Extending PyTorch — PyTorch 1.8.1 documentation</a></p>
Saving Computational Time with DNN Pruning

username: Stefano_Barone
cooked: <p>I am following <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" rel="noopener nofollow ugc">this</a> tutorial on pruning.<br>
It seems particularly attractive to me as it is said that if we reduce the number of parameters of the model we will reduce the execution time, which is relevant in my application.</p>
<p>However, from this example that I applied to my case, it just seems to me that a certain percentage of parameters are zeroed, but actually, they remain there in the model with a value of 0.</p>
<blockquote>
<p>They are not removed, so it seems to me that the execution time remains the same.</p>
</blockquote>
<p>What am I doing wrong? How to do it?</p>
How can I add hook in more than one layer?

username: Ragnor
cooked: <p>t_FM = []<br>
s_FM = []<br>
hooks = []</p>
<pre><code>def hook1(module1, input, output):
    print("hooker Working")
    t_FM.append(output)

hooks.append(t_model.tbackbone.layer2[1].conv1.register_forward_hook(hook1))

def hook2(module2, input, output):
    print("hooker Working")
    s_FM.append(output)

hooks.append(s_model.sbackbone.layer2[1].conv1.register_forward_hook(hook2))

your_model(input_tensors)  # forward with firing all the hooks registered
for hook_handle in hooks:
    hook_handle.remove()</code></pre>

username: sio277
cooked: <p>Hello again <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:">.</p>
<p>Try to attach the hook to the module you want, after registering it in <code>t_model</code>'s <code>__init__</code>.</p>

username: sio277
cooked: <p>Or, you can use <code>nn.Module.apply</code> method:</p>
<pre><code class="lang-auto">with torch.no_grad():
    def add_hook(module):
        def hook1(module, input, output):
            print("hooker Working")
            t_FM.append(output)
        
        if module is t_model.tbackbone.layer2[1].conv1:
            hooks.append(module.register_forward_hook(hook1))

        def hook2(module, input, output):
            print("hooker Working")
            s_FM.append(output)
    
        if module is s_model.sbackbone.layer2[1].conv1:
            hooks.append(module.register_forward_hook(hook2))

    your_model.eval()
    your_model.apply(add_hook)
    your_model(input_tensors)  # forward with firing all the hooks registered
    for hook_handle in hooks:
        hook_handle.remove()
</code></pre>

username: Ragnor
cooked: <p>Thank you very much for your time. <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>
Appending to Dataset for Active Learning

username: maxcompression
cooked: <p>I am trying to build an Active learning system, and have my full labelled Dataset now split up into two Datasets. The first I call the labelled Dataset, which the model can train on, the second I call the unlabelled Dataset, which actually contains labels, but the model has no access to them, and can only query samples from that set to find out what the label is, so that it can learn from it in the next AL iteration. Could somebody please explain me what an efficient way would be to ‘append’ a sample to the labelled Dataset, and remove it from the unlabelled Dataset? Is using the Dataset class a good option for Active Learning anyway? It seems that you will have to construct the Dataset every AL iteration fully over, instead of just being able to append to it. I would really like to be able to have the ease of Dataloaders for training, so that would point at using Dataset, but I don’t want huge inefficiency, because you cannot append easily to Dataset.</p>
<p>I would really value a small code snippet showing how you can efficiently remove a subset from the unlabelled Dataset, and move it to the labelled Dataset. Or of course something equivalent and more efficient, if it is the case that it cannot be done without fully reconstructing the Dataset again every time you move a subset from the one Dataset to the other</p>

username: samsja
cooked: <p>I would like to know the best way to do it as well !</p>
Memory issue when concatenating with torch.cat()

username: Hari_Krishnan
cooked: <p>In the below code i am converting the input data into embedings using a pre-trained BERT model. Since I only require the pooler output which is a vector of length 768 and performing a loop throughout the data and concatenate it into the <code>passage_vectors</code> tensor. But this causes the memory to run out after 15-20 data points. I tried running the program after commenting out the <code>torch.cat()</code> statement, which worked completely fine.</p>
<pre><code class="lang-auto">passage_vectors = torch.tensor([])
for i in range(len(dicts)):
    preprocessed = preprocessor.process(dicts[i])
    for j in range(len(preprocessed)):
      passage = preprocessed[j]['text']
      passages.append(preprocessed[j]['text'])
      ctx_tokens = context_tokenizer(passage, return_tensors = "pt", padding='max_length',truncation=True)
      ctx_vector = context_model(input_ids = ctx_tokens.input_ids, attention_mask = ctx_tokens.attention_mask)['pooler_output']
      #passage_vectors = torch.cat((passage_vectors,ctx_vector), 0)
</code></pre>
<p>There are around 2000 input data. So the final output tensor would be of dimensions 2000x768 which shouldn’t take up that much memory. I am running this in Google Colab which has 12gb RAM. Is there something is can do to make it more memory efficient.</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you do this on Tensors that require gradients, the autograd state might take quite a bit of memory as all the intermediary result might be saved (you go from linear to quadratic).</p>
<p>Also why do you concatenate inside the loop? Why not just append to the list inside the loop and concatenate once at the very end?</p>

username: Hari_Krishnan
cooked: <p>Yes, I was able to solve it by using <code>detach()</code> on <code>ctx_vector</code>.<br>
I did try appending at first, this was another approach I did to check whether it would solve the memory issue.</p>
Apply multiple heads to single tensor

username: CherubimHD
cooked: <p>Hi,</p>
<p>say I have an input tensor <strong>x</strong> and my model contains multiple heads where each head processes <strong>x</strong> separately. Afterwards, the outputs shall be concatenated. If the heads were linear layers this would not be a problem as I could model this as a single linear layer. However, the heads are convolutional networks. So far I have tried putting all the heads in a ModuleList and then iterating over the ModuleList, each time appending the new result to some existing tensor. However, this is really slow. Are there better approaches to do this?</p>
<p>Thanks!</p>

username: KFrank
cooked: <p>Hi Cherubim!</p>
<aside class="quote no-group" data-username="CherubimHD" data-post="1" data-topic="118844" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/7ab992/48.png" class="avatar"> CherubimHD:</div>
<blockquote>
<p>I have tried putting all the heads in a ModuleList and then iterating over the ModuleList, each time appending the new result to some existing tensor. However, this is really slow.</p>
</blockquote>
</aside>
<p>I saw a suggestion in an answer to a related question:  Consider adding<br>
each result to a regular python list, and then run <code>torch.cat()</code> on that<br>
list.  This way you don’t incur the expense of a tensor-append for every<br>
loop iteration.</p>
<p>Best.</p>
<p>K. Frank</p>

username: CherubimHD
cooked: <p>Thank you I will try that!</p>
Merge two model on one

username: aazz
cooked: <p>Hello,<br>
I have make two training on image classification.<br>
First training on type of glass<br>
Second on type of clothes<br>
So I have two files: glass.pth and clothes.pth<br>
I would like now merge this two files on one.<br>
Do you know how it is possible to do this ?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>You could use <a href="https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008/2">this approach</a>, but would need to check, if just averaging different parameters is the right approach.<br>
E.g. if both models converged to a completely different set of parameters, I would assume that the average of these models could perform badly.</p>

username: aazz
cooked: <p>Thank you.<br>
Second question, how get the labels name when we predict image.<br>
For load the model I used :</p>
<pre><code class="lang-auto">model=torch.load('aerialmodel.pth')
model.eval() 

image_tensor = transforms(image).float()
image_tensor = image_tensor.unsqueeze_(0)
input = image_tensor
input = input.to(device)
output = model(input)
index = output.data.cpu().numpy().argmax()

</code></pre>

username: ptrblck
cooked: <p>You could use a mapping between the class indices and the label names, e.g. via a <code>dict</code>, where the keys could be the class indices and the values the class names.</p>

username: aazz
cooked: <p>Yes, but it’s impossible to save this label in the model ?</p>

username: ptrblck
cooked: <p>You can assign any attribute to the model, but note that only parameters and buffers would show up in the <code>state_dict</code>. If you want to store it, you could add the mapping to the checkpoint manually and save it in a similar way as e.g. the <code>epoch</code> argument is stored in the <a href="https://github.com/pytorch/examples/blob/aef78a5b48f53956f4c7acf555a84cc9c937bc28/imagenet/main.py#L256-L262">ImageNet example</a>.</p>

username: aazz
cooked: <p>Thanks you, I try this</p>
Non-deterministic evaluating results with different batch sizes

username: 111471
cooked: <p>Hi, colleagues. We have a problem with determinism with different batch sizes<br>
We are evaluating a GRU model, but with different batch sizes we get different results after concatenation. We use flag CUBLAS_WORKSPACE_CONFIG=:4096:2, but it does not help.<br>
Their absolute difference is stable, namely 1.3411e-07, but it is important for our task to get equal results with different batch sizes.<br>
Here is link for input <a href="https://drive.google.com/drive/folders/1D-JnY9dKCLMlWCOo7WOdGyF1SqOD7YHu?usp=sharing" rel="noopener nofollow ugc">tensor</a>.</p>
<pre><code class="lang-auto">import torch
from torch import nn

#Simple GRUModel
class GRUModel(nn.Module):
    def __init__(self, data_dim, hidden_size, num_layers, dropout_prob, output_size, device):
        super(GRUModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.output_size = output_size
        self.gru = nn.GRU(input_size=data_dim, hidden_size=self.hidden_size, num_layers=self.num_layers,
                          dropout=dropout_prob, batch_first=True)
        self.linear = nn.Linear(self.hidden_size, self.output_size)
        self.device = device

    def forward(self, x, hidden):
        x, hidden = self.gru(x, hidden)
        x = x.contiguous().view(x.size(0), -1)
        x = self.linear(x)
        return x, hidden

    def init_hidden_state(self, batch_size, dtype=torch.float32):
        hidden = torch.zeros(self.num_layers, batch_size, self.hidden_size, dtype=dtype, device=self.device)
        return hidden

#Setting seeds
torch.set_num_threads(1)
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

device = torch.device('cuda')
batch_x = torch.load("batch_x0", map_location=device)
batch_x = batch_x.float()
batch_x.to(device)

model = GRUModel(data_dim=256, hidden_size=2, num_layers=1, dropout_prob=0, output_size=4, device=device)
model.to(device)
model.eval()

all_probs = []

for i, el in enumerate(batch_x):
    hidden_ = model.init_hidden_state(1)
    el = el.unsqueeze(0)
    batch_probs, _ = model(el, hidden_)
    all_probs.append(batch_probs)

all_probs = torch.stack(all_probs).squeeze(1)
batch_probs, hidden = model(batch_x, model.init_hidden_state(32))
print(torch.equal(all_probs, batch_probs))
</code></pre>

username: ptrblck
cooked: <p>You could try to disable the cublas workspace by setting both sizes to zero via <code>=:0:0</code>, but I don’t think you can assume that potentially different algorithms will yield bitwise identical results.</p>
Torch.unique() without sorting on GPU

username: B.W_Zhang
cooked: <p>Is there some way that torch.unique() can return the unordered unique tensor on GPU?<br>
e.g. when inputting [6,2,7,2,6,5] I expect [6,2,7,5] . Or some other torch functions that can do this?<br>
I referred to <a href="https://discuss.pytorch.org/t/is-there-any-unique-function-which-keeps-the-order-of-occurence/90696">this</a> and <a href="https://discuss.pytorch.org/t/is-it-a-bug-for-torch-unique-on-gpu/33095/8">this</a> issue but didn’t get much help. The custom function given in the second link works but the efficiency is very low as I have a very long tensor.</p>
Cropping Nifti (3d volume) images by coordinates

username: nmvr
cooked: <p>Hello,</p>
<p>I’m trying to figure out a way to crop 3d Nifti images.<br>
They are loaded using the <a class="mention" href="/u/monai">@MONAI</a>  data loaders, and currently, my objective is to crop an area of the image.<br>
<em>e.g</em><br>
Say, the img/volume is 320x320*12 and I would like to remove the ‘border’ outside of the x- [100,220] and y-[90,220] of each slice of the volume.</p>
<p>I could do that manually using <code>numpy</code> or even torch if the image/volume is transformed into a tensor, but is there any built-in cropping tool either in <a class="mention" href="/u/monai">@MONAI</a> or maybe <code>torchvision</code> that given a <code>[xmin,ymin,xmax,ymax]</code> set of coordinates, like a bounding box, it would crop the image?</p>

username: ptrblck
cooked: <p>MONAI supports some transformations such as <a href="https://docs.monai.io/en/latest/transforms.html#monai.transforms.SpatialCrop"><code>monai.transforms.SpatialCrop</code></a>, which might be useful.<br>
Would any of them work for you?</p>

username: nmvr
cooked: <p>Looking at the descriptions they provide for the transforms it doesn’t look to be what I needed.</p>
<p>However, <code>SimpleITK</code> did have such a thing with the <code>RegionOfInterest</code> method which solved my problem.</p>
<pre><code class="lang-auto">image_ct = sitk.ReadImage(path_ct, sitk.sitkInt16)
origin_indx = [zmin, xmin, ymin] #crop area start
size_bounding_box = [(zmax-zmin), (xmax-xmin), (ymax-ymin)]
cropped_volume = sitk.RegionOfInterest(image_ct, size_bounding_box, origin_indx)
</code></pre>
How can I extend Detectron 2 (and PointRed) that detects (segments) COCO classes to detect 20 more new object classes?

username: Mona_Jalal
cooked: <p>Could you please kindly guide me on how I can extend Detectron 2 that already detects COCO classes to detect 20 more new object classes on top of 80 COCO classes?</p>
<p>How about <code>[PointRed](https://github.com/facebookresearch/detectron2/tree/master/projects/PointRend)</code> instance segmentation? How can I expand it such that it can create an instance segmentation mask for new object classes (20) besides the current 80 COCO classes?</p>
<p>Here’s an example of PointRed output for COCO object classes.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/28cc9b3eb9f156f13044c5f92669ae12a1045aff.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/28cc9b3eb9f156f13044c5f92669ae12a1045aff" title="Screenshot from 2021-04-21 01-45-34"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28cc9b3eb9f156f13044c5f92669ae12a1045aff_2_690x467.jpeg" alt="Screenshot from 2021-04-21 01-45-34" data-base62-sha1="5OVsTy1uSw6EQwzKzBV7HbyPZ2D" width="690" height="467" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28cc9b3eb9f156f13044c5f92669ae12a1045aff_2_690x467.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28cc9b3eb9f156f13044c5f92669ae12a1045aff_2_1035x700.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28cc9b3eb9f156f13044c5f92669ae12a1045aff_2_1380x934.jpeg 2x" data-dominant-color="778A95"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-04-21 01-45-34</span><span class="informations">2448×1658 762 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ul>
<li>This is a <a href="https://github.com/facebookresearch/detectron2/discussions/2945" rel="noopener nofollow ugc">double-post</a> but thought someone from here might know by chance.</li>
</ul>
How to apply PyTorch lr scheduler to my self-defining optimizer

username: sunyx523
cooked: <p>I want to apply torch.optim.lr_scheduler.CosineAnnealingWarmRestarts() to my self-defining optimizer. However, the error occurs TypeError: XXX is not an Optimizer(XXX is my custom optimizer). I creat my optimizer by modifying torch.optim.SGD and it works well. How to solve this problem?</p>

username: yjiang05
cooked: <p><a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py#L28" class="inline-onebox" rel="noopener nofollow ugc">pytorch/lr_scheduler.py at master · pytorch/pytorch · GitHub</a> shows that the TypeError will be raised when your customized optimizer is not an instance <code>Optimizer</code>.</p>
<p>Does your customized optimizer subclass the base class <code>Optimizer</code>?</p>

username: sunyx523
cooked: <p>Yes. My customized optimizer is an instance Optimizer. I directly modified the torch.optim.SGD code to create my customized optimizer.</p>

username: yjiang05
cooked: <p>You should subclass the base <code>Optimizer</code> class imported from <code>torch.optim.optimizer</code>, i.e., <code>from torch.optim.optimizer import Optimizer</code></p>

username: sunyx523
cooked: <p>Thanks! That works! I find that I import the old version optimizer:  from Grad_optimizer import Optimizer.</p>
UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape

username: aatiibutt
cooked: <p>Hi, I am implementing Mask-RCNN (following <a href="https://github.com/matterport/Mask_RCNN" class="inline-onebox" rel="noopener nofollow ugc">GitHub - matterport/Mask_RCNN: Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow</a>). However, while starting training, I get the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff858b3c9314cd727283b1f620e565cbc1655d57.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff858b3c9314cd727283b1f620e565cbc1655d57" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff858b3c9314cd727283b1f620e565cbc1655d57.png" alt="image" data-base62-sha1="AsrS1NybIL5waqfvtQS42HaE0sf" width="690" height="32" data-dominant-color="131313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1773×84 3.84 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I shall be grateful, if anybody help me out with this issue.</p>

username: ptrblck
cooked: <p>It seems the warning is raised by TensorFlow, so you might want to ask in their mailing list or on StackOverflow, as you might find more TF experts there <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Element-wise matrix multiplication along a certain dimension

username: import-antigravity
cooked: <p>Hi! I have an input of shape <code>(b, x, y)</code> and a weight matrix of shape <code>(x, y, y)</code>, where <code>b</code> is the batch size and <code>x</code> is a dimension I would like to also broadcast across. That is, I’d like to do the equivalent of the following for loop:</p>
<pre><code class="lang-python">def forward(x):
    # x.shape = (b, x, y)
    # self.W.shape = (x, y, y)
    out = torch.zeros((b, x, y))
    for i in x.shape[0]:
        for j in x.shape[1]:
            out[i, j] = torch.matmul(self.W[j], x[i, j])
    return out.sum(1) # Returns shape (b, y), summing over x
</code></pre>
<p>As you can see, the weight matrix is being batch multiplied on dimension zero, and element-wise multiplied on dimension 1. This would be somewhat similar to a convolution operation, where the kernel is the size of the entire input. I can’t figure out how to implement this, can anyone help?</p>
How to temporarily freeze a layer?

username: Jachilles
cooked: <p>I’d like to minimize</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b09cec17b5181f5a2ea7a20ea2965e564a993a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/3b09cec17b5181f5a2ea7a20ea2965e564a993a9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b09cec17b5181f5a2ea7a20ea2965e564a993a9_2_207x28.png" alt="image" data-base62-sha1="8qhbntP9E5dFjSFqMLpz7LfH1I5" width="207" height="28" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b09cec17b5181f5a2ea7a20ea2965e564a993a9_2_207x28.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b09cec17b5181f5a2ea7a20ea2965e564a993a9_2_310x42.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3b09cec17b5181f5a2ea7a20ea2965e564a993a9_2_414x56.png 2x" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">928×127 5.81 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>where</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a270a583ddc3be9f3af67bfc2b498a2fa6c6ec7.png" alt="image" data-base62-sha1="lZHaNol7gRT1eakwrhRDDOZmGgv" width="191" height="21">.</p>
<p>To be more precise, at each iteration, U receives the values of W_2. The gradient update at each iteration won’t change U.</p>
<p>To implement this in pytorch, I wrote</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
x = torch.randn([1,10])
layer1 = nn.Linear(10,10)
layer2 = nn.Linear(10,10)

y1 = layer2(layer1(x))
z2 = layer1(x)

with torch.no_grad():
    y2 = layer2(z2)



y = y1 + y2
loss = torch.norm(y)
print(y)
</code></pre>
<p>But I’m not sure if this is correct implementation of the above idea.</p>

username: TinfoilHat0
cooked: <p>Here’a code excerpt</p>
<pre><code class="lang-auto">model = // define your model here
for param in model.parameters():
     param.requires_grad = False
</code></pre>
<p>This freezes all the layers of the model. To freeze only a portion of it, you can do conditioning in the loop.</p>
Does FasterRCNN resize images internally before processing

username: Lakshay_Sharma
cooked: <p>In the FasterRCNN model, are the bounding box predictions made on a scaled image size? I’m asking because I would like to write some of my own evaluation code, and would like to know if I need to scale my ground truth-annotations accordingly.</p>

username: kareemamr
cooked: <p>On inspecting the Faster R-CNN model the applied transforms are:<br>
<code>  (transform): GeneralizedRCNNTransform(       Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])       Resize(min_size=(800,), max_size=1333, mode='bilinear')   )</code><br>
I do not know if it scales the corresponding bounding boxes accordingly as well or if that should be handled manually. Also the docs for <code>torchvision.transforms.Resize</code> show no <code>min_size</code> parameter so I don’t know if <code>800</code> is the acceptable lower bound and anything above stays the same or what.</p>
Help implementing Distributed Data Parallel?

username: Vivek_Reddy
cooked: <p>I am trying to utilize distributed data parallel to perform distributed training one node/machine with 4 GPUs.<br>
I think there is a problem with the saving/loading however.</p>
<p>I have the current code below:</p>
<blockquote>
<pre><code>     if config.local_rank == 0: 
        torch.save(net.state_dict(), os.path.join(config.output_directory, "latest_net.pytorch"))
        torch.save(optimiser.state_dict(), os.path.join(config.output_directory, "latest_optimiser.pytorch"))

        # barrier() to finish saving before loading
        torch.distributed.barrier() 
      map_location = {'cuda:%d' % 0: 'cuda:%d' % local_rank}

      try:
        net.load_state_dict(torch.load(os.path.join(config.output_directory, "latest_net.pytorch"), map_location=map_location))
        optimiser.load_state_dict(torch.load(os.path.join(config.output_directory, "latest_optimiser.pytorch"), map_location=map_location))
      except:
        print ('It couldn\'t load on local rank %d'%local_rank)
</code></pre>
</blockquote>
<p>For every epoch run, loss doesn’t decrease steadily as it does in a single thread.  I suspect this is because the saving and loading is not occuring properly.  I always get the printouts: It couldn’t load on rank 1, rank 2, and rank 3.    Can someone please provide some insight into what is blocking the saving loading?</p>
CrossEntropyLoss vs BCEWithLogitsLoss in multi-label classification

username: Sam_Shamsan1
cooked: <p>I am doing multi-label classification, when using CrossEntropyLoss I get this error:<br>
RuntimeError: 1D target tensor expected, multi-target not supported<br>
and when change it to BCEWithLogitsLoss I get this error:<br>
ValueError: Target size (torch.Size([1, 1])) must be the same as input size (torch.Size([1, 18]))</p>
<p>The label indices look as follow for the first 10 examples:<br>
[[0, 1], [0], [0], [0, 1], [0, 2], [0, 1], [0], [0, 1], [0], [0]]</p>
<p>I have 18 classes. Would you please let me know what I am doing wrong.</p>
<p>Thank you</p>

username: cpeters
cooked: <p>Assuming I’m remembering correctly, BCEWithLogitsLoss is expecting an all-ones encoding rather than a sparse representation, i.e. the target labels should be length 18 with each value representing one of the classes.</p>
Increase dataset size using Data Augmentation

username: Lucky_Magna
cooked: <p>Is there any way to increase dataset size using image augmentation in pytorch, like making copies of same images with variations like cropping or other techniques that are available in torchvision transforms.   I used the code mentioned below, but I want to oversample the dataset and check how that affects the models performance.</p>
<p>transform = {</p>
<pre><code>'train':

transforms.Compose([

    transforms.RandomResizedCrop(size=256, scale=(0.8, 1.0)),

    transforms.RandomRotation(degrees=15),

    transforms.ColorJitter(),

    transforms.RandomHorizontalFlip(),

    transforms.CenterCrop(size=224),  # Image net standards

    transforms.ToTensor(),

    transforms.Normalize([0.485, 0.456, 0.406],

                         [0.229, 0.224, 0.225])  # Imagenet standards

]),

'val':

transforms.Compose([

    transforms.Resize(size=256),

    transforms.CenterCrop(size=224),

    transforms.ToTensor(),

    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

]),

'test':

transforms.Compose([

    transforms.Resize(size=256),

    transforms.CenterCrop(size=224),

    transforms.ToTensor(),

    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

]),
</code></pre>
<p>}</p>
<p>Thankyou</p>

username: cpeters
cooked: <p>You mean sample more than the entire dataset per epoch? That’s functionally identical to simply increasing the number of epochs you have when you’re using random transformations.</p>

username: Lucky_Magna
cooked: <p>Well, actually yes, but not during each epoch instead I want to prepare a secondary dataloader with images that are double the size of my original dataloader and use them for training in each epoch. But your answer made me think of using different set of transforms in each epoch on the original dataset. Can I do that? This my be a silly doubt but please help me.</p>

username: cpeters
cooked: <p>You could initialise the dataloader inside the epoch loop with a different set of transforms for every epoch, yes.</p>

username: Lucky_Magna
cooked: <p>Does this regularize the model in any way? If so can you explain why?</p>

username: cpeters
cooked: <p>It probably provides regularisation the same as any other augmentation?</p>
<p>You seem to be suggesting something along the lines of</p>
<pre><code class="lang-auto">for i in epoch:
    if i % 2 == 0:
        transforms = first set of transforms
    else:
        transforms = second set of transforms
    # make dataloader with transform
    # train
</code></pre>
<p>which is functionally identical to just having half as many epochs and sequentially training it on the dataset with each set of transforms. It wouldn’t be as nicely shuffled, but if you’re using the same source images anyway it probably wouldn’t matter.</p>

username: Lucky_Magna
cooked: <p>Thanks a lot, that cleared a lot of things.</p>
Unity ML-Agents with older GPU(CC 3.0)

username: mcdenyer
cooked: <p>I am trying to use an ‘older’ gpu, gtx 660, to do some Machine Learning in Unity(ML-Agents).<br>
I just learned that the compute capability of my GPU is not supported by pytorch anymore.</p>
<p>Is there any way I can get a work around for this or is my GPU simply to old and if so what is a safe compute capability to get so that it wont be unsupported in a few months time after purchase?</p>

username: ptrblck
cooked: <p>The binaries are not shipping with this compute capability anymore, but you might try to build PyTorch from source using CUDA&lt;=10 as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>

username: mcdenyer
cooked: <p>I have gone through that tutorial but am lost at the part of installation for windows using Cuda.<br>
At that point is mentions you must build using NVTX(Nsight Compute) and all the correction versioning but does not explicitly explain how to “build”.  I don’t think I am advanced enough to follow that tutorial, do you know of any other build from source walkthroughs  on windows that will baby me through it more?</p>

username: mcdenyer
cooked: <p>I am just going to return my GPU since I just got it.  Is it risky to get a GPU with cc 3.5 as it may not continue to be supported?  I am only hoping that the GPU I get is supported for the next 8 months ish.</p>

username: ptrblck
cooked: <p>I don’t know the exact roadmap for the binaries and wheels and thus cannot comment on future support. Also, note that I can’t comment which GPU you should buy (as I’m working at NVIDIA) and would thus recommend to check public benchmarks etc.</p>
Segmentation fault after killing processes

username: ppower1
cooked: <p>I am training multiple feedfoward neural networks in parallel via ray. Most of the time the code runs with no problems. Occasionally, though, the code takes a while to run, so I end up killing it. When I do though, I get a segmentation fault when I try to re-run the code. Specifically, I am unable to run any python script where I “import torch”. I am able to execute other python scripts that don’t use pytorch though. I am unsure as to whether this is an issue with my computer, ray, or pytorch.</p>
<p>Thanks</p>