akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CyclicLR scheduler

username: Jin_Shan_Ng
cooked: <p>hi guys! i’m new to pytorch and was wondering how does a Scheduler work with an Optimizer? To my understanding, the scheduler changes the LR of the optimizer right? say a SGD optimizer has a learning rate of 0.01 and i have my CyclicLR scheduler with a base of 1e-6 and max of 0.006, how does the learning rate of the optimizer change? I am confused as to how this works when the LR of the optimizer is not in the range of the base/max LR of the scheduler.</p>
<p>or must the LR of the optimizer be in the range of the base LR and max LR of the cyclicLR? thank you in advance</p>

username: ptrblck
cooked: <p>The scheduler manipulates and thus defines the learning rate of the passed optimizer as seen here:</p>
<pre><code class="lang-python">model = nn.Linear(3*224*224, 10)
optimizer = optim.SGD(model.parameters(), lr=1.)
print(optimizer.param_groups[0]['lr'])
scheduler = optim.lr_scheduler.CyclicLR(
    optimizer, base_lr=1e-3, max_lr=1e-2, step_size_up=5)


for epoch in range(10):
    print('Epoch {}, optim.lr {}, scheduler.lr {}'.format(
        epoch, optimizer.param_groups[0]['lr'], scheduler.get_last_lr()))
    optimizer.step()
    scheduler.step()
</code></pre>
"RuntimeError: 'inputs' to backward is not a leaf Tensor" in torch.autograd.backward while IS a leaf tensor

username: Shiny_Sheepy
cooked: <p>Hi,</p>
<p>I am trying some experiments on autograd and found this following small example gives a RuntimeError: One of the differentiated Tensors given as ‘inputs’ to backward is not a leaf Tensor.</p>
<p>However, printing <code>x.is_leaf</code> shows <code>True</code>. So why is this error thrown?</p>
<pre><code class="lang-python">
import torch
import torch.nn as nn
import torch.nn.functional as F

class TestModule(nn.Module):
    def __init__(self, input_size, output_size):
        super().__init__()
        internal_size = 5
        self.linear1 = nn.Linear(input_size, internal_size)
        self.linear2 = nn.Linear(internal_size, output_size)
        self.model = nn.Sequential(self.linear1, nn.ReLU(), self.linear2)

    def forward(self, x):
        x = self.model(x)
        return x

model = TestModule(4, 3)

x, y = torch.rand(size=(20, 4)), torch.randint(low=0, high=3, size=(20,))
x.requires_grad = True
loss = F.cross_entropy(model(x), y)

print(x.is_leaf)
torch.autograd.backward(loss, create_graph=False, inputs=x)

</code></pre>

username: ptrblck
cooked: <p>I cannot reproduce the error using <code>1.9.0a0+git6d45d7a</code> and get a valid gradient in <code>x</code>, so you might need to update to the latest nightly (or the stable release in case you are using an older version).</p>

username: Shiny_Sheepy
cooked: <p>I am using the stable torch-1.8.1 version. So this is a bug that will be fixed in 1.9.0 then. Strange that how such a simple backward pass is still bugged in today’s pytorch.</p>
Creating a tensor mask based of rate/percentage/ratio

username: KyleOng
cooked: <p>Good day,</p>
<p>How can I generate a mask tensor that has a specific ratio of 0 and 1? For example <code>70:30</code> of 0s and 1s in a 5 by 10 tensor will generate</p>
<pre><code>[[0,0,0,0,0,0,0,0,1,0],
[1,1,1,0,0,0,1,1,0,0],
[0,0,1,0,0,1,1,0,1,0],
[0,1,0,1,0,1,1,0,0,0],
[0,0,0,1,0,0,0,0,0,0]]
</code></pre>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>You could create the values using the defined number of samples beforehand, shuffle them, and reshape to the desired output:</p>
<pre><code class="lang-python">ones, zeros = 30, 70
x = torch.cat((torch.zeros(zeros), torch.ones(ones)))
x = x[torch.randperm(x.size(0))]
x = x.view(10, 10)
print(x)
&gt; tensor([[0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
          [1., 0., 0., 0., 0., 0., 1., 1., 1., 1.],
          [1., 0., 0., 0., 0., 0., 1., 0., 0., 0.],
          [0., 0., 1., 1., 0., 0., 1., 0., 1., 0.],
          [0., 1., 0., 0., 0., 1., 1., 0., 0., 0.],
          [0., 0., 0., 1., 1., 0., 0., 0., 1., 1.],
          [1., 0., 0., 0., 0., 0., 1., 1., 0., 0.],
          [0., 0., 0., 0., 1., 1., 0., 0., 0., 0.],
          [0., 1., 1., 0., 0., 0., 1., 0., 0., 0.],
          [1., 0., 0., 0., 0., 0., 1., 0., 1., 0.]])
</code></pre>

username: KyleOng
cooked: <p>It works like a charm. Thank you very much.</p>
Correct way to pass multiple samplers

username: Sayak_Paul
cooked: <p>Along with <code>DistributedSampler</code> what if I wanted to pass another sampler like so to <code>DataLoader</code>?</p>
<pre><code class="lang-python">class PneumoSampler(Sampler):
    def __init__(self, train_df, positive_perc=0.8):
        assert positive_perc &gt; 0, 'percentage of positive pneumothorax images must be greater then zero'
        self.train_df = train_df
        self.positive_perc = positive_perc
        self.positive_idxs = self.train_df.query('has_mask==1').index.values
        self.negative_idxs = self.train_df.query('has_mask!=1').index.values
        self.n_positive = len(self.positive_idxs)
        self.n_negative = int(self.n_positive * (1 - self.positive_perc) / self.positive_perc)
        
    def __iter__(self):
        negative_sample = np.random.choice(self.negative_idxs, size=self.n_negative)
        shuffled = np.random.permutation(np.hstack((negative_sample, self.positive_idxs)))
        return iter(shuffled.tolist())

    def __len__(self):
        return self.n_positive + self.n_negative
</code></pre>
<p>The sampler is taken from <a href="https://github.com/amaarora/amaarora.github.io/blob/master/nbs/Training.ipynb" rel="noopener nofollow ugc">here</a>.</p>

username: Sayak_Paul
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any ideas here?</p>
LGBFS not converge on GPU but CPU works fine

username: qcyza
cooked: <p>I am running some training, and the code is as in <a href="https://github.com/jayroxis/PINNs/blob/master/Burgers%20Equation/Burgers%20Inference%20(PyTorch).ipynb" class="inline-onebox" rel="noopener nofollow ugc">PINNs/Burgers Inference (PyTorch).ipynb at master · jayroxis/PINNs · GitHub</a><br>
I found that the training has very different behaviors on CPU (converge to small values as shown in the repo) and GPU (not converge - stop at loss=0.08).</p>
<p>May I ask for possible reasons for this?</p>
Request for make PyTorch more multithread friendly - make pseudo r.n. generators part of the model

username: Konstantin_Burlachen
cooked: <p>Hello. I would like to ask to append into PyTorch mechanism to control the seed of random number generators. For people who try use PyTorch from the multi-threaded application, they can face the problem that if several threads performs model initialization with randomized schemas (He initialization, Lecun initialization, etc.) it’s very likely that models this initialization is really a part of global state for which users from Python API has only access via functions like TORCH.MANUAL_SEED.</p>
<p>It’s better to redesign, and make random generator state as a part of the model I think.</p>
<hr>
<p>Code snippet which demonstates problems with global states for numpy.random and Python’s random.</p>
<hr>
<p>#!/usr/bin/env python3</p>
<p>import numpy as np<br>
import random<br>
import threading, time</p>
<p>class WorkerThread(threading.Thread):</p>
<pre><code>def __init__(self, i, sleep_seconds):
    threading.Thread.__init__(self)
    self.th_number = i
    self.sleep_seconds = sleep_seconds

def run(self):
    np.random.seed(123)
    random.seed(123)

    time.sleep(self.sleep_seconds)
    print(self.th_number, np.random.random(), "(np.random)")
    print(self.th_number, random.random(), "(random)")
</code></pre>
<p>th = [WorkerThread(k, 1*k) for k in range(3)]<br>
for t in th: t.start()<br>
for t in th: t.join()</p>
<hr>

username: ptrblck
cooked: <p>This known limitation is caused by the <code>fork</code>ed subprocesses and third party libraries as given by your code snippet and explained in the <a href="https://pytorch.org/docs/stable/notes/faq.html#my-data-loader-workers-return-identical-random-numbers">FAQ</a> as well as the <a href="https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading">Randomness docs</a>.<br>
While the <code>DataLoader</code> already sets the seed for the <code>random</code> module (as well as PyTorch), note that <code>numpy</code> is not a requirement, which is why previous suggestions to seed third party libraries in PyTorch’s code were declined.</p>

username: Konstantin_Burlachen
cooked: <p>Thanks, that you are talking about is very interesting and thanks for reference, but I did not mean DataLoaders and forking strategy during having parallel threads.</p>
<p>I mostly care about the initialization of the models and have the ability to specify the initialization of the model in a thread-safe way.</p>
<p>So my suggestion append into Public API this controlling seeds technics at the level of Models, if it is possible for cases when people do not use <code>DataLoader</code>'s.</p>
<p>Very often for low-level libraries - memory and logging is user-specfied callbacks, but for PyTorch in my opinion this random seeds should be controlled with allow user to control and it’s important thing for me as user.</p>
Is there any way around this problem?

username: Tony_T
cooked: <p>I have a dataset with over 1000 classes with just 10 images in each classes. In the end, I have to predict if any new image is part of the dataset or not.I tried implementing with my own neural network, tried Siamese networks. My model goes up to 72% val accuracy but shows wrong outputs.I’ve been on this for days.Could you guys help me out ?</p>

username: m3tobom_M
cooked: <p>Are you applying proper pre-process to input?</p>

username: eduardo4jesus
cooked: <p>Would the images for testing be exactly the same?</p>
Why can NOT save leaf tensor using ctx.save_for_backward?

username: Wu_jiang
cooked: <p>when  saving leaf tensor using  ctx.save_for_backward :</p>
<pre><code class="lang-auto">class DCNFunction(Function):
def forward(ctx, input, weight, Offset, Mask,
                num_deformable_group,kernel, pad, stride, dilation,
                use_bias , bias):
       ...
       ...
       ctx.save_for_backward(input,weight, bias, Offset, Mask, col)
       ...
def backward(ctx, grad_out):
       ...
       ...
       input, weight, bias, Offset, Mask, col = ctx.saved_tensors
       ...      
</code></pre>
<p>Encountered an error：</p>
<pre><code class="lang-auto">  File "dcnv2.py", line 54, in backward
    input, weight, bias, Offset, Mask, col = ctx.saved_tensors
RuntimeError: No grad accumulator for a saved leaf!
</code></pre>
<p>weight and bias is self defined nn.Parameter , they are leaf node in graph</p>
<pre><code class="lang-auto">self.weight = torch.nn.Parameter(data=torch.empty(planes,inplanes,kernel,kernel),requires_grad= phase=='TRAIN')
</code></pre>
<pre><code class="lang-auto">ipdb&gt; ctx.needs_input_grad
(True, True, True, True, False, False, False, False, False, False, True)
ipdb&gt; ctx.next_functions
((&lt;AccumulateGrad object at 0x7fa0fc5a0c18&gt;, 0), (&lt;AccumulateGrad object at 0x7fa0fc5a0c50&gt;, 0), (&lt;CudnnConvolutionBackward object at 0x7fa0fc5a0c88&gt;, 0), (&lt;MulBackward0 object at 0x7fa0fc5a0cc0&gt;, 0), (&lt;AccumulateGrad object at 0x7fa0fc5a0cf8&gt;, 0))
</code></pre>
<p>when changing code(no error):</p>
<pre><code class="lang-auto">class DCNFunction(Function):
def forward(ctx, input, weight, Offset, Mask,
                num_deformable_group,kernel, pad, stride, dilation,
                use_bias , bias):
       ...
       ...
       ctx.save_for_backward(input, Offset, Mask, col)
       ctx.weight = weight
       ctx.bias = bias 
       ...
def backward(ctx, grad_out):
       ...
       ...
       weight = ctx.weight
       bias = ctx.bias 
       input, weight, bias, Offset, Mask, col = ctx.saved_tensors
       ...      
</code></pre>
<p>using pytorch1.0<br>
wonder why can’t save leaf tensor using ctx.save_for_backward？<br>
and<br>
comparing with ctx.save_for_backward, the disadvantage of using Temporary variables Eg.ctx.weight ?<br>
thx</p>

username: eduardo4jesus
cooked: <p>I never had this problem in <code>torch 1.8.1+cu101</code>, I use to as</p>
<pre><code class="lang-auto">ctx.save_for_backward(input, weight, bias)   
</code></pre>
Error in code CNN

username: randinoo
cooked: <pre><code class="lang-auto">class UNET(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()

        self.conv1 = self.contract_block(in_channels, 32, 7, 3)
        self.conv2 = self.contract_block(32, 64, 3, 1)
        self.conv3 = self.contract_block(64, 128, 3, 1)

        self.upconv3 = self.expand_block(128, 64, 3, 1)
        self.upconv2 = self.expand_block(64*2, 32, 3, 1)
        self.upconv1 = self.expand_block(32*2, out_channels, 3, 1)

    def __call__(self, x):

        # downsampling part
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)

        upconv3 = self.upconv3(conv3)

        upconv2 = self.upconv2(torch.cat([upconv3, conv2], 1))
        upconv1 = self.upconv1(torch.cat([upconv2, conv1], 1))

        return upconv2

    def contract_block(self, in_channels, out_channels, kernel_size, padding):

        contract = nn.Sequential(
            torch.nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.ReLU(),
            torch.nn.Conv2d(out_channels, out_channels, kernel_size=kernel_size, stride=1, padding=padding),
            torch.nn.BatchNorm2d(out_channels),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
                                 )

        return contract

    def expand_block(self, in_channels, out_channels, kernel_size, padding):

        expand = nn.Sequential(torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=padding),
                            torch.nn.BatchNorm2d(out_channels),
                            torch.nn.ReLU(),
                            torch.nn.Conv2d(out_channels, out_channels, kernel_size, stride=1, padding=padding),
                            torch.nn.BatchNorm2d(out_channels),
                            torch.nn.ReLU(),
                            torch.nn.ConvTranspose2d(out_channels, out_channels, kernel_size=3, stride=2, padding=1, output_padding=1) 
                            )
        return expand

</code></pre>
<p>error<br>
Sizes of tensors must match except in dimension 1. Got 4 and 3 in dimension 2 (The offending index is 1)</p>
<p>help me</p>
Can I run experiment on google Colab (mount drive)and shut down the computer and waiting the result?

username: Ximing_Dong11104
cooked: <p>since each epoch will run 30 mins and I may be run it at night, but I can not wait for it to finish. can I run it and then close the computer and see the result in the moring?</p>

username: eduardo4jesus
cooked: <p>I don’t think it is guaranteed to be connected to that same session.</p>
<p>On such case, I would cache the stage every couple epochs and just leave the machine on. If you get disconnected, at least you have the cache and you could continue training from there.</p>
<p>Also, as far as I know, you need you machine to be active. From time to time Colab asks the user to interact with the session. Though there are some javascript snippets (e.g. <a href="https://stackoverflow.com/questions/54057011/google-colab-session-timeout" rel="noopener nofollow ugc">here</a>  and <a href="https://gist.github.com/RodolfoFerro/a12c664330c6eae601e5a828adfb8306" rel="noopener nofollow ugc">here</a>) to do that interaction manually, the computer would still need to be on.</p>

username: Ximing_Dong11104
cooked: <p>but I run it in the mount drive, not web notebook, is the same thing answer?</p>

username: eduardo4jesus
cooked: <p>Having the drive mounted does not change the requirements for interactivity from Colab. However, one of the biggest advantages of using the Drive is to save data in the mounted paths, so that you don’t lose that data when the Virtual Machine of the current session gets reset.</p>

username: Ximing_Dong11104
cooked: <p>thanks you very much</p>
Autoencoders for Anomaly detection [Cost function + Predict_Proba]

username: doctore
cooked: <p><strong>Intro:</strong> I have a dataset where instances are in the form of time series, but I’m generally interested in solving instance-wise (is instance anomaly or not) anomaly detection problems with different types of autoencoders such as plain-autoencoder, GRU-based, LSTM-based autoencoders, etc.</p>
<p><strong>Question 1</strong><br>
Which is the best/recommanded cost function for autoencoders on the anomaly detection problem and why?</p>
<ul>
<li>
<p><strong>Binary Cross Entropy Loss</strong> (BCELoss)<br>
From documentation target y have to be normalized into [0…1], which is usually done with Sigmod on the last layer of decoder, but I guess… I have to normalize input also into [0…1] before training. That means I have to save the min and max calculated with MinMaxScaler from Sklearn and use it in prediction before applying autoencoder. Is there some more elegant way to do the same normalization with PyTorch and save it with PyTorch model together?</p>
</li>
<li>
<p><strong>Mean Squared Error Loss</strong> (MSELoss)<br>
Here I believe that I don’t need normalization nor Sigmoid at the end.</p>
</li>
<li>
<p><strong>Or …</strong> is there some other more appropriate cost function for this problem, related to time series or not?</p>
</li>
</ul>
<p><strong>Question 2</strong><br>
How to implement predict_proba method for this type of autoencoders?</p>
<ul>
<li>I understand that I need to set a threshold that will be used in a way to predict 1 (outlier) if the difference between input and predicted values is greater than the threshold and otherwise 0 (inlier), but I’m not sure how to implement this efficiently and how to calculate the probablities.</li>
</ul>
Backward hooks are called after being removed?

username: Shiny_Sheepy
cooked: <p>Hi,</p>
<p>I come across this situation that even though my backward hooks are cancelled by calling <code>handle.remove()</code>, they are still being triggered during backward pass.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F


def _make_encoder_layer(in_channels, out_channels, kernel_size=3):
    conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=1)
    pool = nn.MaxPool2d(2, stride=2)
    norm = nn.BatchNorm2d(out_channels)
    act = nn.ReLU()

    return nn.Sequential(conv, pool, norm, act)


def print_hook(module: nn.Module, _inputs, _outputs):
    print('hook triggered on', module)


class TestModule(nn.Module):
    def __init__(self):
        super(TestModule, self).__init__()
        self.encoder = _make_encoder_layer(1, 3)
        self.grad_handles = []

    def forward(self, x):
        self._register_backward_hooks()

        x.requires_grad = True
        h = self.encoder(x)

        print("Calling backward when hooks are enabled.")

        torch.autograd.backward(h.mean(), create_graph=True)
        # do something with grad
        blah = x.grad
        self.zero_grad()

        self._remove_backward_hooks()
        return h


    def _register_backward_hooks(self):
        # Iterate through layers
        for m in self.encoder.modules():
            if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear) or isinstance(m, nn.BatchNorm2d):
                handle_g = m.register_full_backward_hook(print_hook)
                self.grad_handles.append(handle_g)
                print(f"{m} registered.")
        print("")

    def _remove_backward_hooks(self):
        for h in self.grad_handles:
            h.remove()
        self.grad_handles = []

model = TestModule()
x = torch.rand(size=(32, 1, 128, 128))

p = model(x)

print("Calling backward after hooks are removed.")
p.mean().backward()

</code></pre>
<p>Running this code gives the following output:</p>
<pre><code class="lang-auto">Conv2d(1, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) registered.
BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) registered.

Calling backward when hooks are enabled.
hook triggered on BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
hook triggered on Conv2d(1, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Calling backward after hooks are removed.
hook triggered on BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
hook triggered on Conv2d(1, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
</code></pre>
<p>Is this behaviour intended, or is this a bug?</p>
Dataset creation from live data feed

username: spi
cooked: <p>Hello,</p>
<p>I am trying to create an end-to-end pipeline of models that can do OCR in a “standard” way: a first model receives an image, detects lines, outputs line coordinates; a second model takes images + lines, crops lines out, and does the text detection. My question is how to make the data pipeline for the second model, so to keep latency as low as possible, high GPU load, while being able to accept new files constantly.</p>
<p>As it is rather CPU-heavy (things like non-max suppression and image rotation do take time), I need multiprocessing. I have tried the following pseudocode-setup for <strong>getitem</strong>:</p>
<ul>
<li>if there is a crop in the buffer, return it</li>
<li>otherwise, look for images to crop from;
<ul>
<li>if you find one (use file locks to avoid interactions with other workers and), crop all the lines in that image and return the first, add the rest to the buffer</li>
<li>otherwise sleep 10ms and check again</li>
</ul>
</li>
</ul>
<p>But this creates two issues for me:</p>
<ul>
<li>The main one is that if I have, say, 4 workers and one of them doesn’t have any file left to process, the dataloader will dutifully wait for it to end without asking data from the other workers. So at any time there will be three unfinished files left until a new one arrives. I could partially fix this by adding crops to a “common” buffer across all workers saving crops on disk, but (1) passing through the disk is very suboptimal and (2) it will only reduce the problem from having “3 files waiting” to having “3 lines waiting”, but that still means at least one file left unprocessed…</li>
<li>The second problem, similar but more easily solvable with some hacking, comes in with batching. Even if the above is solved, I would at some point have an incomplete batch of lines and no further file to process. In that case it should be more efficient to pad to a full batch and disregard the final images, but I guess that to do that I have to disable batching and do it by hand inside the inference code (i.e. loop until I either get N images if at least one worker is working, or pad if all workers are waiting or after a small timeout), or is there a cleaner way?</li>
</ul>
<p>So, long story short: is there a command to make a dataset worker tell the data loader “skip me, go to the next worker”? Like a specific exception that could be raised and somehow caught inside the data loader?</p>
<p>Thanks in advance for any hint!</p>
How to get around `'Proxy' object does not support item assignment`

username: Alexander_Soare
cooked: <p>Here’s the line that triggers this error during <code>torch.quantization.quantize_fx.prepare_fx</code></p>
<pre><code class="lang-auto">patch_embed[:, 1:] = patch_embed[:, 1:] + self.proj(self.norm1_proj(pixel_embed).reshape(B, N - 1, -1))
</code></pre>
<p>Is there some way to get around this without having to avoid item assignment? Can I somehow inform that particular proxy instance that this should be okay?</p>

username: tom
cooked: <p>You would likely want to lose the indexing on the left hand side (which makes the assignment necessarily in-place). How to do that depends on more context than shown here - as shown here, you might just keep the first column of patch_embed separately or so.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Alexander_Soare
cooked: <p>Thanks for your response!</p>
<p>Well, I kind of feel like I got away with a crime, but this didn’t throw the error:</p>
<pre><code class="lang-auto">patch_embed_rest = patch_embed[:, 1:]
patch_embed_rest += self.proj(self.norm1_proj(pixel_embed).reshape(B, N - 1, -1))
</code></pre>

username: tom
cooked: <p>Cool! Likely this is similar to “when can I use inplace without autograd complaining”… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Alexander_Soare
cooked: <p><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> hope you don’t mind I marked my answer as the solution. Thanks again</p>
What to do with with a tensor(1d array of) cosine similarity values

username: SANKALP_SHUBHAM
cooked: <p>I am performing cosine similarity (nn.cosineSimilarity()) between two 2D tensors (of same shape of course). Now, the resultant output is a 1D tensor which contains n single tensors. These single tensors are the pairwise cosine similarities.</p>
<p>Now, my question what can I do with these pairwise cosine similarities. For training, I am passing them directly to my custom loss function and things seems to work fine.</p>
<p><strong>For Prediction</strong></p>
<p>I have M such outputs out of which I have to choose the one with max score/output. But how can I select a max from arrays.</p>
<p>My approach for :<br>
I am flattening the 2d tensors (which are outputs from a GRU) and then calculating the cosine sim thus, getting a single tensor. This approach seems to work but the accuracy of the model is not as expected.<br>
Hence, I am looking for other alternatives.</p>
<p>Another thing I had tried was to just select the array with the maximum single tensor among all.</p>
<p><strong>What can I do with an array of pairwise similarities?</strong></p>
Backpropagating through scatter_add_

username: Wenjie_Qiang
cooked: <p>Hey, I am trying to move a certain point coordinate through the offset output by the network ONet, and then calculate the loss with the moved point map and ground truth. This process backpropagates and updates the ONet parameters.</p>
<p>Here is a brief part of the code.</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
import numpy as np

c,w,h = img.shape

ori_point = Variable(torch.from_numpy(np.array([10,10], dtype=np.float32)), requires_grad=True)

offset = ONet(img)  # (b,1,2)

point_h = offset[0,0,0] + ori_point[0]
point_w = offset[0,0,1] + ori_point[1]

point_index = point_h * w + point_w

dot_map = torch.zeros(w * h).scatter_add_(0, index=point_index.long(), src=torch.ones(w * h)).view(h, w)

loss = criterion(dot_map, gt)
loss.backward()
...

</code></pre>
<p>I want to use scatter_add_() to map the point coordinates to a dot map, but my point coordinates are in the form of index, so when the loss is backpropagated, it seems that the parameters of Onet will not be updated.</p>
<p>What is the correct way of doing this?</p>

username: Wenjie_Qiang
cooked: <pre><code class="lang-auto">x = conv(x)
p.scatter_add_(2, index=x, data)
</code></pre>
<p>Simply put, will gradients flow correctly to the weights of conv, if the loss is calculated using p?</p>

username: KFrank
cooked: <p>Hi Wenjie!</p>
<aside class="quote no-group" data-username="Wenjie_Qiang" data-post="2" data-topic="122866" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wenjie_qiang/48/22459_2.png" class="avatar"> Wenjie_Qiang:</div>
<blockquote>
<p>will gradients flow correctly to the weights of conv</p>
</blockquote>
</aside>
<p>No.  <code>scatter_add_()</code> requires that its <code>index</code> argument be a tensor<br>
of type <code>int64</code>.  Gradients don’t (and can’t) flow back through integers.</p>
<p>(Gradients will flow back through <code>self</code> (your <code>p</code>) and <code>src</code> (your <code>data</code>)<br>
if they have <code>requires_grad = True</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>
Flutter w/ PyTorch Integration

username: Bilal
cooked: <p>Hello Forum!,</p>
<p>I’ve been looking around for some repos of flutter w PyTorch integration to get a know about of the integration of it in the pipeline of an application. Please share any repos here, the more the better.</p>
<p>Thanks<br>
-Bilal <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Natural Image vs Real Image vs Image?

username: Sahil_malik
cooked: <p>Hi,<br>
Could we say the image a natural image collected from internet sources flickers, google , and many other source.</p>
<p>Looking forward to your kind response.</p>
<p>Many Thanks!</p>

username: ptrblck
cooked: <p>Of course it depends on your own interpretation of what “natural” images are, but I would not call all images from Flickr, Google, etc. “natural”. I’ve seen some definitions of “natural” images as images captured by a camera/smartphone/etc. datasets such as CIFAR or ImageNet come to mind.</p>

username: Sahil_malik
cooked: <p>Thank you for your response.</p>
<p>Would you call MIT-Indoor-67 and Event-8 datasets  images natural images? As I have read in some papers calling MIT-Indoor-67 natural images.</p>

username: ptrblck
cooked: <p>Yes, I would probably call these dataset natural, but it would also depend on the setting, i.e. what you are comparing them against.</p>

username: Sahil_malik
cooked: <p>Thank you so much… I am comparing these datasets with Fashion-MNIST. … What we call Fashion-MNIST images then?</p>
CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm`

username: yakhyo
cooked: <p>Installed CUDA version:</p>
<ol>
<li><strong>NVIDIA-SMI 460.27.04</strong></li>
<li><strong>Driver Version: 460.27.04</strong></li>
<li>
<strong>CUDA Version: 11.2</strong><br>
OS:</li>
</ol>
<ul>
<li>Ubuntu 20.04</li>
</ul>
<p><code>&gt;&gt;&gt; python3 -m torch.utils.collect_env</code></p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.8.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 460.27.04
cuDNN version: Probably one of the following:
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] torch==1.8.1+cu111
[pip3] torchaudio==0.8.1
[pip3] torchvision==0.9.1+cu111
[conda] Could not collect
</code></pre>
<p>I am getting following error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`

</code></pre>

username: ptrblck
cooked: <p>Try to install the <code>1.8.1</code> conda binaries or any nightly release, which should have fixed this issue.</p>
Performance drop when using ConvTranspose2d

username: pytorchbearer
cooked: <p>Dear all,</p>
<p>I have a very standard U-Net model for medical segmentation. It seems that when I change some of the Conv2d (with kernel_size = 3, padding = 1 and stride = 1) to ConvTranspose2d (with the same arguments) then there is a marked performance loss (a significant drop in Dice as well as Hausdorff distance).</p>
<p>I could post some code later if needed, but is there any fundamental reason why ConvTranspose2d should work differently than Conv2d?</p>
<p>Torchbearer</p>

username: ptrblck
cooked: <p>I’m unsure if I misunderstand the question, but <code>nn.Conv2d</code> and <code>nn.ConvTranspose2d</code> are working differently as described e.g. in <a href="https://arxiv.org/abs/1603.07285">this conv arithmetic guide</a>.</p>
Error during GAN model evaluation

username: Najeh_Nafti
cooked: <p>I got this error while trying to evaluate my GAN model:<br>
Code:</p>
<pre><code class="lang-auto">fpr, tpr, _ = roc_curve(labels, img_distance)
precision, recall, _ = precision_recall_curve(labels, img_distance)
roc_auc = auc(fpr, tpr)
pr_auc =  auc(recall, precision)
</code></pre>
<p>Output Error:</p>
<pre><code class="lang-auto">C:\Users\nafti\anaconda3\envs\st\lib\site-packages\sklearn\metrics\_ranking.py:943: UndefinedMetricWarning: No negative samples in y_true, false positive value should be meaningless
  UndefinedMetricWarning)
</code></pre>
<p>How can I solve it?</p>

username: ptrblck
cooked: <p>This warning is created by <code>sklearn</code>, as it seems that the target array doesn’t contain any negative samples, and thus the metric calculation would be meaningless.<br>
You could check <code>labels</code> and make sure that it contains positive and negative samples.</p>

username: Najeh_Nafti
cooked: <p>I have positive and negative samples but I still got the same problem</p>

username: ptrblck
cooked: <p>Could you post the label tensor (or just a part of it containing the positive and negative samples) so that we could reproduce this issue, please?</p>

username: Najeh_Nafti
cooked: <p>I train the model on healthy patient and trying to test it on unhealthy patients.<br>
but labels read only positive samples!</p>
<pre><code class="lang-auto">array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1])
</code></pre>

username: ptrblck
cooked: <p>This would explain the raised warning, since scikit-learn wouldn’t be able to calculate the metrics.<br>
Is this a new issue now, as you’ve previously mentioned that you have both, positive and negative, labels?</p>

username: Najeh_Nafti
cooked: <p>My dataset contain both positive and negative but cannot be read, I think a problem in data splitting,<br>
do have you any idea how can I split data in that case (train: healthy data and test: healthy and unhealthydata) because i did it manually but doesn’t work</p>
Understanding reshape

username: tomcc
cooked: <p>For a tensor:</p>
<pre><code class="lang-auto">x = torch.tensor([
    [
        [[0.4495, 0.2356],
          [0.4069, 0.2361],
          [0.4224, 0.2362]],
                   
         [[0.4357, 0.6762],
          [0.4370, 0.6779],
          [0.4406, 0.6663]]
    ],    
    [
        [[0.5796, 0.4047],
          [0.5655, 0.4080],
          [0.5431, 0.4035]],
         
         [[0.5338, 0.6255],
          [0.5335, 0.6266],
          [0.5204, 0.6396]]
    ]
])
</code></pre>
<p>Firstly would like to split it into 2 (<code>x.shape[0]</code>) tensors then concat them. Here, i dont really have to actually split it as long as i get the correct output, but it makes a lot more sense to me visually to split it then concat them back together.</p>
<p>For example:</p>
<pre><code class="lang-auto"># the shape of the splits are always the same
split1 = torch.tensor([
    [[0.4495, 0.2356],
    [0.4069, 0.2361],
    [0.4224, 0.2362]],

    [[0.4357, 0.6762],
    [0.4370, 0.6779],
    [0.4406, 0.6663]]
])
split2 = torch.tensor([
    [[0.5796, 0.4047],
    [0.5655, 0.4080],
    [0.5431, 0.4035]],

    [[0.5338, 0.6255],
    [0.5335, 0.6266],
    [0.5204, 0.6396]]
])

split1 = torch.cat((split1[0], split1[1]), dim=1)
split2 = torch.cat((split2[0], split2[1]), dim=1)
what_i_want = torch.cat((split1, split2), dim=0).reshape(x.shape[0], split1.shape[0], split1.shape[1])
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30c7681223271291cfab8c6bf439bcc77bb22474.png" alt="image" data-base62-sha1="6Xw8ZK9ajJIEwsOIUy9ICYTC0tK" width="655" height="481"></p>
<p>For the above result, i thought directly reshaping <code>x.reshape([2, 3, 4])</code> would work, it resulted in the correct dimension but incorrect result.</p>
<p>In general i am:</p>
<ol>
<li>not sure how to split the tensor into <code>x.shape[0]</code> tensors.</li>
<li>confused about how <code>reshape</code> works. Most of the time i am able to get the dimension right, but the order of the numbers are always incorrect.</li>
</ol>
<p>Thank you</p>

username: googlebot
cooked: <p>reshape is an alias for contiguous().view(), these command:<br>
1)copy data, synchronizing physical format (i.e. one for sequential memory reading as indexes increase) with logical one (for example, changed by permute())<br>
2)change strides, i.e. dimension split points, but still maintain contiguous nested blocks format</p>
<p>as you can see, logical format is not affected, i.e. no reordering is done. in contrast, cat() with dim&gt;0 produces interleaved data, so it is different. better alternative to cat() is permute().reshape()</p>
How to segment future a frame with single channel using past multiple input frames with multiple channels satellite image?

username: bugsuse
cooked: <p>Hi all!</p>
<p>I’m relatively new to PyTorch and I want to segment the next frame of image based on multiple input frames with multiple channels satellite image. i.e. input shape (batch, time_frames, satellite_channels, height, width), such as time_frames is 5, satellite_channels is 7, and output shape (batch, 1, 1, height, width).</p>
<p>I want to achieve the task using unet model, such as unet3d, but I’m not sure how to represent the data in a way that makes sense in PyTorch. Is there any known way to do this?</p>
<p>I’m considering whether to use the satellite channel dimension as depth or the time frame as depth in Conv3d. If the time frame is used as depth in Conv3d, how to ensure that the depth size of output is 1? Is it using the kernel size parameter?</p>
<p>Thank you in advance for any helpful</p>
Any method to resize complex array with interpolation?

username: Yolkandwhite
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/4/141ce87fb997edaea0a5ee784b60ee3fa6b0b2b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/141ce87fb997edaea0a5ee784b60ee3fa6b0b2b7" title="스크린샷 2021-05-31 오후 5.38.41"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/141ce87fb997edaea0a5ee784b60ee3fa6b0b2b7_2_517x113.png" alt="스크린샷 2021-05-31 오후 5.38.41" data-base62-sha1="2RVu46FBDwtPz3VEK7V5b8hpl7F" width="517" height="113" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/141ce87fb997edaea0a5ee784b60ee3fa6b0b2b7_2_517x113.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/141ce87fb997edaea0a5ee784b60ee3fa6b0b2b7_2_775x169.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/4/141ce87fb997edaea0a5ee784b60ee3fa6b0b2b7_2_1034x226.png 2x" data-dominant-color="1E1917"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2021-05-31 오후 5.38.41</span><span class="informations">1332×292 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any way to resize the array with complex value like above?<br>
but not with np.resize, because it doesn’t interpolate the values between.</p>
<p>I need to resize just like resizing the image (need interpolation)</p>
Debugging the new backward hook in v1.8.0

username: Xuechen_Li
cooked: <p>I’m getting the following error and would like to know how to debug this in general.</p>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.</p>
Split data folders into train and test for a GAN model in pytorch

username: Najeh_Nafti
cooked: <p>I have two folders one of normal patients and another for abnormal patients and I would like to train the model just on health patient and test it on health and unhealth. How can I split the data ?</p>

username: Henry_Chibueze
cooked: <p>If u train it on just healthy patients, do u think it’ll predict accurately for unhealthy patients?</p>

username: Najeh_Nafti
cooked: <p>I am looking to detect the anomalies zones. So I think yes the model will be able to detect anomalies in the new images of unhealthy patient!</p>
Reducing CNN model size

username: Samiur_Rahman
cooked: <p>Hello, i am trying to reduce the size(in MB) of my model either during training or post-training. I have tried post-training quantization for this purpose, which works. I am looking for any other methods that might be available on pytorch. I have also tried pruning, but that does not reduce the size of the model.<br>
Any advice would be appreciated.</p>
Custom loss gradient flow

username: cbd
cooked: <p>The output of Generator is given to custom loss function(no parameter learning). Custom loss function used the “exponential function” with some mathematical operation. So during backward, gradients pass through derivatives of exponential function and mathematical operation i used during forward. Finally when <em>gradient of custom loss</em> reaches to <em>output</em>, it is in range of e+04 while other <em>loss  gradients</em> (consider multiple losses) are in range of e-04 and e-05 <em>( I check individual loss gradients at output through hook).</em></p>
<p>Is it needed to convert the gradient of custom loss at output <em>(hook can be used</em>) in range of e-04 or e-05 so that it will not dominate the total loss?</p>
How to deal with mismatch of output image size and target image size

username: anargu
cooked: <p>Hi, I’m building a CNN model with mobilenet’s layers as the encoder. The decoder part is intended to output depth data from image but at a smaller size, let’s say w, h = 30, 30. However I have target image data of a much bigger size (640 x 480), so at the moment of comparing both of them in the loss function I’m not quite sure how to approach this mismatch or difference in the size.</p>
<p>One approach could be resizing the output data (from 30 x 30 to 640 x 480) and comparing both… but doesn’t it going to impact on the error/accuracy?</p>
<p>Another approach that I find could be to resize the target to the smaller output size (640 x 480 to 30 x 30) but I’m not quite sure on that.</p>
<p>What should be the approach to perform a correct computing of loss and/or which one (target our output) should be resized in this case?</p>

username: ptrblck
cooked: <p>I think it depends on your final use case besides the training impact.<br>
I.e. given that your targets are much larger than the model output: how would you want to deal with this shape mismatch when this model is deployed?<br>
Would you need to increase the spatial size of the model output before further utilizing these predictions or would the model output be fine and it could be used with its small size? Depending on this, I would try to adapt this into the training.</p>

username: anargu
cooked: <p>Thanks for the answer <a class="mention" href="/u/ptrblck">@ptrblck</a>. I opted for upsampling the output image as I looked in other examples they are using this approach. Actually i’m working on replicating a model from a paper for depth estimation.</p>
Gradient flow of different loss

username: cbd
cooked: <p>I have 4 losses which is used for backward out of which one is <em>custom loss</em>. These losses have different scaling. For example, Loss1 and Loss2 are in range of e-01, Loss3 is in range of e+01. Through hook i checked the gradients at output are in range of e-04 to e-05 for <em>Loss1 to Loss3</em>. Now custom Loss4 have high value and when it backpropagate to output it gives the gradients in range of e+03 to e+04.</p>
<p>First, i tried to scale Loss4 but there is no improvement in result. Second, I scale the gradient of Loss4 at output through hook but it doesn’t have any effect. I apply all possible scaling. I have used Adam optimizer.</p>
<p>Am i going in right direction or missing something? How to choose hyperparameter value for result improvement?</p>

username: eqy
cooked: <p>By 4 losses, do you mean something like<br>
<code>loss = lambda1*loss1 + lambda2*loss2 + lambda3*loss3 + lambda4*loss4</code> and scaling the lambda parameters? As a sanity check you can verify that the gradients should be different with lambda4 = 0.</p>

username: cbd
cooked: <aside class="quote no-group" data-username="eqy" data-post="2" data-topic="122713">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>By 4 losses, do you mean something like</p>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group" data-username="eqy" data-post="2" data-topic="122713">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>verify that the gradients should be different with lambda4 = 0.</p>
</blockquote>
</aside>
<p>Yes its different.</p>
<p>From your suggestion i start to observe the response of individual loss. Loss1 is GANLoss which is MSE loss and i check for only GANLoss with 150 training images as sample check. I observe that during start Generator Loss  and Discriminator Loss is in range of 1.6 and it quickly drop to range of 0.7 in second epoch and in range of 0.3 in third epoch. I run for 10 epochs but output image is not impressive. PSNR and SSIM during validation(just taken 4 images) is also fluctuating with wide range. Gradients at output image come in range of e-04 to e-05 from 1st epoch to 10th epoch.</p>
<p>Is the network is not converging or i need to check with more number of images?</p>