akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Get a cpu version model without changing the cuda model itself

username: zhiyuanpeng
cooked: <p>I use <code>torch.jit.save(model.cpu(), ...)</code> to save the cuda model to cpu torchscript. My problem is that <code>model.cpu()</code> changes the model from cuda to cpu, while, I need the cuda model un-changed. Is there a function like <code>convert_to_cpu(model)</code> that returns a cpu model without changing the model itself? Thanks.</p>

username: ptrblck
cooked: <p>I don’t think there is an out-of-place method to transfer the model to another device, but in case your GPU has enough memory left you could create a <code>copy.deepcopy</code> of the GPU model and transfer the copy to the CPU only.</p>

username: zhiyuanpeng
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your reply. I will try it.</p>
Vscode debug from python step into c++

username: lalalazy12
cooked: <p>I am using vscode to debug pytorch internals in linux, now I want to step in inner c function using debugger.</p>
<p>Here is an vscode <a href="https://github.com/benibenj/vscode-pythonCpp/issues/2" rel="noopener nofollow ugc">extension</a> to debug mix code(python and c++) , however it works fine with small CPython test, but not working with pytorch.</p>
<p>I also try to use this <a href="https://nadiah.org/2020/03/01/example-debug-mixed-python-c-in-visual-studio-code/" rel="noopener nofollow ugc">method</a> to debug, but not working.</p>
<p>Question: any solution to fix above question? or any other method to efficiently debug python C++ mix code in vscode?</p>
<p>Extra info: I install pytorch from source using:<code>USE_CUDA=0 REL_WITH_DEB_INFO=1 DEBUG=1 python setup.py  build_ext --debug install</code></p>
Pytorch giving blank outputs

username: Aalekh_Mudiraj
cooked: <p>I am relatively new to object detection and it’s my first time using pick pytorch</p>
<p>I trained the model, first with heterogeneous images (160)<br>
Later with only homogeneous (14)</p>
<p>And the text file output is always blank.</p>
<p>Can someone please help me debug where I am wrong?<br>
I’m not well versed with the code and no similar issue was found on Google.</p>
<p>I don’t know what part of code is necessary to post, if anyone can help me I’ll share the code with you. Thank you.</p>
Can pytorch 1.9 installed in cuda10.1?

username: YingdiZhang
cooked: <p>Can pytorch 1.9 installed in cuda10.1?</p>

username: zetyquickly
cooked: <p>Hi <a class="mention" href="/u/yingdizhang">@YingdiZhang</a></p>
<p>It is possible to build <code>pytorch</code> from source with any <code>cuda &gt;= 9.2</code> (as it’s said <a href="https://discuss.pytorch.org/t/is-there-a-table-which-shows-the-supported-cuda-version-for-every-pytorch-version/105846/5">here</a>)</p>
<p>But if you are looking for <code>conda</code> package it is either might be custom build (from someone who built it with <code>cuda==10.1</code> and <code>pytorch==1.9.0</code>) or a <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">previous version</a> with <code>cuda==10.1</code> (from pytorch channel)</p>
Are the parameters of my model (torch.nn.Module) specified correcltly?

username: Saguaro
cooked: <p>I would like to minimize the following cost function</p>
<pre><code class="lang-auto">class SelfSupervLoss(nn.Module):
    def __init__(self):
        super(SelfSupervLoss, self).__init__()
        
    def forward(self, x, edge_index):
        src, dest = edge_index
        return (x[src] * x[dest]).mean()

criterion = SelfSupervLoss()
</code></pre>
<p>and I want to use SGD for this purpose</p>
<pre><code class="lang-auto">from torch.optim import SGD

optimizer = SGD(model.parameters(), lr = 0.01)
</code></pre>
<p>The only parameters of my <code>model</code> are:</p>
<pre><code class="lang-auto">self.hv = torch.nn.Parameter(data.x[data.edge_index[0]]) # argument is a Tensor with the shape [472, 6]
self.hu = torch.nn.Parameter(data.x[data.edge_index[1]]) # argument is a Tensor with the shape [472, 6]
</code></pre>
<p>which I declare in the subclass of <code>torch.nn.Module</code> (If needed I can share the whole model)</p>
<p>Training loop is given as follows:</p>
<pre><code class="lang-auto">def train():
    optimizer.zero_grad()
    out = model(data.x, data.edge_index, deg)
    loss = criterion(out, data.edge_index)
    loss.backward(retain_graph = True)
    optimizer.step()
    return loss
</code></pre>
<p>But the loss remains on the same level over 200+ epochs</p>
<pre><code class="lang-auto"> Epoch: 0, Loss: 0.1043
 Epoch: 1, Loss: 0.1087
 Epoch: 2, Loss: 0.0914
 Epoch: 3, Loss: 0.1007
 Epoch: 4, Loss: 0.0994
...
</code></pre>
<p>I do not understand what I can be doing wrong. Maybe the declaration of the parameters? I am not sure… Does anybody know?<br>
Thank you!</p>

username: AlphaBetaGamma96
cooked: <p>Could you post what your model is?</p>
<p>Also, during the training loop print out what the gradients are! They might be really small (where you’d need to just increase your learning rate) or might be equal to near 0 which would indicate a problem within your model! You can print these out by adding this to your <code>train</code> function.</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
  print(name, param.requires_grad, param.grad)
</code></pre>

username: gphilip
cooked: <p>Try reducing your learning rate by different orders of magnitude. The loss may be “jumping around” a valley because the learning rate is too large. Try <code>lr=1e-3</code>, <code>lr=1e-4</code>, and so on; also try intermediate values such as <code>lr=3e-5</code>. Play around with the learning rate to see if you can hit a sweet spot.</p>
<p>If you see improvements in the loss when the learning rate is reduced, but you get tired of doing this by hand, you can also think of using learning rate schedulers of which PyTorch has many.</p>

username: Saguaro
cooked: <p>Thank you for the answer! I have had to completely rebuild my model and now it seems to perform better. But thanks again for your advice!</p>
TypeError: conv2d(): argument 'padding' (position 5) must be tuple of ints, not str

username: d4riush
cooked: <p>I created following conv layer:</p>
<pre><code class="lang-python">import torch
from torch import nn
in_channels = 1
out_channels = 3
kernel_size = 2
conv1 = nn.Conv2d(in_channels, out_channels, kernel_size, padding='same')
</code></pre>
<p>After calling this layer I get the following error:</p>
<pre><code class="lang-auto">TypeError: conv2d(): argument 'padding' (position 5) must be tuple of ints, not str
</code></pre>
<p>PyTorch Version: 1.7.0 (in kaggle notebook)</p>

username: KFrank
cooked: <p>Hi d4riush!</p>
<aside class="quote no-group" data-username="d4riush" data-post="1" data-topic="130513" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/d4riush/48/40205_2.png" class="avatar"> d4riush:</div>
<blockquote>
<pre><code class="lang-auto">TypeError: conv2d(): argument 'padding' (position 5) must be tuple of ints, not str
</code></pre>
</blockquote>
</aside>
<p>This works for me with pytorch version 1.9.0:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; from torch import nn
&gt;&gt;&gt; in_channels = 1
&gt;&gt;&gt; out_channels = 3
&gt;&gt;&gt; kernel_size = 2
&gt;&gt;&gt; conv1 = nn.Conv2d(in_channels, out_channels, kernel_size, padding='same')
&gt;&gt;&gt; conv1 (torch.randn (1, 1, 5, 5))
&lt;path_to_pytorch&gt;\torch\nn\modules\conv.py:440: UserWarning: Using padding='same' with even kernel lengths and odd dilation may require a zero-padded copy of the input be created (Triggered internally at  ..\aten\src\ATen\native\Convolution.cpp:660.)
  self.padding, self.dilation, self.groups)
tensor([[[[-0.4665,  0.1697, -0.5899, -0.2318,  0.4677],
          [-0.8381,  0.4828, -0.4036, -0.4699,  0.3754],
          [-0.6049, -0.3593, -0.4251,  0.2942, -0.3586],
          [-0.3345, -0.5207, -0.1593,  0.6332, -0.2843],
          [-0.2076, -0.2482, -0.5458, -0.2498,  0.3244]],

         [[-0.3107, -0.4693, -1.0476, -0.4348, -0.1129],
          [-1.0803, -0.3371, -0.1449, -0.2794, -0.3034],
          [-0.7755, -0.4563, -0.4941, -0.8762, -0.8973],
          [-0.5674, -0.5849, -0.1403,  0.5550, -0.0826],
          [-0.4254, -0.4933, -0.7118, -0.7279, -0.4403]],

         [[ 0.2665,  0.9369, -0.0974,  0.4658,  1.2056],
          [-0.4919,  0.7491,  0.1730,  0.5457,  0.9414],
          [-0.4147,  0.0814,  0.3814,  0.6475, -0.2871],
          [-0.0729,  0.0180,  0.5426,  1.3188,  0.5515],
          [ 0.2312,  0.2488,  0.1412,  0.4451,  0.7656]]]],
       grad_fn=&lt;ThnnConv2DBackward&gt;)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: d4riush
cooked: <p>Hi, yes I asked this on <a href="https://stackoverflow.com/questions/68964389/typeerror-conv2d-argument-padding-position-5-must-be-tuple-of-ints-not" rel="noopener nofollow ugc">Stackoverflow</a>, and based on comments turns out this is new feature which added in version 1.9.</p>
Torch.autograd.grad error. But this is my first time calling this fucntion

username: GoingMyWay
cooked: <p>I was trying to call the function <code>torch.autograd.grad</code> to calculate the gradients. However, it returned the following error</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, 
but the saved intermediate results have already been freed. 
Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>Actually, it was my fist time to call it. Any hint to solve this issue?</p>
<p>It was really strange. This error happens when debugging in VS Code at the line below</p>
<pre><code class="lang-auto">grad = torch.autograd.grad(loss, self.model.parameters(), allow_unused=True)
</code></pre>
<p>However, when running without debugging at that line, there is no error.</p>
Loss.backward() Error: RuntimeError: Index out of range

username: GoingMyWay
cooked: <p>I got the error</p>
<pre><code class="lang-auto">  File "/home/x/Projects/ProjectIM/network-opt-rl/src/agent/rl4im/agent_cc.py", line 72, in learn
    loss.backward()
  File "/home/x/anaconda3/envs/torch1.7/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/x/anaconda3/envs/torch1.7/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Index out of range
</code></pre>
<p>Does anyone know how to fix it?</p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet to reproduce the issue, please?<br>
Also, in case you are not using the latest PyTorch release, could you update and rerun the script?</p>
How to write jit model to a file in text format including backward?

username: amitgurintelcom
cooked: <p>Hi,<br>
I’m looking for a way to write the graph (preferably torch.jit.script module) forward and backward into a text format file.<br>
A simple way to write the forward:<br>
<em>module = torch.jit.script(model)    #  JIT for model</em><br>
<em>print(module.code)</em><br>
But, is there something similar to write the backward?<br>
The reason I’m looking for that is: I want to evaluate performance of the forward and backward for a theoretical HW.<br>
I tried to use pytorch profile, but it does not give all the information. Does not give the graph connectivity and tensor sizes<br>
ONNX export support only forward.</p>
<p>Any other idea?</p>

username: AlphaBetaGamma96
cooked: <p>One thing that you could try is using the <code>torchviz</code> library to visualize what the backward pass looks like!</p>
<p><code>torchviz</code> allows you to draw the computation graph of your network. This might be able to help you!</p>
Data augmentation for unsupervised segmentation task

username: Najeh_Nafti
cooked: <p>I have some <em>input images</em> and <em>targets</em> (masks and labels(0,1) : used in evaluation) I need to perform unsupervised segmentation on input images. I want to augment my training dataset before feeding it to GAN model<br>
What are the possible augmentation methods that can be used?<br>
And I am confused if should i define same transform for <em>input image</em> and * targets*<br>
Please help.<br>
Thanks</p>
Do I need to install `llibtorch` to load my triained model with `torchscript` in C++?

username: pytorching
cooked: <p>I have read the tutorials <a href="https://pytorch.org/docs/stable/jit.html#frequently-asked-questions" rel="noopener nofollow ugc">here</a> and <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="noopener nofollow ugc">here</a>.<br>
I know that I can save my <code>pytorch</code> model with <code>torchscript</code>, but If I want to embed my trained model in my C++ program, I need to have <code>libtorch</code> installed in my own program in order to use <code>torchscript api</code>, which sounds not reasonable, since my own program only has <code>100m</code> size big.<br>
Is there any solution for this?</p>
torchvision.datasets.CocoCaptions

username: RoitSky
cooked: <p>What does <code>torchvision.datasets.CocoCaptions</code> can do?<br>
<strong>Downloading coco datasets</strong> or just <strong>loading coco datasets in my local files</strong>.<br>
My code as follows:</p>
<pre><code>cap = datasets.CocoCaptions(root = './Images',
                            annFile = '.Annotations/anno.json',
                            transform=transforms.ToTensor())

print('Number of samples: ', len(cap))
img, target = cap[3] # load 4th sample

print("Image Size: ", img.size())
print(target)`
</code></pre>
<p>But it throw error : <strong>JSONDecodeError: Expecting value: line 1 column 1 (char 0)</strong></p>
Converting placeholder from tensorflow to pytorch

username: ZSheikhb
cooked: <p>I am a newbie programmer in pytorch. I am trying to convert <a href="https://github.com/enalisnick/mixture_density_VAEs/blob/master/models/gaussMMVAE_collapsed.py" rel="noopener nofollow ugc">this code</a> from <a href="http://bayesiandeeplearning.org/2016/papers/BDL_20.pdf" rel="noopener nofollow ugc"><em>Approximate Inference for Deep Latent Gaussian Mixtures</em></a> paper written in tensorflow to pytorch. I have a difficult time finding out how I can translate <code>tf.placeholder</code> syntax from tensorflow to pytorch explicitly for this part of the aforementioned code:<br>
line 84 of <code>gaussMMVAE_collapsed.py</code> script</p>
<pre><code class="lang-auto">class GaussMMVAE(object):
    def __init__(self, hyperParams):

        self.X = tf.placeholder("float", [None, hyperParams['input_d']])
        self.prior = hyperParams['prior']
        self.K = hyperParams['K']

        self.encoder_params = self.init_encoder(hyperParams)
        self.decoder_params = self.init_decoder(hyperParams)
</code></pre>
<p>I have read many posts that there isn’t any one to one function for placeholder but given the above example code, I reckon there should be a case by case solution. Thanks in advance for any instructive suggestion.</p>

username: ptrblck
cooked: <p>As you’ve already read, there are not placeholders in PyTorch and you can directly use tensors (similar to how you would use numpy). I don’t know why the <code>tf.placeholder</code> is assigned to <code>self.X</code>, but your translated code in PyTorch should just accept an input tensor in the <code>forward(self, x)</code> method and could use <code>x</code> directly in all operations.</p>
2nd channel input normalization

username: banikr
cooked: <p>I am using prediction from a neural net as an input to another network in the auto-context algorithm. I concatenate the prediction with images resulting in size: <code>[2, 2, 64, 96, 96]</code>. The second channel is the output of the previous network which has values for different tissue classes [0, 8].</p>
<p>My question is how do I normalize the prediction channel.<br>
N.B. I am normalizing the 3D image(channel 1) in min-max normalization that changes the value to [0, 1]. Do I do the same also for channel 2?</p>
<p>Thanks.</p>
Adding nn.Dropout2d to each conv layer of a pretrained VGG model

username: jarvico
cooked: <p>Hi all,</p>
<p>In my study, I am using pertained VGG models of Pytorch and use them for the classification of Imagenet dataset.</p>
<p>The thing is after each layer of conv layers of the VGG19bn model, I need to add a nn.Dropout2d layer. But I couldn’t figure out how to do it.</p>
<p>Can you please help me to enable dropout2d layer for pretrained vgg19bn model of Pytorch library?</p>
<p>I paste below code to express my need in a better way. After each conv2d, batchnorm2d, ReLU layers, I also want to add nn.Dropout2d() layer. I will be making some experiments about MC dropout sampling by enabling dropout during prediction time.</p>
<pre><code class="lang-auto">cfg = {
    'A': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'B': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'E': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 
          512, 512, 512, 512, 'M'],
}

def make_layers(cfg, batch_norm=False):
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True), nn.Dropout2d(p=0.1)]
                #layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True), nn.Dropout2d(p=0.1)]
                #layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = v
    return nn.Sequential(*layers)

class VGG(nn.Module):
    '''
    VGG model 
    '''
    def __init__(self, features):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512, 512),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(512, 512),
            nn.ReLU(True),
            nn.Linear(512, 10),
        )
         # Initialize weights
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                m.bias.data.zero_()


    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

def vgg19_bn():
    """VGG 19-layer model (configuration 'E') with batch normalization"""
    return VGG(make_layers(cfg['E'], batch_norm=True))
</code></pre>
<p>Using a pretrained model in Pytorch is as easy as below:</p>
<pre><code class="lang-auto">model_cnn = models.vgg19_bn(pretrained=True)
</code></pre>
<p>But after loading the pretrained model, how can I change it to enable dropout in conv. layers?</p>
<p>Any help is really appreciated as I need to solve this problem as soon as possible.</p>
<p>Regards…</p>
Sematic Segmentation metric

username: Sarmad_GTU
cooked: <p>Hi all<br>
I just want to calculate the semantic segmentation metric values like : pixel accuracy, mIoU<br>
and Kappa metric<br>
and I  found some code and then I adjust it as follows:<br>
my question is:<br>
<em>are these functions ok to calculate the above metric?</em>  if not can you suggest me any thing else</p>
<p><em>from sklearn.metrics import jaccard_score,cohen_kappa_score</em><br>
<em>def pixel_accuracy(Pred, ground_truth):</em></p>
<ul>
<li>with torch.no_grad():*</li>
<li>
<pre><code>   Pred = torch.argmax(Pred, dim=1) #dim =1 =&gt; over rows *
</code></pre>
</li>
<li>
<pre><code>   correct = torch.eq(Pred, ground_truth).int()  # the o/p here is 0 , 1 *
</code></pre>
</li>
<li>
<pre><code>   accuracy = float(correct.sum()) / float(correct.numel())*
</code></pre>
</li>
<li>return accuracy*</li>
</ul>
<p><em>def mIoU(pred, ground_truth, smooth=1e-10, n_classes=5):</em></p>
<ul>
<li>
<p>with torch.no_grad():*</p>
</li>
<li>
<pre><code>   # normalize the output becuse (we have identity activation functions) linear (same output)*
</code></pre>
</li>
<li>
<pre><code>   pred         = torch.argmax(Pred, dim=1)*
</code></pre>
</li>
<li>
<pre><code>   pred         = pred.cpu().contiguous().view(-1).numpy() # make it 1D *
</code></pre>
</li>
<li>
<pre><code>   ground_truth = ground_truth.cpu().contiguous().view(-1).numpy() # make it 1D *
</code></pre>
</li>
<li>
<pre><code>   MIoU=jaccard_score(ground_truth, pred,average='macro') #'none' per class *
</code></pre>
</li>
<li>
<pre><code>   return MIoU*
</code></pre>
</li>
</ul>
<p><em>def kapp_score_Value(pred, ground_truth):</em></p>
<ul>
<li>with torch.no_grad():*</li>
<li>
<pre><code>   pred              = torch.argmax(Pred, dim=1)*
</code></pre>
</li>
<li>
<pre><code>   pred              = pred.cpu().contiguous().view(-1).numpy() # make it 1D *
</code></pre>
</li>
<li>
<pre><code>   ground_truth      = ground_truth.cpu().contiguous().view(-1).numpy() # make it 1D *
</code></pre>
</li>
<li>
<pre><code>   *
</code></pre>
</li>
<li>
<pre><code>   kapp_score = cohen_kappa_score(ground_truth, pred_mask)       *
</code></pre>
</li>
<li>
<pre><code>   *
</code></pre>
</li>
<li>return kapp_score*</li>
</ul>
<p>*in mean code: *<br>
*I just call them like this : *<br>
<em>pred = model(image)</em></p>
<p><em>acc = pixel_accuracy(Pred, ground_truth)</em><br>
<em>MiouVal = mIoU(Pred, ground_truth)</em><br>
<em>kappavalue = kapp_score_Value(Pred, ground_truth)</em></p>

username: GeorgeBatch
cooked: <p>I think your functions will work fine for evaluation, but I do not think will not work if you want to use them for training (gradient optimization). I found <a href="https://www.kaggle.com/bigironsphere/loss-function-library-keras-pytorch" class="inline-onebox" rel="noopener nofollow ugc">Loss Function Library - Keras &amp; PyTorch | Kaggle</a> very helpful. You can also modify these functions, e.g. threshold for IoU and add a multiclass version for mIoU, and use them for faster evaluation on GPU.</p>

username: Sarmad_GTU
cooked: <p>Thank you so much<br>
i do not need the metric as loss,  i just want to calculate it</p>
Trainable mask threshold?

username: FruitVinegar
cooked: <p>Hi. I’d like to implement custom Conv2d layer that black-out image(2D tensor) in passed feature map(4D tensor) by preset threshold, which is trainable during the process.</p>
<p>In code…</p>
<pre><code class="lang-auto">class MyConv2d(nn.Module):
    def __init__(self, threshold: float = 0.1, **kwargs):
        self.threshold = nn.Parameter(torch.tensor(threshold, requires_grad=True))
        self.conv = nn.Conv2d(~) # initialized with given kwargs

    def forward(self, x: torch.Tensor):
        assert x.dim() == 4
        activation_of_each_image = torch.mean(x, dim=[2, 3])
        mask = torch.ge(activation_of_each_image, self.threshold)
        mask = mask.unsqueeze(2).unsequeeze(3) # convert dim to fit with original x.
        x = x.masked_fill(~mask, 0) # black-out images (filling 0) which have lower activation than threshold.
        y = self.conv(x)
        return y
</code></pre>
<p>I thought that <code>self.threshold</code> will be automatically changes by internal gradient calculation process, but I found It didn’t trained at all during the training.</p>
<p>How to make my threshold trainable? Is it impossible to calculate gradient for threshold with ge logic?</p>
Column-wise multiplication with matrix and vector

username: xdwang0726
cooked: <p>Is there any built-in function that multiply each column of a matrix by the corresponding element of a vector?</p>
<p>Example:<br>
a = torch.tensor([[[1,2,3],[5,6,7]], [[1,3,5],[5,8,7]], [[1,1,5],[5,8,3]]]), size(2, 2, 3)<br>
b = torch.tensor([[1,0,1], [1, 1, 0], [0, 1, 0]]), size(2, 3)<br>
(where the first dimension is batch size)<br>
expected output:<br>
torch.tensor([[[1, 0, 3], [5, 0, 7]], [[1, 3, 0], [5, 8, 0]], [[0, 1, 0], [0, 8, 0]]) , size(2, 2, 3)</p>
<p>Thanks!</p>

username: KFrank
cooked: <p>Hi xdwang!</p>
<aside class="quote no-group" data-username="xdwang0726" data-post="1" data-topic="130354" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xdwang0726/48/17441_2.png" class="avatar"> xdwang0726:</div>
<blockquote>
<p>Example:<br>
a = torch.tensor([[[1,2,3],[5,6,7]], [[1,3,5],[5,8,7]], [[1,1,5],[5,8,3]]]), size(2, 2, 3)<br>
b = torch.tensor([[1,0,1], [1, 1, 0], [0, 1, 0]]), size(2, 3)<br>
(where the first dimension is batch size)<br>
expected output:<br>
torch.tensor([[[1, 0, 3], [5, 0, 7]], [[1, 3, 0], [5, 8, 0]], [[0, 1, 0], [0, 8, 0]]) , size(2, 2, 3)</p>
</blockquote>
</aside>
<p>Yes.</p>
<p>You can use <a href="https://pytorch.org/docs/1.9.0/notes/broadcasting.html" rel="noopener nofollow ugc">broadcasting</a> together with element-wise multiplication, <code>*</code>,<br>
provided you use <code>unsqueeze()</code> so that the dimensions line up correctly:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; a = torch.tensor([[[1,2,3],[5,6,7]], [[1,3,5],[5,8,7]], [[1,1,5],[5,8,3]]])
&gt;&gt;&gt; b = torch.tensor([[1,0,1], [1, 1, 0], [0, 1, 0]])
&gt;&gt;&gt; a * b.unsqueeze (1)
tensor([[[1, 0, 3],
         [5, 0, 7]],

        [[1, 3, 0],
         [5, 8, 0]],

        [[0, 1, 0],
         [0, 8, 0]]])
</code></pre>
<p>(As an aside, the <code>size</code>s you give are wrong; your batch size is 3 (not 2).)</p>
<p>Best.</p>
<p>K. Frank</p>

username: xdwang0726
cooked: <p>Thanks! It really helps!</p>
How to Load a video dataset with varying video lengths and resolutions using PyTorch and use train test splits?

username: Nahian_Alindo
cooked: <p>Hi there. I am quite new to PyTorch. I just finished the <strong>getting started</strong> tutorial section. I am searching for some examples for loading video datasets. I looked at <a href="https://video-dataset-loading-pytorch.readthedocs.io/en/latest/" rel="noopener nofollow ugc">https://video-dataset-loading-pytorch.readthedocs.io/en/latest/</a> but what I understood is that as I am working for human recognition from the way a person walks, I don’t need segments of frames from the sequences rather I need the whole. So, I am kinda stuck because the videos have varying lengths and the videos have different resolutions. How can I efficiently load videos or frame sequences of varying lengths and then split it into train validation test sets?</p>
<p>My dataset looks like this:</p>
<pre><code class="lang-auto">dataset_folder
------person_1
        |----------- walking sequence 1
                       |-------------  img_0001.jpg
                             .
                             .
                             .
                       |-------------  img_0039.jpg
        |----------- walking sequence 2
                       |-------------  img_0001.jpg
                                            .
                                            .
                                            .
                       |-------------  img_0045.jpg
------person_2
        |----------- walking sequence 1
                       |-------------  img_0001.jpg
                                            .
                                            .
                                            .
                       |-------------  img_0045.jpg

------person_3
        |----------- walking sequence 2
                       |-------------  img_0001.jpg
                                            .
                                            .
                                            .
                       |-------------  img_0038.jpg
        |----------- walking sequence 2
                       |-------------  img_0001.jpg
                                            .
                                            .
                                            .
                       |-------------  img_0027.jpg
</code></pre>
A problem about Module.forward()

username: yuzi
cooked: <p>We try to use Mask R-CNN to make waterflow segmentations. Our code could be compiling successfully, but it occurs errors when running. We look up the source code of module.h and change codes many times.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d59e4b08a6392d3796df365faf8b6d11bcb7f635.png" data-download-href="https://discuss.pytorch.org/uploads/default/d59e4b08a6392d3796df365faf8b6d11bcb7f635" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d59e4b08a6392d3796df365faf8b6d11bcb7f635_2_690x268.png" alt="1" data-base62-sha1="utKQeZwknOJhxRytqTc9tUMUv77" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d59e4b08a6392d3796df365faf8b6d11bcb7f635_2_690x268.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d59e4b08a6392d3796df365faf8b6d11bcb7f635_2_1035x402.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/5/d59e4b08a6392d3796df365faf8b6d11bcb7f635.png 2x" data-dominant-color="2B2B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1365×531 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Custom DoubleBackward method returns invalid shape

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I just have a general question about writing custom autograd Functions that have in-built <code>DoubleBackward</code> methods. I’ve been writing a custom determinant function and I’m getting a mis-match in my <code>DoubleBackward</code> call.</p>
<p>From my understanding, the function takes a Tensor of shape A = <code>[B, N, N]</code> and outputs a scalar loss function so the output’s a Tensor of shape out = <code>[B,]</code>, I then calculate a 1st order gradient for <code>A</code> which is of shape <code>[B,N,N]</code> (which I get), then in my Double Backward I then need to calculate a 2nd order gradient for <code>A</code> and my <code>grad_output</code> of the 1st gradient. In short it looks something like this,</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Function

class Determinant(Function):
  
  @staticmethod
  def forward(ctx, A):
    ctx.save_for_backward(A)
    return torch.linalg.det(A)
    
  @staticmethod
  def backward(ctx, grad_output1):
    A, = ctx.saved_tensors
    return DeterminantBackward.apply(A, grad_output1)
    
class DeterminantBackward(Function):

  @staticmethod
  def forward(ctx, A, grad_output1):
    ctx.save_for_backward(A, grad_output1)
    """
    insert maths here
    ...
    calculate 1st order grad of A (A_grad) #shape [B,N,N]
    #shape of grad_output1 [B,]
    
    """
    return grad_output1*A_grad

  @staticmethod
  def backward(ctx, grad_output2):
    A, grad_output1 = ctx.saved_tensors
    """
    insert maths here
    ...
    calculate 2nd order grad w.r.t A (A_grad_grad) #shape [B,N,N]
    ...
    calculate 2nd order grad w.r.t grad_output1 (A_grad_grad_output1) #shape [B,]
    #shape of grad_output2 [B,N,N]
    """
    return grad_output2*A_grad_grad, grad_output2*A_grad_grad_output1
</code></pre>
<p>But a problem arises here. The first order gradient is of shape <code>[B,N,N]</code>  and is <code>[B,]</code> element-wise multiplied by <code>[B,N,N]</code> (i.e. multiplying a batch of scalars across a batch of matrices). However, when differentiating this expression yields a batch of matrices which by definition has a different shape than the original scalars. So, I’m kinda at a loss (pun not intended) here as to why PyTorch requires a different shaped Tensor than what I derive analytically!</p>
<p>Any help would be greatly appreicated!</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Difference between ":" and "..." in tensor slicing

username: Avishka_Perera
cooked: <p>I am new to PyTorch. Can someone explain to me the difference between “:” and “…” in tensor slicing.<br>
for instance</p>
<pre><code class="lang-auto">tensor = torch.rand(3,4)
print(f"Last column: {tensor[:,-1]}")
print(f"Last column: {tensor[...,-1]}")
</code></pre>
<p>outputs the same results<br>
Thankyou in advance</p>

username: AlphaBetaGamma96
cooked: <p>It seems that when using <code>:</code> and <code>...</code> in this context they yield the same result. However, it seems when using <code>:</code>, and <code>...</code> in terms of adding a new dim to a Tensor they give different results.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; M=torch.randn(3,4)
&gt;&gt;&gt; M
tensor([[-1.1915,  1.3102,  2.4116, -0.6022],
        [-1.4025, -0.0796, -0.6342, -1.5389],
        [-0.3399, -0.1049,  0.0192,  0.0186]])

#grab last column
&gt;&gt;&gt; M[..., -1]
tensor([-0.6022, -1.5389,  0.0186])
&gt;&gt;&gt; M[:, -1]
tensor([-0.6022, -1.5389,  0.0186])

#adding new dim
#shape torch.Size([3, 1, 4])
&gt;&gt;&gt; M[:, None]
tensor([[[-1.1915,  1.3102,  2.4116, -0.6022]],

        [[-1.4025, -0.0796, -0.6342, -1.5389]],

        [[-0.3399, -0.1049,  0.0192,  0.0186]]])
#shape torch.Size([3, 4, 1])
&gt;&gt;&gt; M[..., None]
tensor([[[-1.1915],
         [ 1.3102],
         [ 2.4116],
         [-0.6022]],

        [[-1.4025],
         [-0.0796],
         [-0.6342],
         [-1.5389]],

        [[-0.3399],
         [-0.1049],
         [ 0.0192],
         [ 0.0186]]])
&gt;&gt;&gt; 
</code></pre>
<p>So, in your cases they are the same? but in other cases they’ll give different behaviour!</p>

username: victorc25
cooked: <p>It’s the same as numpy slicing, use colons ( : ) when you have multiple dimensions you need to slice differently, ie: tensor[:-1. 2:-1, :] and semicolons (…) when all following (or previous) dimensions should be kept the same. For example, a tensor of shape: (8, 3, 256) can be sliced as:</p>
<ul>
<li>tensor[0:1, …] or equivalently tensor[0:1, :, :]</li>
<li>tensor[…, -1] or equivalently tensor [:, :, -1]</li>
<li>tensor[0:1, 0, …] or tensor[0:1, 0, :]<br>
and so on and so on</li>
</ul>
Check if n-th tensor row contains n-th vector element

username: Asa-Nisi-Masa
cooked: <p>Hi. I have the following problem. Suppose I have a tensor M of shape (N, 5) and a vector v of size (N).<br>
I am looking for a vectorized solution that would tell me if the nth row of M contains the nth element of v. Example:</p>
<pre><code class="lang-auto">M = torch.tensor([
    [0, 1, 2, 3, 4],
    [7, 5, 3, 4, 8],
    [0, 1, 9, 7, 5],
    [1, 4, 7, 2, 5],
])

v = torch.tensor([0, 3, 2, 6])
</code></pre>
<p>then the function I am looking for should output something like the following</p>
<pre><code class="lang-auto">does_intersect(M, v) == torch.tensor([1, 1, 0, 0])  # or boolean indicators
</code></pre>
<p>I know that I can just iterate over the rows and check if <code>v[n] in M[n]</code> but I want a vectorized solution.<br>
One additional note, each row of M contains unique elements</p>

username: ptrblck
cooked: <p>You could <code>unsqueeze</code> <code>v</code> compare the tensors directly (<code>v</code> will be broadcasted) and then check the columns for any match:</p>
<pre><code class="lang-python">print((M == v.unsqueeze(1)).any(1))
&gt; tensor([ True,  True, False, False])
</code></pre>

username: Asa-Nisi-Masa
cooked: <p>Thanks, couldn’t figure it out</p>
From libtorch to ONNX

username: Ziemin
cooked: <p>There is a way to load an ONNX or traced model in libtorch, but is there an option to go in the opposite direction? I would like to export a model defined and trained using C++ interface to ONNX format. If there is no such a feature, what is the exact reason for that. How difficult would it be to implement?</p>

username: beichen2012
cooked: <p>Hi, Did you have any idea?</p>

username: tom
cooked: <p>As far as I know, quite a bit of the ONNX export is implemented in Python.</p>
<p>So the two main options likely are:</p>
<ul>
<li>Save the weights in C++, rebuild the module in Python, load the weights and export. Totally straightforward but a bit of effort.</li>
<li>You can export a TorchScript module to ONNX, too. So if the module is traceable, you should be able to do that to get a TorchScript module in C++. Then you can load that into Python and export. Given that ONNX export mostly does tracing internally, too, that might be a viable alternative that skips the “implement the model in Python” step.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

username: beichen2012
cooked: <p>Thank you.<br>
" * You can export a TorchScript module to ONNX, too. So if the module is traceable, you should be able to do that to get a TorchScript module in C++. Then you can load that into Python and export. Given that ONNX export mostly does tracing internally, too, that might be a viable alternative that skips the “implement the model in Python” step."</p>
<p>This line: “you should be able to do that to get a TorchScript module in C++”, But now, My net is torch::nn::Moudle, how to convert it to torch::jit::Moudule(which is ScriptModule in python) just like the torch.jit.script does?</p>

username: tom
cooked: <p>So in principle, tracing (as in torch.jit.trace) should work in C++ as well as in Python, but I must admit I didn’t try. (And when I said, anomaly detection should work in C++, I felt the need to implement it just to not be handing out bad advice too often.)</p>
<p>Maybe the easiest is to create that Python model after all.</p>
<p>Best regards</p>
<p>Thomas</p>
Error when enumerating dataloader

username: Cedric_Baron
cooked: <p>When enumerating over my dataset I get this error:</p>
<p>mask_path = self.mask_files[idx]<br>
IndexError: list index out of range</p>
<p>from this line:</p>
<p>for index, (tiles, labels) in enumerate(loader):</p>
<p>I tried with other datasets it is working well. The problem seem to come from a subsetting I am doing when loading the dataset to gain time. When I use this dataset I get this issue. Maybe it comes from that but I do not know in what way.</p>
<p>class GreenhouseDataset(Dataset):<br>
def <strong>init</strong>(self, folder_data):</p>
<pre><code>    # lists to store the data
    self.mask_files = []
    self.img_files = []
    
    # getitng the list of files
    list_files = fun.get_files(folder_data)
    list_files = random.sample(list_files, len(list_files)//10) 
    
    # extracting the data
    for filename in list_files:
           self.mask_files.append(filename)
            self.img_files.append(filename)
</code></pre>
<p>PS: its my first post on this forum</p>

username: tom
cooked: <p>One way to get this error could be to have the dataset claim a length larger than the list you are indexing.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>PS Enclosing your code in triple backticks ``` will give it code formatting.</p>
<p>PPS Welcome here!</p>

username: Cedric_Baron
cooked: <p>thanks for your comment.<br>
I am not sure I completely follow you on your advice, the error comes at the first element from the for loop, as if there was no first element.<br>
Strangely, when I tried to run it outside of my train function it was working.<br>
I checked everything again but the dataset is not altered in the process so I dont understand why I get this error inside the function</p>

username: tom
cooked: <p>You could just print the idx in your getitem and also compare to len(ds) and len(ds.mask_files).</p>

username: Cedric_Baron
cooked: <p>I found the issue.<br>
When I was subsetting the data I was doing it on the whole folder so there was an unbalance between masks and images</p>
Summarize the reasons for the common error: "Illegal Memory Access"

username: fanl
cooked: <p>Hi everyone,<br>
I come across the following error occasionally:</p>
<blockquote>
<p>File “/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/mmcv/runner/hooks/optimizer.py”, line 259, in after_train_iter<br>
scaled_loss.backward()<br>
File “/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/tensor.py”, line 195, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/root/.pyenv/versions/3.6.8/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  CUDA error: an illegal memory access was encountered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:764)</p>
</blockquote>
<p>The occasion of this error is random. It might be raised after several epochs of training, it’s time-consuming to reproduce the error. I write the code with strict assertions to keep the variables reasonable, so there should not be some trivial errors like <code>out of range</code>.</p>
<p>I ask for help for the following things:</p>
<ol>
<li>I hope someone can help me to summarize the possible reasons (no need to be very specific) for this error.</li>
<li>Is it possible that running out of memory may cause this error?</li>
</ol>
<p>I will appreciate it if you guys give me help.</p>

username: ptrblck
cooked: <ol>
<li>
<p>Any invalid memory read or write can cause this issue. The mentioned out of range indexing is a common issue, which could also be caused by e.g. race conditions etc.</p>
</li>
<li>
<p>I don’t think it can (at least this would be the first time an OOM is causing the memory violation).</p>
</li>
</ol>
Load saved TorchScript model as regular nn.Module

username: calvinmccarter
cooked: <p>I would like to load a TorchScript model file (ie saved using torch.jit.save) into a regular torch.nn.Module. IE, rather than have</p>
<pre><code class="lang-auto">my_recursive_script_module = torch.jit.load(model_path)
</code></pre>
<p>I’d have:</p>
<pre><code class="lang-auto">my_nn_module = some_function(model_path)
</code></pre>
<p>Is this possible? I would think it would be straightforward, since TorchScript models are essentially a restricted subset of torch models, but it’s not working.</p>

username: ptrblck
cooked: <p>I’m not aware of this “inverse” operation to revert the scripting and create a Python model from the scripted one, but you might be able to manually restore the model based on the graph.</p>
RuntimeError trying to run PyTorch over GeForce RTX 3080Ti

username: Carlos
cooked: <p>When I run pytorch on the newly installed RTX 3080ti machine, “segmentation fault” or “CUDA error unspecified launch failure” always appears.</p>
<p>My hardware configuration is i5-11600kf + 32GB + rtx3080ti.</p>
<p>The software configuration is anaconda3 + Python 3.8.11 + Pytorch 1.8.1 + cuda11.1 + cudnn8.0.5 + NVIDIA driver 460.</p>
<p>I’ve also tried pytorch 1.9.0 and CUDA 11.2, but they all have the same problem. I’ve tried many versions, but when they all start running, the loss drops normally, but there will be an error in less than 10 minutes. In addition, the torch model is mainly based on the LSTM layer, and the code is normal on tasla-v100-12g.</p>
<p>I have no way. Who can help me???</p>

username: ptrblck
cooked: <blockquote>
<p>I’ve also tried pytorch 1.9.0 and CUDA 11.2</p>
</blockquote>
<p>I guess you’ve build PyTorch from source using this CUDA version or did you use a pip wheel from another source (as the official binaries would either ship with CUDA10.2 (won’t work for your 3080) or CUDA11.1)?</p>
<p>In any case, could you post a minimal, executable code snippet which we could use to reproduce this issue, please?</p>

username: Carlos
cooked: <p>Thank you for your reply. I use conda for installation.<br>
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia</p>

username: Carlos
cooked: <p>I may find some problems.<br>
After many times of verification, when the GPU temperature exceeds 65 ℃ for a long time, there will be errors in training, resulting in training interruption. If the temperature is reduced to below 50 ℃, it can be trained normally.</p>
What is CUDA GUARD?

username: 111480
cooked: <p>I found ‘CUDA GUARD’</p>
<p>but I don’t know exactly what is cuda guard and when it’s necessary.</p>

username: ptrblck
cooked: <p>Assuming you are referring to the <code>CUDAGuard guard(device_index_);</code> used in the backend: it’s used to set the specified device in the scope and reset it afterwards.</p>
Neural Networks on CIFAR-10

username: Nil_MSh
cooked: <p>Hello!<br>
i have a ResNet trained on CIFAR-10. after giving the model images of cats and dogs and horses, how can I know the prediction of my model for each image? I want to know that for a pic x, did my model predicted correctly and was the pic a dog?</p>

username: RaLo4
cooked: <p>Hey again.<br>
I am guessing this is the same question as <a href="https://discuss.pytorch.org/t/label-checking-in-cifar-10/130294">here</a>.<br>
But there you said you only want to check for the dog labels.<br>
If you want to check for all cifar10 labels you could try something like this:</p>
<pre><code class="lang-auto">classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

num_label_lst = [0]*10
correct_lst = [0]*10

for images, labels in test_loader:
    images = images.to(device)
    labels = labels.to(device)
    outputs = model(images)
    _, predicted = torch.max(outputs.data, 1)

    #go through all labels
    for i, label in enumerate(labels):
        for j in range(10):
            #check with which cifar10 label it matches
            if label == j:
                #count up the counter for the number of occurrence for that label 
                num_label_lst[j] += 1
                #check if the prediction is the same
                if predicted[i] == j:
                    #count up the counter for the number of correct occurrences for that label
                    correct_lst[j] += 1
                #no need to check the other cifar10 labels so break the inner loop
                break


for label_index, correct in enumerate(correct_lst):
    print(f"number of correct {classes[label_index]} labels: {correct}")
    print(f"{classes[label_index]} classification accuracy: {100*correct/num_label_lst[label_index]:.2f}%")
</code></pre>
Second derivative - passing from linear to non-linear formula

username: jamesboom
cooked: <p>Hi guys!</p>
<p>I am interested about passing from this linear function created below, to a non-linear one, in order to estimate the second derivative. Is there exist some trick that I can use to have the second derivative different from zero? I am trying to estimate a greek called “Gamma” for a Call Option, that is priced using Montecarlo method</p>
<pre><code class="lang-auto">import torch as np
values = torch.tensor([1., 1.1, 1.2], requires_grad=True)

def delta_gamma(xi):

    k,T,j,sigma = 1.5,1.,10000,0.5  

    S = np.broadcast_to(xi,(j,) + xi.shape).T    

    mean = -.5 * sigma * sigma * T

    volatility = sigma

    BM = torch.randn(1, j)*volatility+mean   

    product = S*np.exp(BM)   

    p = torch.maximum(product-k,torch.zeros_like(product))   

    result = torch.mean(p, 1)

    return result

for i in values:

    print(torch.autograd.functional.hessian(delta_gamma,i))
</code></pre>