akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Output problem with torch.nn.CrossEntropyLoss()

username: meshghi
cooked: <p>Hello. I’m doing semantic segmentation with Unet. My target class of 0 (background) and 1 (class-of-interest) is in one channel. So, my target is [batch_size, 256, 256]. And, my model output has two channels (one for each class) [batch_size, 2, 256, 256]. I want to use torch.nn.CrossEntropyLoss() for my loss function but I don’t know how to properly reduce the output of my model to one channel after forward pass so that I could insert it in the loss function. Any help is appreciated.</p>

username: thecho7
cooked: <p>I think <code>[B, C, 256, 256]</code> is correct to apply <em>nn.CrossEntropyLoss</em>.<br>
see how it works <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.10 documentation</a></p>

username: meshghi
cooked: <p>Thank you. You’re right. I had a problem in my code and I thought the channel difference was the issue. The issue was something else. thanks again.</p>
Reduce Tensor Dimension by Random Sampling/Indexing

username: osama-usuf
cooked: <p>I have a 3-D tensor of shape <code>(10, 15, 20)</code>. I want to reduce it to be a 2-D tensor with shape <code>(10, 15)</code>, making sure that I select only a random element from the original 3-D tensor’s third dimension (of length 20).</p>
<p>I have tried index notation by generating an index tensor that matches the third dimension, but am running into dimensional broadcasting issues.</p>
<p>Any help would be appreciated!</p>

username: thecho7
cooked: <p>How do you use index tensor?</p>
<pre><code class="lang-auto">a = torch.rand(10, 15, 20)
result = a[:, :, idx]
</code></pre>
<p>for generated random integer <code>idx</code></p>

username: osama-usuf
cooked: <p>That’s exactly how I’m using my index tensor, but I am ending up with the same dimensional tensor.</p>
<p><code>result = a[:, :, 1]</code> gets me the correct dimensional result, but <code>result = a[:, :, idx]</code> returns the same dimension as <code>a</code>, the original tensor.</p>
<p>What dimensions should the <code>idx</code> tensor have for correct slicing?</p>

username: thecho7
cooked: <p>I understood what bothers you</p>
<pre><code class="lang-auto">a = torch.rand(10, 15, 20)
idx = torch.LongTensor([idx])

# torch.Size([10, 15, 1])
result1 = a[:, :, idx] 
# torch.Size([10, 15])
result2 = a[:, :, idx].squeeze()
</code></pre>
<p><code>tensor.squeeze()</code> is what you are looking for.<br>
Indexing with a list (or tensor) keeps the dimension of the original.</p>
<p>You can compare the different outputs by<br>
<code>a[:, :, idx]</code> and <code>a[:, :, idx.item()]</code>.<br>
First one is indexing with a tensor and second one is with an integer.</p>

username: osama-usuf
cooked: <p>Thank you for your input <a class="mention" href="/u/thecho7">@thecho7</a> - much appreciated.</p>
<p>The problem still persists for me.</p>
<p><code>tensor.squeeze()</code> only works for removing dimensions of size 1. I am ending with a dimension <code>(10, 15, 20)</code>, which remains unaltered by the squeeze operation.</p>

username: thecho7
cooked: <p>Please let me show your code snippet.<br>
Isn’t your question about making <code>(10, 15, 20)</code> to <code>(10, 15)</code> by sampling one element?</p>
Get torch::jit::Graph from torch::jit::script::Module

username: Andre_Chang
cooked: <p>Is there a way to get <code>std::shared_ptr&lt;torch::jit::Graph&gt;&amp; graph</code><br>
from <code>torch::jit::script::Module</code></p>
<p>I exported a module in python using:</p>
<pre><code class="lang-auto">torchscript_trace = torch.jit.trace(my_module, example_inputs)
torchscript_trace = torch.jit.freeze(torchscript_trace.eval())
torchscript_trace.save('model.pt')
</code></pre>
<p>and wanted to get the graph object in C++ using:</p>
<pre><code class="lang-auto">torch::jit::script::Module module;
module = torch::jit::load('./model.pt');
</code></pre>

username: Andre_Chang
cooked: <p>found solution <a href="https://github.com/pytorch/extension-cpp/blob/master/cpp/lltm.cpp" rel="noopener nofollow ugc">here</a></p>
CosineAnnealingLR does not reset

username: eisbaer
cooked: <p>Hey friendly people,</p>
<p>I am currently testing a bunch of learning rate schedulers and I was wondering if it is intentional behavior, that the CosineAnnealingLR does not reset (at least if coupled with an Adam optimizer) and the learning rate does follow a full cosine.</p>
<p>In contrast to that the CosineAnnealingWarRestarts in default configuration does have resets.</p>
<p>From my understanding of those methods their behavior in default configuration should be identical, or am I missing something? Maybe CosineAnnealingLR is waiting for an reset signal from the optimizer because it is intended to be coupled with momentum SGD but it does not come because I used Adam?</p>
<p>Would be great if someone could help me with this.</p>
<p>EDIT: I revisited the documentation of the CosineAnnealingLR and as I understand it, it is not intended to reset at all. So the parameter T_max does only alter the period of the cosine right?</p>
<p>I did a little mokup to test this, if it is of any help:</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt
params = torch.tensor((10,10))
opt = torch.optim.Adam([params])
cos = torch.optim.lr_scheduler.CosineAnnealingLR(opt, 10)
warm = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(opt, 10)
warm_log = []
cos_log = []
for i in range(100):
    warm_log.append(*warm.get_last_lr())
    cos_log.append(*cos.get_last_lr())
    warm.step()
    cos.step()
plt.figure()
plt.plot(warm_log)
plt.plot(cos_log)
plt.show()
</code></pre>
Module 'torch' has no attribute 'searchsorted'

username: 1206089395k
cooked: <p>The version of my torch is 1.10.1, why did this happen</p>

username: ptrblck
cooked: <p>I cannot reproduce the issue and the method is available in <code>1.10.0</code> as well as <code>1.10.1</code>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.0+cu113'
&gt;&gt;&gt; torch.searchsorted
&lt;built-in method searchsorted of type object at 0x7f141a8a61e0&gt;

&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.1+cu113'
&gt;&gt;&gt; torch.searchsorted
&lt;built-in method searchsorted of type object at 0x7f5e7b2371e0&gt;
</code></pre>
Pytorch layer with no trainable parameters

username: Luca_Pamparana
cooked: <p>I am just playing around a bit with pytorch and have a model which has the following structure:</p>
<pre><code class="lang-auto">Layer A - 100 trainable parameters
Layer B - 0 trainable parameters
Layer 3 - 5 trainable parameters
</code></pre>
<p>In my forward function, I have something like:</p>
<pre><code class="lang-auto">def forward(x, y):
    a = layer_a(x)
    b = layer_b(a)
    loss = layer_c(b, y)
    return {"loss": loss}
</code></pre>
<p>The <code>layer_b</code> is simply defined as:</p>
<pre><code class="lang-auto">class LayerB(nn.Module):
     def __init__(self, params):
         super().__init__()
         self.params = params

    def forward(self, x):
        return x.clone()
</code></pre>
<p>Now when I run this model, it makes one step through the training process and then crashes with:</p>
<pre><code class="lang-auto">one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [32, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead
</code></pre>
<p>I am trying to understand why this is happening. Do you think it is necessary to call <code>x.clone().detach()</code> since no trainable parameters are in that layer? I ask because at least the model training does not crash when I do that.</p>

username: pascal_notsawo
cooked: <p>If you do <code>x.clone().detach()</code> it is possible that <code>layer_a</code> is never being updated by backpropagation, because this completely breaks the computation graph.<br>
Using a <code>torch.nn.Identity</code> in <code>layer_b</code> might be more convenient, but is that where the error comes from?</p>
<ul>
<li>If so, try this:</li>
</ul>
<pre><code class="lang-auto">class LayerB(nn.Module):
		def __init__(self, params):
				...
				self.net = nn.Identity()
   	
   	    def forward(self, x):
             return self.net(x)
</code></pre>
<ul>
<li>Otherwise, it is possible that the error comes from the wrong assignment  somewhere, as for example : <code>x += ...</code> with <code>x</code> requiring the gradient…</li>
</ul>
<p>…</p>

username: Luca_Pamparana
cooked: <p>Thanks for the reply:</p>
<p>So this layer is basically just:</p>
<pre><code class="lang-auto">def forward(self, x):
        return x.clone()
</code></pre>
<p>So, this is why I am puzzled as there is no such update but I will keep digging.</p>
Random transformation are different across calls?

username: samin_hamidi
cooked: <p>I want to rotate all the images in my data_set with a random degree between [0,180]. If I compose a transformation function and pass my images to this function in <strong>getitem</strong> function of my data_set class, does this mean, 1) every single image is randomly rotated?, 2) images in each batch get rotated with an identical degree but this degree randomly changes across batches (calls)?</p>
<p>I would appreciate if you could clarify this for me.</p>

username: eqy
cooked: <p>Generally if you are using the <code>torchvision</code> transforms inferface, <em>each</em> image is randomly transformed, meaning within a batch the rotations will be different. If you want control over this, you can craft your own data preprocessing function using the functional transfom: <a href="https://pytorch.org/vision/0.8/transforms.html#torchvision.transforms.functional.rotate" rel="noopener nofollow ugc">torchvision.transforms — Torchvision 0.8.1 documentation (pytorch.org)</a></p>
How can I calculate F1 score in object detection?

username: SU801T
cooked: <p>Hi,</p>
<p>I’ve followed the object detection tutorial (<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.10.1+cu102 documentation</a>) and adapted the code for my problem.</p>
<p>I understand that the average precision and recalls are calculated when put through evaluation:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
  # train for one epoch, printing every 10 iterations
  train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
  # update the learning rate
  lr_scheduler.step()
  # evaluate on the test dataset
  evaluate(model, valid_data_loader, device=device)
</code></pre>
<p>How can I calculate the f1 score from these metrics? Would I need to extract additional information? I’m guessing I would need to also calculate the number of false negatives, false positives, true positives and true negatives based on a specific IoU threshold…</p>
<p>Is there an example that already exists?</p>

username: my3bikaht
cooked: <p>You can use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html#sklearn.metrics.f1_score" rel="noopener nofollow ugc">sklearn.metrics.f1_score</a> if you don’t want to calculate f1 manually</p>

username: my3bikaht
cooked: <p>right, was thinking about something else <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
if you already have precision and recall, why don’t just directly calculate<code> F1 = (2 * precision * recall)/(precision+recall) ?</code></p>
How can I determine whether predicted bboxes are true or false positives from faster rcnn in a binary problem

username: SU801T
cooked: <p>Hi,</p>
<p>As the title suggests, how can I determine whether a bbox is considered to be “correct”? I have followed the object detection tutorial where I evaluate my model via:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
  # train for one epoch, printing every 10 iterations
  train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
  # update the learning rate
  lr_scheduler.step()
  # evaluate on the test dataset
  evaluate(model, valid_data_loader, device=device)
</code></pre>
<p>I’m wondering if there is a simple way of retrieving this information without the need to amend the engine script…</p>
Calculating index of nearest value to a nested tensor

username: osama-usuf
cooked: <p>Greetings,</p>
<p>I have a problem where I want to find the nearest indices of a nested tensor based on a flat tensor having arbitrary dimensions.</p>
<p>For example:</p>
<pre><code class="lang-auto">G = tensor([1, 4, 9])
X = tensor([[2, 3, 7, 9], [8, 6, 7, 5]])
</code></pre>
<p>For every element in <code>X</code>, I want to find the index of the nearest element in G (minimum distance) as follows:</p>
<pre><code class="lang-auto">result = nearest_idx(X, G)
print(result) # should return: [[0, 1, 2, 2], [2, 1, 2, 1]] 
</code></pre>
<p>I am able to solve this problem using list comprehensions, but I want to be able to tensorize this operation (i.e. do it in terms of PyTorch operations for better performance).</p>
<p>Any input is greatly appreciated!</p>

username: pascal_notsawo
cooked: <p>What you can do is resize G by giving it the dimension <code>(n, m, p)</code>, where <code>(n, m)</code> is the dimension of <code>X</code> (<code>(2, 4)</code> in your example) and <code>p</code> the dimension of <code>G</code> (<code>3</code> in your example).</p>
<p>Once this is done, <code>argmin_along_the_p_dimension |X - G|</code> will give you the result.</p>
<pre><code class="lang-auto">G = torch.tensor([1, 4, 9]) # (p,)
X = torch.tensor([[2, 3, 7, 9], [8, 6, 7, 5]]) # (n, m)

#n, m, p = 2, 4, 3
#(G.repeat(n, m, 1) - X.unsqueeze(-1)).abs().argmin(dim=-1)
(G.repeat(*X.shape, 1) - X.unsqueeze(-1)).abs().argmin(dim=-1)

"""
tensor([[0, 1, 2, 2],
        [2, 1, 2, 1]])
"""
</code></pre>
The loss is not decreasing

username: OBouldjedri
cooked: <p>I am training a neural nework the architecture is pretty simple ? conv layers with bach norm and 3 linear layers with relu , the loss is never decreasing but the accuracy is getting good (train loss and train accuracy are fine but test loss is not decreasing and test accuracy is not in the propoer shape .<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68d2113a90737601fd7393d341057e2f92efaae8.png" alt="test_accuracy" data-base62-sha1="eXhITdeSaO3av1zTGLQVs5ZGgWY" width="432" height="288"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3817f456131bd1a9b3e17ba8b8dc418fe07f63e.png" alt="test_loss" data-base62-sha1="ub4bWu7CVC9q1tzCMGfGVIrupT8" width="432" height="288"></p>
<p>the training loss and accuracy are :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/facba16cceb3ee2cc8a890767e3d2e1d8be1c6f2.png" alt="train_accuracy" data-base62-sha1="zMDDVSqGeRWkzBZa0lmbe1jc2WK" width="432" height="288"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc8c3cca57c88ae084d8685f1f77d2c96f500723.png" alt="train_loss" data-base62-sha1="qTYdKGqCzyKnthUR0R4nIfQLWo3" width="432" height="288"></p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/obouldjedri">@OBouldjedri</a> without looking at further details, it would be difficult to ascertain the exact reason. Some pointers</p>
<ul>
<li>
<p>Ensure that the distribution of labels in train and test data do not vary a lot. This shows up as train accuracy high but test accuracy low</p>
</li>
<li>
<p>Overfitting. Can you overlay the test as well as the train accuracy in the chart</p>
</li>
</ul>

username: OBouldjedri
cooked: <p>I uploaded what the train accuracy and loss looks like<br>
.</p>
Can't run training on PyTorch 1.10.2 with NVidia RTX 3050

username: Kola73
cooked: <p>I have installed new NVidia RTX 3050, run the training code, and got following message on resnet50 model creation:</p>
<pre><code class="lang-auto">/home/tester/anaconda3/lib/python3.8/site-packages/torch/cuda/__init__.py:143: UserWarning: 
NVIDIA GeForce RTX 3050 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
If you want to use the NVIDIA GeForce RTX 3050 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
</code></pre>
<p>PyTorch version 1.10.2, OS: Ubuntu 18.04, Python 3.8.8</p>
<p>Code runs ok on RTX 2060</p>

username: eqy
cooked: <p>How was PyTorch installed on this system? It looks like a CUDA 10.2 version was installed; see e.g., <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">Start Locally | PyTorch</a> where CUDA 11.3 should be selected for <code>sm_86</code>.</p>

username: Kola73
cooked: <p>Yes, CUDA version is 10.2.<br>
I will update and check<br>
Thanks a lot</p>
How do I tell my model that a new sequence starts when batch training timeseries?

username: Patrick1
cooked: <p>I am training an LSTM model on timeseries data to build a classifier. So when I do batch training with for example batch size = 8. How does my model know when one sequence ends and another one starts? I have read a couple times that I would need to reset hidden and cell state inside my model, but I am not sure if this is correct or not, as I have seen some people not doing it. Either way I have tried it and the Problem I have is that when initializing h0 und c0 in my model it expects the size of (Num_layers, batch_size, hidden size) but as I get to the end of my traindataset not enough data is available for a whole batch so i get the error  Expected hidden[0] size (1, 14, 64), got [1, 16, 64].</p>
<p>So basically my questions just are:</p>
<ol>
<li>How does my model know when one sequence ends and another one starts?</li>
<li>How do I implement this?</li>
</ol>
<p>My model without resetting the states looks like this:</p>
<pre><code class="lang-auto">class Model_LSTM(nn.Module):

    def __init__(self, n_features, n_classes, n_hidden, n_layers):            # bidirectional möglich
        super().__init__()
        self.lstm = nn.LSTM(
            input_size=n_features,
            hidden_size=n_hidden,
            num_layers=n_layers,
            batch_first=True,
            dropout=0.75
        )

        self.classifier = nn.Linear(n_hidden, n_classes)

    def forward(self, x):
        _, (hn, _) = self.lstm(x)                
        out=hn[-1]                                 
        return self.classifier(out)

</code></pre>
<p>With resetting the States (Which does not work) it looks like this:</p>
<pre><code class="lang-auto">class Model_LSTM(nn.Module):

    def __init__(self, n_features, n_classes, n_hidden, n_layers):            # bidirectional möglich
        super().__init__()
        self.lstm = nn.LSTM(
            input_size=n_features,
            hidden_size=n_hidden,
            num_layers=n_layers,
            batch_first=True,
            dropout=0.75
        )

        self.classifier = nn.Linear(n_hidden, n_classes)

    def forward(self, x):
        h0 = torch.zeros(self.lstm.num_layers,batch_size,self.lstm.hidden_size)
        c0 = torch.zeros(self.lstm.num_layers,batch_size,self.lstm.hidden_size)

        _, (hn, cn) = self.lstm(x, (h0,c0))                  
        out=hn[-1]                                 
        return self.classifier(out)</code></pre>
2D vector to 1D vector by over convolution

username: lanka
cooked: <p>hi,<br>
I have a 1024<em>1024 dimension feature vector to reduce dimension to 1,1024, so how can I get the final dimension using conv2d operation over 1024</em>1024. please explain the forward function for this or any other way</p>

username: ptrblck
cooked: <p>I don’t know which dimensions are referenced by the shape <code>[1024, 1024]</code>, but assume the spatial ones.<br>
If so, you could use a conv layer with <code>kernel_size=(1024, 1)</code> to create an output of <code>[batch_size, out_channels, 1, 1024]</code>:</p>
<pre><code class="lang-python">conv = nn.Conv2d(3, 4, (1024, 1))
x = torch.randn(2, 3, 1024, 1024)
out = conv(x)
print(out.shape)
# &gt; torch.Size([2, 4, 1, 1024])
</code></pre>

username: lanka
cooked: <p>can you please explain why we use<code> x = torch.randn(2, 3, 1024, 1024)</code> instead of <code>x = torch.randn(1024,1024)</code></p>

username: ptrblck
cooked: <p><code>nn.Conv2d</code> expects an input in the shape <code>[batch_size, channels, height, width]</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html?highlight=conv2d#torch.nn.Conv2d">docs</a>, so I’ve used random values for the missing dimensions.</p>

username: lanka
cooked: <p>ok. in my case its like  <code>x = torch.randn(1, 1, 1024, 1024)</code></p>
Does build_vocab_from_iterator "flatten" the Iterable input?

username: hogru
cooked: <p>Hi,</p>
<p>basic question/issue about building a vocabulary. The toy code:</p>
<pre><code class="lang-auto">from torchtext.vocab import build_vocab_from_iterator
tokens = ['a', 'bc', 'def']
voc = build_vocab_from_iterator(tokens)
voc.get_stoi()  # {'f': 5, 'c': 2, 'e': 4, 'd': 3, 'b': 1, 'a': 0}
</code></pre>
<p>I would have expected 3 entries in the vocabulary (instead of 6).</p>
<p>Is there another way to achieve this or is this not intended/supported?</p>
<p>Best,<br>
Stephan</p>
<p>torchtext version 0.11.2 on macOS</p>
Feature weighting

username: sparshgarg23
cooked: <p>I am currently training an architecture for regression purposes.<br>
The training data consists of images and their ground truth labels (which is a 15x1) vector representing information relevant to my purpose.<br>
Out of these 15 features,some are important while others can be ignored.</p>
<p>As such is there a way to assign weight to these features and then train the network so that the final prediction is consistent with the weights assigned during training.<br>
I read somewhere that feature weighting shouldn’t be done</p>
<pre><code class="lang-auto">https://stackoverflow.com/questions/38034702/how-to-put-more-weight-on-certain-features-in-machine-learning/38042546
</code></pre>
<p>My approach:<br>
let’s say I have about 2000 samples ,so my training data will be of size 2000x199<br>
I can compute the covariance matrix of this and use that to determine the weights for the features.<br>
Would that be correct?</p>
CUDA Graphs with the C++ API

username: Hamster
cooked: <p>To my knowledge there isn’t an official way from libtorch to use CUDA Graphs like on the latest PyTorch release. In this case, is it still possible (&amp; safe?) to use “vanilla” CUDA Graphs from the CUDA API as described here: <a href="https://developer.nvidia.com/blog/cuda-graphs/" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with CUDA Graphs | NVIDIA Developer Blog</a> ? What would be safe to record using such an approach? Are <code>.backward()</code> and stepping through an optimizer safe to record? What about <code>NoGradGuard</code> or things like <code>model-&gt;train()</code> and <code>model-&gt;eval()</code>? Thanks !</p>

username: ptrblck
cooked: <p>I don’t know how exactly you want to use CUDA graphs directly, but would claim it’s prone to error due to the strict memory requirements. The better way might be to try out to use the backend API directly as defined <a href="https://github.com/pytorch/pytorch/blob/5cae40c169b3da48d2782947d072d20b7acb15d1/aten/src/ATen/cuda/CUDAGraph.cpp">here</a>.</p>

username: Hamster
cooked: <p>Wow thank you, I have missed that there is a CUDA Graphs backend in the C++ api! (no entry in the C++ documentation though)</p>
Script, RuntimeError: Module X has no attribute Y:

username: Developers_Fox
cooked: <p>Code is not needed to explain it, but basically, X class is (nn.Module) with other custom (nn.Module)s as its attributes.<br>
The attribute Y is also defined &amp; running outside of script normally.<br>
However in the code I’m getting this exact error…</p>
<p>The problematic scenario is shortly as below;</p>
<p>Module’s init()<br>
self.attn_cond_layer = AttentionConditioningLayer(<br>
input_dim=2, attention_n_filters=32,<br>
attention_kernel_sizes=[5, 3],</p>
<p>Module’s forward()<br>
attn_cond_vector = self.attn_cond_layer(attn_cat).permute(2, 0, 1)</p>
<p>—&gt; error occurs right here… (Module ‘AR_Step’ has no attribute ‘attn_cond_layer’) What could be the problem?</p>
<p>for extra information, the definition of Y attribute is below;</p>
<p>class AttentionConditioningLayer(nn.Module):<br>
“”“Adapted from the LocationLayer in<br>
<a href="https://github.com/NVIDIA/tacotron2/blob/master/model.py" rel="noopener nofollow ugc">https://github.com/NVIDIA/tacotron2/blob/master/model.py</a><br>
1D Conv model over a concatenation of the previous attention and the<br>
accumulated attention values “””<br>
def <strong>init</strong>(self, input_dim=2, attention_n_filters=32,<br>
attention_kernel_sizes=[5, 3], attention_dim=640):</p>

username: ptrblck
cooked: <p>Could you post a minimal, executable code snippet to reproduce the issue, please?</p>
Are there conditions where slicing changes gradients/backprop?

username: jkr
cooked: <p>We are training a network and changing<br>
<code>x = [cls_score for cls_score in cls_scores]</code><br>
to<br>
<code>x = [cls_score[:] for cls_score in cls_scores]</code><br>
in the loss calculation of the network yields different losses but only after a few iterations. This is the only change we made. The differences in the loss start very small and then become bigger. (Our actual use case is different, but this is the smallest change in code that elicited the difference in the loss.)<br>
Our guess is, that gradients are different during backpropagation in the two cases. This would explain why the losses became different.<br>
We have seen different examples in the forum where slicing did not have a negative effect on backpropagation, but we can’t think about another explanation for this effect.<br>
We have also read that there were problems with backprop with torch versions before 1.0.9, but we are using 1.10.2.</p>
<p>Does anyone have an idea/guess why the losses differ? Is is because of different gradients or is there another explanation for our observation? We are happy about any suggestions and hints!</p>
How to have a custom test train split in MNIST (torchvision dataset)?

username: divya
cooked: <p>(newbie in pytorch here)<br>
In the torchvision dataset - MNIST, dataloader has 60000 images as training set and 10000 images as the test set. My requirement is to have around 20000 images as training set and the rest as test set. Is this possible to do?</p>

username: mMagmer
cooked: <p>if you want (not recomanded, leave testset alone)  you can first merge them to one set with:</p>
<pre><code class="lang-auto">mergeDataSet = torch.utils.data.ConcatDataset([trainSet,testSet])
</code></pre>
<p>and then you can split them to val and train set again with:</p>
<pre><code class="lang-auto">newTrainSet , valSet = torch.utils.data.random_split(mergeDataSet, [20000, 50000], generator=torch.Generator().manual_seed(42))
</code></pre>
<p>I have not tested the code. but it should work.</p>

username: divya
cooked: <aside class="quote no-group" data-username="mMagmer" data-post="2" data-topic="143553">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p><code>newTrainSet , valSet = torch.utils.data.random_split(mergeDataSet, [20000, 50000], generator=torch.Generator().manual_seed(42))</code></p>
</blockquote>
</aside>
<p>This works perfectly!! Thank you so much!!</p>
Custom dataset/dictionary of labels and images

username: avalon1511
cooked: <p>Hi, I have a tricky problem (at least to me) and am not sure how to proceed.</p>
<p>Let’s say I have a dataset of images and I have generated some labels for every batch. These are stored in batches of size <code>b_size</code></p>
<p>How this goes for <code>b_size = 32</code>:</p>
<ol>
<li>Traverse dataset and generate batches of size 32 so something like <code>(32, 1, 64, 64)</code>.</li>
<li>For every batch I have a set of labels of size <code>(32, 100, 1, 1)</code>
</li>
</ol>
<p>I want to create a dataset/dictionary that can store these image batches and corresponding labels so that when I later iterate over this dataset/dictionary it returns the <code>(x,y)</code> pair where <code>x is the batch of images</code> and <code>y is the corresponding batch of labels</code></p>
<p>I would prefer a dataset since I also want to subset it in the future. But I honestly have no idea how to start this.</p>
<p>Any ideas would be appreciated thanks!</p>

username: ptrblck
cooked: <p>You could check <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a> which explains how to write a custom <code>Dataset</code>. In the simplest approach you would need to implement the logic to load and process a single sample in the <code>__getitem__</code> and the <code>DataLoader</code> would shuffle, create the batches, etc.</p>
Convert gpumat to pytorch tensor

username: treasureqqm
cooked: <p>How can i convert a gpumat to pytorch tensor? My gpumat comes from cv::cudacodec as the following code:</p>
<pre><code class="lang-auto">cv::cuda::GpuMat d_frame;
cv::Ptr&lt;cv::cudacodec::VideoReader&gt; d_reader = cv::cudacodec::createVideoReader(video_path);
d_reader-&gt;nextFrame(d_frame);
</code></pre>
<p>Now I want to convert the d_frame to pytorch gpu tensor, and I have searched for a while but still don’t know how to do it properly.</p>

username: ptrblck
cooked: <p>I guess you might be able to use <a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1ac009244049812a3efdf4605d19c5e79b.html?highlight=from_blob"><code>torch::from_blob</code></a> to create a tensor from the raw data.</p>
How to Properly Use Data Sets in PyTorch

username: johnny_ginger
cooked: <p>Hi,<br>
What is the clearest way to go about using datasets in PyTorch if the dataset is in csv format and appears in the following form:</p>
<p>cell_id,2020-01-07,2020-01-14,2020-01-21,2020-01-28,2020-02-04,2020-02-11,2020-02-18,2020-02-25,2020-03-03,2020-03-10,2020-03-17,2020-03-24,2020-03-31,2020-04-07,2020-04-14,2020-04-21,2020-04-28,2020-05-05,2020-05-12,2020-05-19,2020-05-26,2020-06-02,2020-06-09,2020-06-16,2020-06-23,2020-06-30,2020-12-01,2020-12-08,2020-12-15,2020-12-22,2020-12-29,2021-01-05,2021-01-12,2021-01-19,2021-01-26,2021-02-02,2021-02-09,2021-02-16,2021-02-23,2021-03-02,2021-03-09,2021-03-16,2021-03-23,2021-03-30,2021-04-06,2021-04-13,2021-04-20,2021-04-27,2021-05-04,2021-05-11,2021-05-18,2021-05-25,2021-06-01,2021-06-08,2021-06-15,2021-06-22,2021-06-29<br>
000863e7-21e6-477d-b799-f5675c348627,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0</p>
<p>“region”: “other” }, “geometry”: { “type”: “Polygon”, “coordinates”: [ [ [ -119.401672788750602, 37.024004907174238 ], [ -119.401672788750602, 37.031176567834088 ], [ -119.410655941591799, 37.031176567834088 ], [ -119.410655941591799, 37.024004907174238 ], [ -119.401672788750602, 37.024004907174238 ] ] ] } },<br>
{ “type”: “Feature”, “properties”: { “cell_id”: “000ba8d9-d6d5-48da-84a2-1fa54951fae1”,</p>

username: ptrblck
cooked: <p>The cleanest way would be to write a custom <code>Dataset</code> to load and process the data in the desired way as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.</p>
While running load traced_torch_model.ot exit with code -1

username: HuiLin_Chang
cooked: <p>Hello all, I followed the tutorial and would like to run traced_torch_model.pt in C++. However, the app existed code -1 automatically. Any advice is appreciated.</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;memory&gt;

using namespace std;
#include &lt;torch/script.h&gt;
#include &lt;iostream&gt;
#include &lt;memory&gt;

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr &lt;&lt; "usage: example-app C:\\Users\\hchang\\Documents\\Gladies\\Python at work\\0813_2021\\traced_torch_model.pt\\";
            return -1;
    }

    torch::jit::script::Module module;
    try {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load(argv[1]);
    }
    catch (const c10::Error&amp; e) {
        std::cerr &lt;&lt; "error loading the model\n";
        return -1;
    }
    std::cout &lt;&lt; "Model " &lt;&lt; argv[1] &lt;&lt; " loaded fine\n";

    // Create a vector of inputs.
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(torch::randn({ 1, 1, 64, 101 }));

    // Execute the model and turn its output into a tensor.
    at::Tensor output = module.forward(inputs).toTensor();
    std::cout &lt;&lt; output &lt;&lt; "\n";
    int y_hat = output.argmax(1).item().toInt();
    std::cout &lt;&lt; "Predicted class: " &lt;&lt; y_hat &lt;&lt; "\n";
}
</code></pre>

username: ptrblck
cooked: <p>I assume you are not seeing any error messages?<br>
If so, then try to remove the model loading part of your code and check if a simple example of creating a dummy model and random inputs would work. If that’s the case, check if your paths etc. are valid to load the scripted model.</p>
Pytorch update hyper-parameter using current loss, RuntimeError: Trying to backward through the graph a second time

username: MichaelH
cooked: <p>I am defining my own loss function, and my own loss function has a hyper-parameter <strong>Lambda</strong>. For example, if the prediction is <strong>y</strong>, then I define the loss function as <strong>Loss = Lambda * y</strong>. I want to update my <strong>Lambda</strong> at some iteration using the current round’s <strong>Loss</strong>. For example, at some specific iteration, I want my <strong>Lambda</strong> to be updated as <strong>Lambda = Lambda + Loss</strong>, then it returns the error of</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
</code></pre>
<p>Specifically, my naive code is as follows:</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

batch_size, input, output = 10, 3, 3
model = nn.Linear(input, output)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
lam = torch.from_numpy(np.array([0.1, 0.1, 0.1]))
lam.requires_grad = False

for i in range(10):
    x = torch.rand(batch_size, input)
    output = model(x)
    loss = torch.sum(lam*output)
    if i == 5:
        lam = lam + torch.clone(loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    print(loss)
</code></pre>
<p>I had the feeling that the error was caused by using <strong>Loss</strong> to update my <strong>Lambda</strong>. So I used the code <code>torch.clone(loss)</code>, hoping not to influence <code>loss</code>, it didn’t help. Does anyone know how to fix the problem? Some explanation about why this error occurs would be great!</p>

username: ptrblck
cooked: <p>Since you are manually updating <code>lam</code>, try to wrap the update into <code>with torch.no_grad()</code> to avoid tracking this operation inside a computation graph.</p>
Overflow on CPU, but not GPU

username: OverLordGoldDragon
cooked: <pre><code class="lang-auto">import torch
torch.manual_seed(0)
x = torch.ones(1000000).half()

print(x.mean())
print(x.cuda().mean())
</code></pre>
<pre><code class="lang-auto">tensor(nan, dtype=torch.float16)
tensor(1., device='cuda:0', dtype=torch.float16)
</code></pre>
<p>Why?</p>

username: ptrblck
cooked: <p>I guess the accumulation kernel on the CPU might not be using <code>float32</code> to represent the intermediate values as mixed-precision training with <code>float16</code> is usually used on the GPU (if I’m not mistaken, <code>bfloat16</code> is the preferred numerical format on the CPU, but let’s wait for others to chime in and correct me).</p>
Calculate differential of a tensor before loss

username: Y_Y
cooked: <p>Hi all,</p>
<p>If I have the following piece of code</p>
<pre><code class="lang-auto">  similarity_matrix = torch.matmul(features, features.T)
  mask = torch.eye(labels.shape[0], dtype=torch.bool).cuda() 
  labels = labels[~mask].view(labels.shape[0], -1)      
  similarity_matrix = similarity_matrix[~mask].view(similarity_matrix.shape[0], -1)
  positives = similarity_matrix[labels.bool()].view(labels.shape[0], -1)
  negatives = similarity_matrix[~labels.bool()].view(similarity_matrix.shape[0], -1)
  logits = torch.cat([positives, negatives], dim=1)
  P = logits[0, 0]
</code></pre>
<p>Is it possible to calculate the differential of P using something like P.backward()?<br>
I have register hook of middle layer. The final goal is to calculate the differential of \partial P/\partial middle layer.<br>
I tried but got all 0s.</p>
<p>Thank you for any help.</p>

username: osama-usuf
cooked: <p>You can take use of PyTorch’s <code>autograd</code> to compute the differential for P - here’s a link to the documentation <code>https://pytorch.org/docs/stable/generated/torch.autograd.grad.html</code></p>
<p>If this is a repeated operation, I would recommend you register a hook (as you say you’ve done) on the tensor which would be called every time a gradient with respect to that tensor is calculated. Then, when <code>loss.backward()</code> is called, it would trigger your hook and you would have your required gradient (assuming that’s what you required at the first place).</p>
<p>I would have to see your code for the hook and maybe the training loop to be able to pinpoint why your gradient is all 0s even after the hook. One possible explanation of that could be that your hook is never called, because again, tensor hooks are called ONLY when a gradient with respect to that tensor is computed somewhere in your computational graph. Please try logging the hook to verify if it’s actually called as a starting step.</p>
<p>Here’s a link that might be helpful: <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.register_hook — PyTorch 1.10 documentation</a></p>
How to only sample data from a dataloader with target == class

username: Dylan_Yung
cooked: <p>Is there a way to take a dataloader and only sample a batch of the predefined batch size, but only when the target is class N?</p>

username: osama-usuf
cooked: <p>Based on the nature of what you’re trying to achieve - you might be able to apply a mask on the loaded train/test sets and create custom train/test sets that contain only required classes.</p>
<p>If this is a viable option for you, share some working code and I could assist as I’ve dealt with something similar to this previously.</p>
Register_hook with GradScalar

username: Qwwq
cooked: <p>I am using a register_hook on a tensor to print the gradients on stdout while using GradScaler in mixed precision mode during training. Do the prints on stdout reflect the scaled gradients or the unscaled ones?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>If you are using a <code>torch.cuda.amp.GradScaler</code> and print the gradients during the <code>backward</code> pass, the scaled gradients are shown.</p>
Debug conv transpose kernel

username: bhavana_p
cooked: <p>I’m trying to debug conv transpose kernel implementation in pytorch. I have built pytorch from source and in develop mode. With a sample application I tried to put breakpoints using gdb but am not able to locate the kernel and the file in which it exists, any help?</p>

username: ptrblck
cooked: <p>The native implementations can be found <a href="https://github.com/pytorch/pytorch/blob/a18cfb790d0998aa7939c01174fb2f40237897eb/aten/src/ATen/native/NaiveConvolutionTranspose2d.cpp">here</a> for the CPU and <a href="https://github.com/pytorch/pytorch/blob/a18cfb790d0998aa7939c01174fb2f40237897eb/aten/src/ATen/native/cuda/NaiveConvolutionTranspose2d.cu">here</a> for the GPU. You won’t be able to debug lib implementations, if they are closed source (e.g. such as cuDNN).</p>
Indexing.cu error when use_deterministic_algorithms is True

username: Olafyii
cooked: <p>To reproduce the bug:</p>
<pre><code class="lang-auto">import torch
torch.use_deterministic_algorithms(True)
a = torch.full(torch.zeros(2,10).shape, 0.1, device='cuda:0')
b = (torch.tensor([0, 1, 1]), torch.tensor([1,2,3]))
a[b] = 1
print(a)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a14e36dc98d9328aacbc7981bca4f48fabf2918.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1a14e36dc98d9328aacbc7981bca4f48fabf2918" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a14e36dc98d9328aacbc7981bca4f48fabf2918_2_690x47.jpeg" alt="image" data-base62-sha1="3IJanwkWYguWvqPmFWgI5BncN6M" width="690" height="47" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a14e36dc98d9328aacbc7981bca4f48fabf2918_2_690x47.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a14e36dc98d9328aacbc7981bca4f48fabf2918_2_1035x70.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a14e36dc98d9328aacbc7981bca4f48fabf2918_2_1380x94.jpeg 2x" data-dominant-color="2B2B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2160×148 67.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Without the second line the code would exit normally.</p>

username: ptrblck
cooked: <p>Could you update to the latest nightly release, as this issue should be solved already?</p>
What does `optimizer.state_dict()` store?

username: Yash_Rathi
cooked: <p>Hello<br>
I found about <code>optimizer.state_dict</code>, but I am not able to understand what this data is? Because while using Adam we don’t need to store anything else in memory. It computes rolling average.<br>
If I want to continue training later, should I also save optimizer state dict.</p>

username: osama-usuf
cooked: <p>An optimizer’s state dictionary contains two types of information - parameters that are being optimized and any hyperparameters in use.</p>
<p>For correct model loading in order to resume your training, you should indeed be saving (and re-loading) the <code>state_dict</code> of your model AND the Adam optimizer.</p>
<p>Hope this helps - here’s a few links that might assist.</p>
<ol>
<li><a href="https://pytorch.org/docs/stable/generated/torch.optim.Optimizer.state_dict.html" class="inline-onebox" rel="noopener nofollow ugc">torch.optim.Optimizer.state_dict — PyTorch 1.10 documentation</a></li>
<li><a href="https://stackoverflow.com/questions/62260985/what-are-saved-in-optimizers-state-dict-what-state-param-groups-stands-for" class="inline-onebox" rel="noopener nofollow ugc">deep learning - What are saved in optimizer's state_dict? what "state","param_groups" stands for? - Stack Overflow</a></li>
</ol>
Parallel Training on multiple GPUs without first GPU saturation

username: ajboloor
cooked: <p>tl;dr:<br>
I am trying to increase the batch size for training ResNet50 with the pytorch imagenet training example, but since the first GPU holds both the model and data, I can’t increase the batch size without saturating the first GPU. How can put data just on the other GPUs?</p>
<p>Details:<br>
I’m doing this over 4 GPUs, currently with a batch size of 64.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abae0c547e742e2cde823341285be6e8c3f6522f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/abae0c547e742e2cde823341285be6e8c3f6522f" title="Screenshot of"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abae0c547e742e2cde823341285be6e8c3f6522f_2_690x374.jpeg" alt="Screenshot of" data-base62-sha1="ouKxIP7qLNMg0bF278Ivm6aFDB5" width="690" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abae0c547e742e2cde823341285be6e8c3f6522f_2_690x374.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abae0c547e742e2cde823341285be6e8c3f6522f_2_1035x561.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abae0c547e742e2cde823341285be6e8c3f6522f_2_1380x748.jpeg 2x" data-dominant-color="111111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot of</span><span class="informations">1580×858 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>You can see in the screenshot above about how the ResNet model and Imagenet data are distributed across the four GPUs. You can also see that GPU 0 has more memory used than 1,2,3 because it holds the resnet model in addition to the data.<br>
How do I go about putting the data only on GPUs 1,2,3 while keeping the model on GPU 0? So that I can increase the batch size to fully use all the GPUs.</p>

username: mrshenli
cooked: <p>I assume you are using DataParallel, and that’s why the input have to stay on <code>cuda:0</code>? Will DistributedDataParallel (DDP) (<a href="https://pytorch.org/docs/stable/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch 1.10 documentation</a>) help? With DDP, each process can have its own dataloader loading data into its own device.</p>