akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Accuracy decrease after load the saved resnet50 model

username: saluei
cooked: <p>Hi<br>
This Problem reported in</p><aside class="quote" data-post="2" data-topic="81628">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmandivarapu1/48/3544_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/accuracy-decrease-after-load-the-saved-model/81628/2">Accuracy decrease after load the saved model</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    My guess is here model accuracy was depend on the batches that go through the testing phase so instead of performing once test for 5 time and average it then save the model and then redo it
  </blockquote>
</aside>
<p>
However, does not solve.<br>
I have the same problem,</p>
<p>validation and train accuracy of ResNet50 decreased 70% when I loaded the model with load_state_dict() function !!!<br>
However, when I save the training results everything was OK<br>
I use torchvision.models.resnet50 train it with CFAR100 and get 96%, 76% train and test accuracy, save the model satte and then load it with the same hyperparameter.<br>
However I get 40%, 3% train, and test accuracy,<br>
any help is appreciated.</p>

username: ptrblck
cooked: <p>Could you compare the outputs of the model for a static input (e.g <code>torch.ones</code>) and after calling <code>model.eva()</code> before saving the <code>state_dict</code> and after loading it?<br>
If the result is equal (up to floating point precision) then I would recommend to check the data loading and make sure both scripts use the same data processing steps.</p>

username: saluei
cooked: <p>Thank,s ptrblck<br>
I do what you suggest and find my bug, in fact, it is my fault in the forward function,<br>
in the first training I use torch.vision forward function,<br>
but the second time when I load the saved model I implemented a custom forward function,<br>
It is my error in custom forward function implementation  ,</p>
Batch data selection at intermediate layer

username: Josh_Steve
cooked: <p>Dear community,</p>
<p>I want to implement the model where a certain batch selection occurs at the intermediate layer of the model. The selection outputs the subset of the current batch data and feed it to the classifier. Below is the model code</p>
<pre><code class="lang-auto">class ConvNet(Module):
    def __init__(self, input_shape = (28,28), num_classes=62):
        self.featdim = 160
        self.num_classes = num_classes
        self.input_shape = input_shape
        
        super(ConvNet, self).__init__()
        self.conv1 = Conv2d(1, 5, 5)
        self.pool = MaxPool2d(2, 2)
        self.conv2 = Conv2d(5, 10, 5)
        self.fc1 = Linear(self.featdim, 129)
        self.fc2 = Linear(129, self.num_classes)

    def get_feature(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(x.shape[0],-1)
        return x

    def classifier(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
        
    def forward(self, x, repr=True):
        if repr:
            x = self.get_feature(x)
        x = self.classifier(x)
        return x

</code></pre>
<p>and the main code</p>
<pre><code class="lang-auto">m = ConvNet()
d = datasets.MNIST(root='dataset', train=True, transform=transforms.ToTensor())
l = DataLoader(d, 8)
for X, y in l:
    h = m.get_feature(X)
    idx_sel = torch.randperm(8)[:4]
    y_s = y[idx_sel]
    h_s = h[idx_sel]
    c = m.classifier(h_s.view(4, -1))
    loss = torch.nn.CrossEntropyLoss()(c, y_s)
    loss.backward()
</code></pre>
<p>I am unsure whether the unselected data will also affect the feature layer during the backpropagation. Is this essentially the same as training data directly on X[idx_sel]? Thanks.</p>
Difference between square bracket [] and curly () bracket in torch.randn()

username: sandeep1
cooked: <p>There are two ways of creating random tensor :-<br>
<code>torch.randn([1,2], dtype=torch.float32, requires_grad=True)</code></p>
<p>and</p>
<p><code>torch.randn((1,2), dtype=torch.float32, requires_grad=True)</code></p>
<p>The random tensor’s dimension is 1x2 written in two ways [1,2] and (1,2).</p>
<p>Both of them returns a random tensor of 1x2 dimension. So, what is the difference between them?</p>

username: ptrblck
cooked: <p>There is no difference as you can specify the shape using a <code>tuple</code>, <code>list</code>, or direct arguments:</p>
<pre><code class="lang-python">torch.randn(1, 2, dtype=torch.float32, requires_grad=True)
</code></pre>
<p>The <code>list/tuple</code> arguments are also supported in other classes e.g. in the <code>kernel_size</code> argument of conv layers:</p>
<pre><code class="lang-python">nn.Conv2d(1, 1, kernel_size=(3, 3))
nn.Conv2d(1, 1, kernel_size=[3, 3])
</code></pre>
Debug backward deadlock without gdb?

username: vadimkantorov
cooked: <p><code>loss.backward()</code> deadlocks for me quite frequently (using CPU, no distributed mode). Unfortunately, I don’t have root / access to gdb. Is there any way to still debug / trace this?</p>
<p>Is it possible to enable any tracing / logging of autograd backward?</p>

username: ptrblck
cooked: <p>You could try to set <code>export TORCH_SHOW_CPP_STACKTRACES=1</code>, run your script until it hangs, and kill it e.g. via <code>SIGHUP</code>. You might be able to see the stacktraces in the terminal which could point to the hanging line of code. I haven’t tried this approach as <code>gdb</code> is available in my setup, but it might be worth a try.</p>
Can PyTorch1.10 cuda113 profiling show op's name in aten levels

username: zejun-chen
cooked: <p>Hello</p>
<p>Can PyTorch 1.10 profiling show aten level ops’ name ?</p>
<p>When we previously used PyTorch 1.7, the profiling was very useful and we can see aten level ops’ performance according to its name.</p>
<p>While i use PyTorch 1.10, the profiling info is shown as below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/947ee75f10e6896b5b33afaa0b67e9bc37e611b9.png" data-download-href="https://discuss.pytorch.org/uploads/default/947ee75f10e6896b5b33afaa0b67e9bc37e611b9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/947ee75f10e6896b5b33afaa0b67e9bc37e611b9_2_690x104.png" alt="image" data-base62-sha1="lbEysjeIVwsC5c02IWBr4lDB0eJ" width="690" height="104" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/947ee75f10e6896b5b33afaa0b67e9bc37e611b9_2_690x104.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/947ee75f10e6896b5b33afaa0b67e9bc37e611b9_2_1035x156.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/947ee75f10e6896b5b33afaa0b67e9bc37e611b9_2_1380x208.png 2x" data-dominant-color="181717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1733×263 282 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With this name, i can not know which is conv and which is batchnorm. Could we profile like PyTorch1.7 ?</p>
<p>Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Saving the best model on epoch validation loss or epoch validation accuracy

username: Oussama_Bouldjedri
cooked: <p>I am using the epoch validation accuracy as a metric to save the model current status if there is a progress but I was wondering if it is the right way or should I use the lowest epoch validation loss or a combination of both ?<br>
I am working on multi class classification problem</p>

username: Eta_C
cooked: <p>From my own experience, I always save all model after each epoch so that I can select the best one after training based on validation accuracy curve, validation loss curve and training loss curve.</p>

username: Oussama_Bouldjedri
cooked: <p>yes but how do you proceed if you dont have the same epoch with the lowest validation loss and the highest validation accuracy?</p>

username: Eta_C
cooked: <p>When everything else is normal, I prefer accuracy.<br>
Others are welcome to answer this question.</p>

username: Oussama_Bouldjedri
cooked: <p>in my case the validation plots accuracy and loss are not looking like the exemplar curve I suspect an overffiting so i am only releying on the accuracy so far</p>

username: huahuanZ
cooked: <p>As what I know, selecting the checkpoint at the lowest loss is the most common way. And as <a class="mention" href="/u/eta_c">@Eta_C</a> says, I would also recommend to save all checkpoints and do model averaging after the training is done. Here is an example <a href="https://github.com/pytorch/fairseq/blob/main/scripts/average_checkpoints.py" rel="noopener nofollow ugc">fairseq/average_checkpoints.py at main · pytorch/fairseq (github.com)</a></p>

username: Oussama_Bouldjedri
cooked: <p>let us say I care more about accuracy this is what I should use as improvement  and I dont get the best epoch in which the lowest loss is and the highest validation , + the fact that I am getting overfitting</p>

username: Eta_C
cooked: <p><a class="mention" href="/u/huahuanZ">@huahuanZ</a> <code>torchcontrib.optim.SWA</code> also implements model averaging <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Pytorch nn.functional.grid_sample fill with minimum value?

username: Cjia_Wang
cooked: <p>I believe it is very useful to support fill with minimum value to the grid_sample function. Is it possible to support this new “padding” modes in future pytorch releases?</p>
<p>Because normalizing the input tensor to zero mean 1 std is very popular. In this case, the background intensity value is &lt;0, and all the three “padding” modes (zero, border, reflect) introduce an obvious “frame” to the transformed images/volumes even if they are noise-free. This have direct negative effects of, e.g., image registration applications. If we have native support to fill a “Minimum” value, this problem can be easily solve. Thank you very much!</p>
Why does Pytorch data loader load an image num_workers times?

username: Rhea_Anand
cooked: <p>I was plotting the images loaded by my data loader using an iterator and I observed that the images get repeated num_workers times in the iterator.</p>
<p>I am also using batch_size 1, is there a relationship between the batch size and max number num_workers can have? Should it be less than or equal to the batch_size</p>
<p>initialising the dataloader</p>
<p><code>train_dataloader = DataLoader(train_datagen1, shuffle=True, num_workers=2, batch_size=1)</code></p>
<p>iterating through the images</p>
<pre><code class="lang-auto">dataiter = iter(train_dataloader)
images, labels = dataiter.next()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/4651d930006a503b89b33ec88044e4ae1a68940d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4651d930006a503b89b33ec88044e4ae1a68940d" title="train_results_2_shuffle"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/4651d930006a503b89b33ec88044e4ae1a68940d_2_500x500.jpeg" alt="train_results_2_shuffle" data-base62-sha1="a24Mv8UG7k8cTHDFbhvq1b3yraR" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/4651d930006a503b89b33ec88044e4ae1a68940d_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/4651d930006a503b89b33ec88044e4ae1a68940d_2_750x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/4651d930006a503b89b33ec88044e4ae1a68940d_2_1000x1000.jpeg 2x" data-dominant-color="A39EA4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">train_results_2_shuffle</span><span class="informations">1920×1920 157 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: huahuanZ
cooked: <p>Just confirm:</p>
<ol>
<li>what type of dataset is the <code>train_datagen1</code>? Is is an <code>IterableDataset</code>? If is, the duplicate data is expected. For more info, have a look at <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.10 documentation</a>
</li>
<li>Maybe it’s not the problem of dataloader. Ensure the images in <code>train_datagen1</code> are not duplicated.</li>
</ol>
Can't use convnext models

username: fike
cooked: <p>Hi all,</p>
<p>I am trying to use ConvNeXt models in my implementation, but everytime I call the model (either it’s the tiny, base, or whatever) I get the following error:</p>
<p><code>self.model = models.convnext_tiny(pretrained=True)</code><br>
<code>AttributeError: module 'torchvision.models' has no attribute 'convnext_tiny'</code></p>
<p>The last torch installation I have was made using:</p>
<p><code>pip3 install torch==1.10.1+cu102 torchvision==0.11.2+cu102 torchaudio===0.10.1+cu102 -f https://download.pytorch.org/whl/cu102/torch_stable.html</code></p>
<p>Any idea what might be going on? Thanks!</p>

username: ptrblck
cooked: <p>It seems this model just landed 28 days ago in <a href="https://github.com/pytorch/vision/pull/5330">this PR</a> so you would need to install the nightly release, not the stable one.</p>

username: fike
cooked: <p>Great, I was figuring it had to do something with the version. Thanks a lot!</p>
Compute Neural Network Gradients

username: jinmang2
cooked: <p>Hi! After reading the material below, I have a question.</p>
<aside class="onebox pdf" data-onebox-src="https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1184/readings/gradient-notes.pdf">
  <header class="source">

      <a href="https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1184/readings/gradient-notes.pdf" target="_blank" rel="noopener nofollow ugc">web.stanford.edu</a>
  </header>

  <article class="onebox-body">
    <a href="https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1184/readings/gradient-notes.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1184/readings/gradient-notes.pdf" target="_blank" rel="noopener nofollow ugc">gradient-notes.pdf</a></h3>

  <p class="filesize">198.22 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>According to ch2 useful identities (5), (6) in the pdf above, the formula is defined as follows.</p>
<p>(5) Matrix times column vector with respect to the matrix</p>
<p>(6) Row vector time matrix with respect to the matrix</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d43d42c7a6a735d74cf4fd1428b54d28e279565.png" alt="image" data-base62-sha1="hS90ooRgpFalO9l5qUY0Fy2GJIV" width="176" height="81"></p>
<p>I was wondering if this would actually be true, so I implemented it with pytorch</p>
<pre data-code-wrap="python"><code class="lang-python">import random
import numpy as np
import torch


def set_seed(seed: int = 42):
    """Seed fixer (random, numpy, torch)
    Args:
        seed (:obj:`int`): The seed to set.
    """
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)  # if use multi-GPU
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    
    
set_seed()

# (5) column vector case
x = torch.randn(5, 1).requires_grad_()
W = torch.randn(3, 5).requires_grad_()

y = W @ x

torch.isclose(
    torch.ones_like(y) @ x.T, # delta * x^T
    torch.autograd.grad(y, W, torch.ones_like(y), retain_graph=True)[0]   
)

# (6) row vector case
x = torch.randn(1, 2).requires_grad_()
f = torch.nn.Linear(2, 3, bias=False).requires_grad_()

y = f(x)
torch.autograd.backward(y, torch.ones_like(y), retain_graph=True)

assert torch.isclose(
    x.T @ torch.ones_like(y), # x^T * delta
    # Since 
nn.Linear in torch stores weights as transpose
    torch.autograd.grad(y, f.weight, torch.ones_like(y), retain_graph=True)[0].T 
)
</code></pre>
<p>In my results, the row vector case was the same as the pdf, but the column vector case was different.</p>
<p>And even when I understood it as a formula, I did not understand it.</p>
<p>In (5), delta means \frac{\partial J}{\partial z}, which will be a column vector. In order to perform outer product, it is necessary to calculate column vector X row vector, but it is strange to take transpose.</p>
<p>Is the formula wrong? Or is the implementation wrong?</p>
Custom multi-target cross entropy loss

username: OrestisPnk
cooked: <p>Hi, i am writing a custom multi-target cross entropy loss, using the sum of log_softmax for the wanted target classes per sample. Something like:</p>
<pre><code class="lang-auto">  def _mce_loss(scores, targets):
                loss = []
                for k in range(len(scores)):
                    loss.append(-F.log_softmax(scores[k])[targets[k]].sum())
                return torch.tensor(loss, requires_grad=True)
</code></pre>
<p>I am not sure how backwards() is going to penalize/reward each class, considering that I am just summing the log_softmax score of the wanted classes each time.<br>
Any suggestions?</p>
Grad is a 'NoneType' Object

username: Paritosh
cooked: <p>Hi all,</p>
<p>I am trying to zero out some gradients. While trying to access these gradients, I am getting an error saying that grad is a ‘NoneType’ object. I am trying to access grads in the following way:<br>
<code>x.grad[:,6:] = 0.</code></p>
<p>Although I can print out the grads, as shown in the following. Can someone please clarify what I am doing wrong? Instead what I should be doing? I would like to zero out some of the grads. The parameters are SMPL model’s parameters, if that information is critical. I have checked that <code>requires_grad</code> is set to <code>True</code>.</p>
<p>I am using PyTorch version 1.7.1.</p>
<blockquote>
<p>x grads:  tensor([[-0.0270, -0.0827,  0.1278, -0.0646,  0.0180,  0.1799, -0.0155, -0.0112,<br>
-0.0456, -0.0226, -0.0908, -0.0457,  0.0180, -0.0526,  0.0139,  0.0440,<br>
…<br>
0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,<br>
0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000]])</p>
</blockquote>
<p>Thank you very much.</p>
TensorBoard with BERT

username: TARUN_BHATIA
cooked: <p>I am trying to use add_graph method of tensorboard to display my bert layers in a graphical manner. However, I am getting the folowing error when i am trying to do that :</p>
<pre><code class="lang-auto">'NoneType' object is not iterable
</code></pre>
<p>My statement goes like :</p>
<pre><code class="lang-auto">writer = SummaryWriter()
model = XLMRobertaModel.from_pretrained(config.BASE_MODEL,output_attentions = False, output_hidden_states = False)
writer.add_graph(model)
writer.close()
</code></pre>
<p>Can anyone tell if i am missing anything ? The model is already a nn.Module .</p>

username: ptrblck
cooked: <p>I guess the error is raised as <code>input_to_model</code> is set to <code>None</code>by default and might be expected to create the graph.<br>
Try to provide a model input and see if this solves the error.</p>
Optimizer.step() single step?

username: vojkan7
cooked: <p>Hi everyone.</p>
<p>At the moment I’m working on split learning</p>
<p>Forward propagation is done from “n” to “m” layer. And then I calculated gradients x.backward()<br>
The next task is to “Apply gradients up to Ln+2”, which means from layer last layer m until layer n+2.</p>
<p>My question is, when I call optimizer.step(), is it going to apply for all until layer n or just one layer?</p>

username: ptrblck
cooked: <p><code>optimizer.step()</code> will update all parameters, which were passed to this <code>optimizer</code> during its initialization (or added later via <code>.add_param_group()</code>), using their <code>.grad</code> attributes.</p>

username: vojkan7
cooked: <p>thank you for your answer<br>
Is there any possibility/ way to say “just update until this layer”?</p>

username: ptrblck
cooked: <p>You could create separate optimizers by passing only the desired subset of parameters to them and call the <code>step()</code> function of the corresponding optimizers during the training.</p>
RTX 3090 Compatibility

username: RicciH8
cooked: <p>Hello,</p>
<p>I’ve seen many posts online from last year about this but I’m still to find a solution that works for me.</p>
<p>I get this message: “GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation”.</p>
<p>Is there a solution now or a planned release date for a version with compatibility? I’ve tried most answers from other posts with no avail.</p>
<p>Thanks in advance,</p>
<p>Ricci</p>

username: ptrblck
cooked: <p>You would have to install the PyTorch binaries with the CUDA11 runtime from <a href="https://pytorch.org/get-started/locally/">here</a> or alternatively you could also build form source.</p>

username: RicciH8
cooked: <p>Thanks. It’s working.</p>

username: ykpgrr
cooked: <p>You can use ‘lambda stack’ instead of installation pytorch from source.<br>
Fast and easy:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://lambdalabs.com/blog/favicon.png" class="site-icon" width="400" height="400">
      <a href="https://lambdalabs.com/blog/install-tensorflow-and-pytorch-on-rtx-30-series/" target="_blank" rel="noopener nofollow ugc" title="03:00PM - 31 January 2021">Lambda Blog – 31 Jan 21</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/369;"><img src="https://lambdalabs.com/blog/content/images/2020/11/lambda-stack-1.png" class="thumbnail" width="690" height="369"></div>

<h3><a href="https://lambdalabs.com/blog/install-tensorflow-and-pytorch-on-rtx-30-series/" target="_blank" rel="noopener nofollow ugc">Install TensorFlow &amp; PyTorch for the RTX 3090, 3080, 3070</a></h3>

<p>Instructions for getting TensorFlow and PyTorch running on NVIDIA's GeForce RTX 30 Series GPUs  (Ampere), including RTX 3090, RTX 3080, and RTX 3070.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: SumNeuron
cooked: <p>I did so via conda (cudatoolkit=11.3) and yet torch.version.cuda is 10.2.</p>

username: ptrblck
cooked: <p>In that case your install command might either have been wrong or you might have different PyTorch versions installed in the same environment. Create a new virtual environment and reinstall PyTorch with the CUDA 11 runtime.</p>
GPU Memory is always high

username: nicozhou
cooked: <p>May I know why pytorch always need to stall the GPU memory? Or is there any way to release it?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89367d84e66e6ca6fb412b70ed4c574e4a7af733.png" data-download-href="https://discuss.pytorch.org/uploads/default/89367d84e66e6ca6fb412b70ed4c574e4a7af733" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89367d84e66e6ca6fb412b70ed4c574e4a7af733.png" alt="image" data-base62-sha1="jzQ9OTg8UCLtx6B9unztakxRkRB" width="683" height="500" data-dominant-color="F4F7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">932×682 34.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>PyTorch uses a caching mechanism to be able to reuse already allocated memory.<br>
You can check the reserved and allocated memory e.g. via <code>print(torch.cuda.memory_summary())</code> to see how much memory is currently allocated and in the cache.<br>
To release the memory from the cache (and to reallocate it with a synchronizing <code>cudaMalloc</code>) you could use <code>torch.cuda.empty_cache()</code> so that other processes can use it.</p>
Visualize heterogeneous graph

username: yona
cooked: <p>Hello all</p>
<p>I am learning heterogeneous graph via this example : <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/load_csv.html" class="inline-onebox" rel="noopener nofollow ugc">Loading Graphs from CSV — pytorch_geometric 2.0.4 documentation</a><br>
I would like to visualize the graph built.</p>
<p>I thought about using the function via to_networkx() to access to networkx library and see the graph.</p>
<p>My problem is : heteregenous class seems to not have this function. Only data class.</p>
<p>How can I transform heteregenous data to data and then use to_networkx() ?</p>
<p>Any code/advices/ressources will be appreciated.</p>
Large input files for each channel

username: UCDuan
cooked: <p>Hi all,<br>
I’m trying to train a NN with multiple channels as inputs. Each channel is approximately 6 GB. It works when I only use 5 channels but failed with 8 channels because of the memory limit. The code for my dataset is the following:</p>
<pre><code class="lang-auto">n_features = len(forcings)
f_data = xa.open_dataarray(path+forcings[0]+'.nc')
self.x_d = np.zeros((
    f_data.shape[0], n_features, f_data.shape[1], f_data.shape[2]))
for i, forcing in enumerate(tqdm(forcings, desc='Forcings')):
    f_data = xa.open_dataarray(path+forcing+'.nc').data
    print(forcing, ' loaded')
    f_data[np.isnan(f_data)] = 0
    self.x_d[:, i] = f_data
    del f_data
self.x_d, self.y_d = torch.from_numpy(self.x_d.astype('float32')), \
            torch.from_numpy((self.y_d.astype('float32')))
</code></pre>
<p>And getitem is simple:</p>
<pre><code class="lang-auto">def __getitem__(self, index):
    return self.x_d[index], self.x_s, self.y_d[index]
</code></pre>
<p>Is it possible to do a ‘lazy’ loading with different channels? And is there any way to speed up the io process? For the first four channels, it takes 30 s to load one channel and the time increases to 15 mins for the rest channels.</p>
<p>Thanks!</p>

username: nivek
cooked: <p>It is possible to lazily load your data using <code>torch.utils.data.IterableDataset</code>, you can find the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" rel="noopener nofollow ugc">documentation here</a>. You should be able to write a <code>__iter__</code> method that opens all the files and yield one tuple of <code>(x_d, x_s, y_d)</code> at a time.</p>
<p>You can also use <code>IterDataPipe</code> from the <a href="https://github.com/pytorch/data#what-are-datapipes" rel="noopener nofollow ugc">TorchData library</a>, but do note that the project is still in the prototype stage (will be in beta soon).</p>

username: UCDuan
cooked: <p>Thanks! I have checked <code>IterableDataset</code> and it works for me. I have put the IO process in the <code>__iter__</code> method. I assume we cannot avoid this I/O slowdown during the training right?</p>
Pytorch model inference time optimization

username: vivekstorm
cooked: <p>I am trying to reduce the model inference time/computation time in pytorch by setting number of threads to the max available threads. But it is not helping with inference time reduction, it have increased the overall inference time. By default the num of threads are half the available cores but it is faster compared to setting to max threads. Can anyone explain why its happening and suggest some of the ways to optimize inference time on CPU.</p>
<p>Things I have tried:</p>
<blockquote>
<p>Setting num_threads to max cpu cores available<br>
Try using python multiprocessing library to do parallel inference on multiple images<br>
Tried Batching the images<br>
What other ways can I try to reduce the inference time.</p>
</blockquote>
Predict test data

username: Roua_Rouatbi
cooked: <p>I have my model ready trained on data with labels called ‘labels2’ that have 0 and 1. I want to predict the new data and for it to have labels 0, 1, and 2 if it doesn’t recognize it.<br>
This is the code I have written so far.<br>
the dataset in imgset_loader’s shape is (4, 1, 32, 32)</p>
<pre><code class="lang-auto">labels=[0,1]
for i, images in enumerate(imgset_loader):
    images = images.to(device)
    net = net.double()
    outputs = net(images)
    _, predicted = torch.max(outputs, 1)

    print((i,labels[predicted]))
</code></pre>
<p>But I think that something is wrong with it because I keep getting 1 as labels for all of them and when reloading the data it would show something random.</p>
<p>would it also be possible to show the name of the image along with the predicted value?</p>

username: Yuv
cooked: <aside class="quote no-group quote-modified" data-username="Roua_Rouatbi" data-post="1" data-topic="145267">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roua_rouatbi/48/41425_2.png" class="avatar"> Roua_Rouatbi:</div>
<blockquote>
<p><code> print(labels[predicted[0]])</code></p>
</blockquote>
</aside>
<p>Why you use only the first element of predicted?</p>

username: Roua_Rouatbi
cooked: <p>predicted is a tensor of 1 but even if I change it to predicted[i] for example. it would still come off as 1</p>

username: Yuv
cooked: <p>is your minibatch size 1?<br>
also - how exactly the model should guess the label 2 when you teach it only on label 0 and 1?</p>

username: Roua_Rouatbi
cooked: <p>yes, it is a batch of 1.<br>
and about label 2 it’s actually part of my question. do I need to retrain the model to recognize label 2?</p>

username: aiGeekProgrammer
cooked: <p>There is no such thing as “doesn’t recognize” as in:</p>
<blockquote>
<p>I want to predict the new data and for it to have labels 0, 1, and 2 <strong>if it doesn’t recognize it</strong>.</p>
</blockquote>
<p>There will be always either 0 prediction or 1 prediction.</p>
<p>In other words, the model must be trained to recognize all labels you want to predict.</p>
Valid padding or zero padding?

username: ethanlong
cooked: <p>Let’s say I want the convolutional layer output volumes to preserve the spatial size of the input volumes, but the pooling layer output volumes reduce the dimension of the input volume. Is this to say that I’ve to use zero padding? When the stride is 1 and the pooling value is 2, how will this affect the relationship between the kernel size and the padding? The input dimension is [60000, 28, 28]</p>

username: aiGeekProgrammer
cooked: <p>I don’t think you can use maxpool and still be able to preserve dimensions.</p>
<pre><code class="lang-auto">import torch
from torch.nn import MaxPool2d
input = torch.rand((1, 60000, 28, 28))
maxpool = MaxPool2d(2,stride=1, padding=1)
output = maxpool(input)
output.size()
&gt;&gt;&gt;torch.Size([1, 60000, 29, 29])
</code></pre>
<p>You may play with different values of maxpool parameters, but “…pad should be smaller than or equal to half of kernel size…”</p>
Loss Permuting Tensor Along Axis

username: Jack_Rolph1
cooked: <p>I have a loss function where I must perform a weighted means squared, where I check every possible permutation of the output along a certain axis. This is because I cannot be certain of the order my network will output its values in, and do not want to bias the output.</p>
<p>I have implemented and commented the following code:</p>
<pre><code class="lang-auto">import torch
from itertools import permutations
import torch.nn.functional as F

batch_size = 20
n_vertices = 100
n_features = 5


## Generate a set of mock output and truth data of size B, V, F
y_hat = torch.randn(batch_size, n_vertices, n_features, requires_grad=True)
y = torch.randn(batch_size, n_vertices, n_features, requires_grad=True)

## Activate mock output and truth data  using logsoftmax
y_hat = torch.exp(F.log_softmax(y_hat, dim=-1))
y = torch.exp(F.log_softmax(y, dim=-1))


## Generate a set of mock weight data of size B, V
weight = F.relu(torch.randn(batch_size, n_vertices, requires_grad=True))


## Generate the indices all the possible permutations of the F axis
idxs = list(permutations(np.arange(output.size(-1))))


## Reshape weight factor, output-&gt; B,V,1,1

weight =  weight.view(batch_size, n_vertices, 1, 1)


## Index output tensor -&gt;  output-&gt; B,V,Idx,F
y_hat = y_hat[:, :, idxs]

## Repeat truth tensor -&gt;  output-&gt; B,V,Idx,F
y = y.view(batch_size, n_vertices, -1, n_features).repeat(1, 1, len(idxs), 1)

## Calculate loss
loss = weight*(y_hat - y)**2


## Permute the axes, output-&gt; B,Idxs, V, F
## Flatten the last two axes, output -&gt; B, Idxs, VxF
loss = loss.permute(0,2,1,3).flatten(start_dim=2)


## Calculate the sum across the VxF axis (sum of each permutation)
loss = torch.sum(loss, axis=-1)


## Calculate the minimum loss across the Idx axis (min of each batch) 
loss = torch.min(loss, axis=-1)[0]


## Calculate the mean across the batches
loss = torch.mean(loss)

</code></pre>
<p>Can someone confirm this is the correct operation? Are the gradients still calculated in spite of the indexing? Is there a better way of doing what I am trying to achieve?</p>
Variational Autoencoder for sparse data

username: Federico_Ottomano
cooked: <p>Hello, I’m using a variational autoencoder in a project and I wanted to ask some opinions about few issues:</p>
<ul>
<li>
<p>My inputs are made by the concatenation of two sparse vectors which (independently) sum to one (So they belong to some $D$-dimensional simplex).</p>
</li>
<li>
<p>I’m not sure about which loss function might be suitable to reconstruct these kind of data, I’ve actually read that<code> torch.nn.CosineLossEmbedding()</code> could be a good choice but I’m not seeing noticeable results for now.</p>
</li>
<li>
<p>I’ve tried to eliminate biases from all my linear layers.</p>
</li>
<li>
<p>As my input consists of two separate vectors, could it make sense to provide a stack of different encoders/decoders to the architecture (2 in this case) to be sure that they are somehow processed <em>separately</em>?</p>
</li>
</ul>
DataLoader: is returning CUDA tensors always bad in distributed training?

username: yutanagano
cooked: <p>Hi all,</p>
<p>I was reading through the pytorch documentation for the DataLoader class and I noticed that it recommends against DataLoaders returning CUDA tensors in distributed training.</p>
<blockquote>
<p>It is generally not recommended to return CUDA tensors in multi-process loading because of many subtleties in using CUDA and sharing CUDA tensors in multiprocessing (see <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-cuda-note" rel="noopener nofollow ugc">CUDA in multiprocessing</a>). Instead, we recommend using <a href="https://pytorch.org/docs/stable/data.html?highlight=dataloader#memory-pinning" rel="noopener nofollow ugc">automatic memory pinning</a> (i.e., setting <code>pin_memory=True</code>), which enables fast data transfer to CUDA-enabled GPUs.</p>
</blockquote>
<p>However, I also notice that it says “generally”. I’m wondering if my particular case is an exception.</p>
<p>I have written up a distributed training pipeline using the <code>DistributedDataParallel</code> framework. I have DataLoader objects that are instantiated in each child process spawned by <code>torch.multiprocessing.spawn()</code>. One CUDA device is allocated per child process, and the index of the device is passed to each of the processes so they know which device they should use. When the DataLoaders are created in each of these child processes, the DataLoaders also get told which devices they are operating on (I have written a custom DataLoader class) so that in the <code>collate_fn</code> of each of these data loaders, the tensors are created directly on their respective device.</p>
<p>In my case, I can’t see how creating the tensors directly on the CUDA device and returning that could be a bad thing. The dataloaders always know exactly which CUDA device the tensors need to be on, and since only one process uses each device there shouldn’t be any CUDA memory sharing problems right?</p>
<p>I just wanted to double check this with more knowledgable people here on the forum to make sure I haven’t completely misunderstood the point.</p>
<p>Cheers,<br>
Yuta</p>

username: ptrblck
cooked: <p>I don’t think the quoted recommendation targets a distributed training setup, but the usage of multiprocessing e.g. via <code>num_workers&gt;0</code> in a <code>DataLoader</code>.</p>
<p>The main issues in such a use case is to avoid re-creating the CUDA context in each process, which will fail.</p>

username: yutanagano
cooked: <p>Ah, understood. Thank you for clarifying!</p>
Windows 10 GPU profiling

username: Neuroxic
cooked: <p>Hello,</p>
<p>I tested the same code on Linux and Windows.</p>
<p>Code snipped:</p>
<pre><code class="lang-auto">    with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
        main(args)
    prof.export_chrome_trace("trace.json")
</code></pre>
<p>While on Linux I saw the CPU and GPU workload on Windows I only saw the CPU usage. I’ve used “chrome://tracing/”.</p>
<p>Is there any option or setting that I’ll have to activate to see the GPU workload under Windows?</p>
<p>Thanks for your help!</p>

username: Artius
cooked: <p><a class="mention" href="/u/neuroxic">@Neuroxic</a> hi! I faced the same problem. Have u found a solution?</p>
Image normalization and transformation

username: Roua_Rouatbi
cooked: <p>I’m trying to normalize the intensity of the images in the dataset that I have and crop the images to 32 by 32 pixels. The image dataset shape is (36, 2152, 2552, 3) and I want it to be something like (x,32,32)</p>

username: ptrblck
cooked: <p>You could use <code>torchvision.transforms</code> to crop and normalize the input data.<br>
Could you describe a bit more where exactly you are stuck or which errors you are facing?</p>

username: Roua_Rouatbi
cooked: <p>my data is not normalized between 0 and 1<br>
here is my code</p>
<pre><code class="lang-auto">from PIL import Image
import os, sys
import cv2
import numpy as np

path = "C03/"
dirs = os.listdir( path )
dirs.sort()
x_test=[]

def load_dataset():
    for item in dirs:
        if os.path.isfile(path+item):
            im = Image.open(path+item).convert('RGB')
            im = np.array(im)
            x_test.append(im)
            
load_dataset() 
imgset=np.array(x_test)
np.save("imgds.npy",imgset)

import torchvision.transforms as T
preprocess = T.Compose([
   T.ToPILImage(),
   T.CenterCrop(32),
   T.ToTensor(),
   T.Normalize(
       mean=[0.485, 0.456, 0.406],
       std=[0.229, 0.224, 0.225]
   ),
   T.Grayscale(num_output_channels=1)
   
])
x=[]

for i in imgset: 
    i=np.array(preprocess(i))
    x.append(i)
x=np.array(x)
</code></pre>
<p>And I would love to, later on, apply this code to the data:</p>
<pre><code class="lang-auto">imgset_test = x.astype(np.float64)
imgset_test = imgset_test/ (2**14)

imgset_test = imgset_test.reshape(imgset_test.shape[0],1,32,32)
imgset_loader  = DataLoader(dataset=x,  batch_size=batch_size, shuffle=False).
</code></pre>
<p>and then predict the dataset labels by a model I already trained</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Roua_Rouatbi" data-post="3" data-topic="145131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roua_rouatbi/48/41425_2.png" class="avatar"> Roua_Rouatbi:</div>
<blockquote>
<p>my data is not normalized between 0 and 1</p>
</blockquote>
</aside>
<p>I don’t know which range your data values are in, but <code>T.Normalize</code> using the current stats would expect an input with values in <code>[0, 1]</code>, so you might need to normalize the raw data before applying the transformation.</p>

username: Roua_Rouatbi
cooked: <p>yes I guess I forgot to add that since I switched notebooks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
U-NET - Black Tiles

username: TWman
cooked: <p>Hi, everybody!</p>
<p>New pytorcher here. I am trying to implement a binary semantic segmentation approach using UNET.<br>
To get myself started I took some code from <a href="https://medium.com/analytics-vidhya/creating-a-very-simple-u-net-model-with-pytorch-for-semantic-segmentation-of-satellite-images-223aa216e705" rel="noopener nofollow ugc">this example.</a></p>
<p>My dataset contains some 250 images and I used rotations to expand that to nearly 1000.</p>
<p>Here is an example of my images and targets:<br>
<a href="/uploads/short-url/wkMcXO7E93vykPUfw71ysPfff9s.jpeg">target mask|537x252</a></p>
<p>Now when I train the model two odd things happen:</p>
<ol>
<li>The model achieves high accuracy very quickly, after just a couple of epochs.</li>
<li>All prediction masks are blank. The more epochs the blanker they are.</li>
</ol>
<p>After a certain amout of reading, I believe the problem is that the targets in my images are fairly small.<br>
The model thus gets a high accuracy for predicting zeroes for everything, thus essentially training itself to paint everything black.</p>
<p>The best solution appears to be:</p>
<ol>
<li>Way more training epochs</li>
<li>A weighted scoring approach where positive pixels are favoured.</li>
</ol>
<p>Any thoughts on this analysis and hints on how to achieve the weight scoring would be appreciated, I’m still very new to all this.</p>
<p>Cheers!</p>

username: TWman
cooked: <p>Update:</p>
<p>I trained the model for 50 epochs and the resulting predictions were again blanks.</p>
<p>I then assembled a data set in which all the targets made up a large part of the images.<br>
(small tagets potentially being the issue)</p>
<p>After training the new set for 10 epochs I got this result. Not perfect but not terrible.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b27e000dadca591524475fe3d603afa2d2b92c7.jpeg" alt="10(1)" data-base62-sha1="jR1NC1mS46TumXdeLiIgdxeFyHt" width="577" height="268"></p>
<p>However retraining the same set with the same settings from scratch, delivers this were everything is turned around:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4ce73b210d39843513d685351742f7be8d095c6a.jpeg" alt="10(2)" data-base62-sha1="aYjHgsdRF96RCYXNSoOmraH2QHg" width="554" height="266"></p>
<p>Also training for more than 10 epochs again creates blank predictions.</p>
<p>Has anyone encountered this before?</p>

username: ptrblck
cooked: <p>The blank predictions can indeed be produced by a highly imbalanced target distribution in the mask images and you could try to use a weighted loss function to counter this effect.<br>
I’m unsure about the last effect of flipped predictions and then going to a blank prediction again.</p>

username: TWman
cooked: <p>Many thanks for the reply.</p>
<p>I have experimented with smaller image tiles 256px instead of 512px, and the results appear to be similar.</p>
<p>I will look into putting together a weighted loss function.</p>

username: ver
cooked: <p>Can you please mention if got the correct results with weighted loss and what was that</p>
Error when running DistributedDataParallel

username: watchdog90
cooked: <p>Hi, I followed this post: <a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html" rel="noopener nofollow ugc">Distributed Pytorch Tutorial</a> in order to run my model on two gpus. However, when i run it I got the following message "<code>fused_weight_gradient_mlp_cuda</code> module not found. gradient accumulation fusion with weight gradient computation disabled." It does not say if it is a warning or error, and the script does nothing, meaning that the model is not running. Does anyone have an idea of what might be the issue?</p>

username: ptrblck
cooked: <p><code>fused_weight_gradient_mlp_cuda</code> is a custom layer used in <code>apex/Megatron</code> and is optional if you build the <code>apex</code> repository in your setup.<br>
I couldn’t quickly find any reference to Megatron in the blog post, but it seems that the author was using <code>apex</code> in 2019 for the automatic mixed-precision util. (this is not necessary anymore as you should use <code>torch.cuda.amp</code> in newer PyTorch releases) and might be using these custom model/pipeline-parallel utils now.</p>

username: watchdog90
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  thank you for clarifying this. Actually, I realized that the reason for not running my program is the followin. Before the spawn of training code to different nodes, there is this piece of code:</p>
<blockquote>
<pre><code>            args.world_size = args.gpus * args.nodes
            os.environ['MASTER_ADDR'] = args.MASTER_ADDRESS            
            os.environ['MASTER_PORT'] = '8088'  
            mp.spawn(training, nprocs=args.gpus, args=(args,))
</code></pre>
</blockquote>
<p>Which uses port 8088. When my code initializes visdom server, on port ‘8097’ it suddenly refuses to continue and stalls. Is there any workaround on this? I tried to change port but nothing happens.</p>
Compute weights for L1 loss

username: talhaanwarch
cooked: <p>I have a dataframe and i want to apply regression. But i want to use weighted L1 loss, so i calculated the weight of each label.</p>
<pre><code class="lang-auto">bins = np.digitize(train_df.label.values,[50,70,90], right=False)
weights=np.select([bins==1,bins==2,bins==3],[0.75,0.5,0.75])
weights=dict(zip(train_df.label.values,weights))
</code></pre>
<p>Now i have a dictionary which contain label and its corresponding weight. I want to pass these to my loss function</p>
<p>If i print the loss value after setting <code>reduction='none'</code><br>
i got following output</p>
<pre><code class="lang-auto">tensor([72.0564, 85.8526, 86.7617], device='cuda:0',
       grad_fn=&lt;SmoothL1LossBackward&gt;)
</code></pre>
<p>but i am confused how to calculate weighted loss.</p>
<p>I am trying some thing like this</p>
<pre><code class="lang-auto">loss=self.criterion(out,batch["ground_label"].float())
w_ar=torch.tensor([weights[i] for i in batch["ground_label"].numpy()]).cuda()
loss=torch.sum(loss*w_ar)
</code></pre>
<p>I dont know, this approach is correct or not, and this approach is slow because of cpu gpu conversion, how can i make it faster</p>

username: ptrblck
cooked: <p>If the tensor creation is slow in your code, you could calculate the weights once, store them on the GPU and index them using the current label tensor.</p>
<p>Your general approach look alright, but you should note that a weighted loss is usually normalized to avoid a dependency on the currently used class label distribution in the batch as described e.g. <a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/10">here</a>.</p>