akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Split segmentation mask into annotated areas

username: Mukeyii
cooked: <p>Hello there,</p>
<p>is there an easy way in PyTorch to split an index mask tensor into multiple masks, each containing only one contiguous labeled area (e.g. [num_areas, H, W])?</p>
<p>Tensor [1, 512, 512] (Index Tensor containing 0 or 1) to a Tensor [6, 512, 512]. 6 is the amount of contiguous areas, which is not static. The colored areas are equal to 1 in my current Tensor:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bcb7609c42de9936ff94e7138cd4bb3f18b41208.png" alt="Blobs-divided" data-base62-sha1="qVsEdweDM6BhfrY7z1ltKbtsoWc" width="384" height="372"></p>
<p>Greetings, Mukeyii</p>

username: KFrank
cooked: <p>Hi Kemal!</p>
<aside class="quote no-group" data-username="Mukeyii" data-post="1" data-topic="151294" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mukeyii/48/46742_2.png" class="avatar"> Mukeyii:</div>
<blockquote>
<p>is there an easy way in PyTorch to split an index mask tensor into multiple masks, each containing only one contiguous labeled area</p>
</blockquote>
</aside>
<p>What you  are looking for are the <em>connected components</em> (of an image,<br>
rather than of a graph).</p>
<p>I do not believe that pytorch supplies a built-in connected-components<br>
algorithm, but several pytorch connected-components implementations<br>
appear to be available on the internet.  (I haven’t tried any of them, so I<br>
have no opinion as to which of them might work.)</p>
<p>You could also write your own pytorch implementation – doing so would<br>
be a task, but relatively straightforward.</p>
<p>Or you could pre-process your segmentation masks with a non-pytorch<br>
image-processing library that does have a connected-components<br>
algorithm.</p>
<p>Best.</p>
<p>K. Frank</p>
Resnet50 example errors(Ubuntu Beginner)

username: angelos_b147
cooked: <p>Hello, I’m a beginner in Glow. I attempted to run the Resnet50 in my Ubuntu VMWare Virtual Machine and I consistently get the error below. I followed the Glow installation in <a href="https://github.com/pytorch/glow" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/glow: Compiler for Neural Network hardware accelerators</a> and made a special buildwithbundles folder for the Glow build as it is mentioned here <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md" class="inline-onebox" rel="noopener nofollow ugc">glow/AOT.md at master · pytorch/glow · GitHub</a> so I’m not entirely sure what I am doing wrong. Does the error have to do with Ubuntu being on a Virtual Machine, me not installing Glow correctly in some step? I’m not sure at this point, hence the forum post. Thank you for your time and sorry in advance if I’m asking a dumb question.</p>
<p>angel@ubuntu:~/buildwithbundles/bundles/resnet50$ ./ResNet50Bundle cat_285.png<br>
Allocated weights of size: 102121920<br>
Expected weights of size: 102121920<br>
Loaded weights of size: 102121920 from the file resnet50.weights.bin<br>
Allocated mutable weight variables of size: 606144<br>
ResNet50Bundle: /home/angel/glow/examples/bundles/resnet50/main.cpp:189: void loadImagesAndPreprocess(const std::vector&lt; std::string&gt; &amp;, float *&amp;, size_t *): Assertion `loadSuccess &amp;&amp; “Error reading input image.”’ failed.<br>
Aborted (core dumped)<br>
angel@ubuntu:~/buildwithbundles/bundles/resnet50$</p>

username: rpitonak
cooked: <p>Hello. I experienced the same problem. Did you find the solution?</p>

username: angelos_b147
cooked: <p>yes, I gave the full file address for the cat.png file so basically the command in my case would be<br>
./ResNet50Bundle /home/ampampos/tries/buildwithbundles/tests/images/imagenet/cat_285.png which is executed inside the folder of the bundle in the terminal and it worked</p>
Error while filtering MNIST dataset

username: Jorge_Garcia
cooked: <p>Hello!</p>
<p>I’m trying to filter MNIST dataset by using a desired label using the following code:</p>
<pre><code class="lang-auto">mnist_full = datasets.MNIST(self.data_dir, train=True, transform=self.transform)
idx = mnist_full.targets == self.labels
mnist_full = mnist_full[idx]
</code></pre>
<p>with self.labels=1</p>
<p>The expected behaviour would be to obtain a dataset only with 1’s, but I get the following error:</p>
<pre><code class="lang-auto">  File "/Work/vidanodet/./src/simple_GAN.py", line 55, in setup
    mnist_full = mnist_full[idx]
  File "/usr/local/lib/python3.9/dist-packages/torchvision/datasets/mnist.py", line 127, in __getitem__
    img, target = self.data[index], int(self.targets[index])
ValueError: only one element tensors can be converted to Python scalars

</code></pre>
<p>I’ve checked idx is a tensor with 1 dimension containing booleans. Any idea of the problem?</p>
<p>Thanks!</p>

username: Matias_Vasquez
cooked: <p>The error you are getting is because you are trying to cast a Tensor with many values into an int. If the result of <code>self.targets[index]</code> was a tensor with only one value then there would be no problem, but the python built-in function does not know what to do with a tensor with many values.</p>
<pre><code class="lang-python"># Error
int(self.targets[index])

# This should work
self.targets[index].int()
</code></pre>

username: Jorge_Garcia
cooked: <p>¡Hello!</p>
<p>You’re right but that is torchvision code. I wasn’t accessing to the dataset inner data structures to index them correctly. Here the correct code:</p>
<pre><code class="lang-auto">mnist_full = datasets.MNIST(self.data_dir, train=True, transform=self.transform)
idx = mnist_full.targets == self.labels
mnist_full.data = mnist_full.data[idx]
mnist_full.targets = mnist_full.targets[idx]
</code></pre>
Image augmentation techniques for Yolov5 based image segmentation

username: Venkatesh_Adiga
cooked: <p>Hi Team,</p>
<p>I am working on an image dataset where each image contains multiple labels and frequencies of some of the labels in the dataset is very few. Thus dataset is totally imbalanced. I am planning to use Yolov5 based object detection algorithm to detect the labels in the images.<br>
Can you please someone share some hints or guidance to solve the imbalance the dataset with respect to the labels and their annotations?<br>
Thank you.</p>
<p>Venkatesh</p>
Requirements.txt for multiple CUDA architectures

username: addisonklinke
cooked: <p>Say I’m developing a model with a Turing architecture GPU. Based on the official installation <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">guide</a>, my <code>requirements.txt</code> could simply be</p>
<pre><code class="lang-auto">torch&gt;=1.10
</code></pre>
<p>which works fine because CUDA 10.2 is the default runtime (which is compatible with Turing). However, if a colleague wants to continue developing the model or we’re looking to deploy on a machine with an Ampere architecture GPU, we’d need CUDA &gt;= 11.1 from</p>
<pre><code class="lang-auto">-f https://download.pytorch.org/whl/cu113/torch_stable.html
torch==1.10.0+cu113
</code></pre>
<p>Is there a best practice which would allow both environments to share a requirements file such that running <code>pip install -r requirements.txt</code> would result in the correct CUDA version for both GPU architectures?</p>
<p>I suppose one solution is always using the latest CUDA that PyTorch is shipping, but what if the older GPU architecture is not backwards compatible? Are there any other downsides to using the latest?</p>

username: marksaroufim
cooked: <p>So either you can have a conditional <code>requirements.txt</code> <a href="https://stackoverflow.com/a/35614580" class="inline-onebox" rel="noopener nofollow ugc">pip - Is there a way to have a conditional requirements.txt file for my Python application based on platform? - Stack Overflow</a> or create a different one for each cuda version (which is pretty common)</p>

username: addisonklinke
cooked: <p><a class="mention" href="/u/marksaroufim">@marksaroufim</a> I’m familiar with conditional <code>requirements.txt</code> based on Windows vs. Linux, but AFAIK there isn’t a way to specify CUDA version with that syntax. Is there a trick I’m missing?</p>
'GradSampleModule' object has no attribute

username: donarni
cooked: <p>Hello Opacus Team.</p>
<p>After wrapping a <strong>nn.Module</strong> with <strong>GradSampleModule</strong>, how can I access the methods (e.g., <em>def init_weights (self) :</em>) of the model?</p>
<p>Specifically, I want to use Opacus without <code>PrivacyEngine</code> to privatize the <code>Discriminator</code> of CT-GAN, however, the forward of CT-GAN fails to call <code>calc_gradient_penalty</code>.</p>
<pre><code class="lang-auto">class Discriminator(Module):

    """Discriminator for the CTGANSynthesizer."""

    def __init__(self, input_dim, discriminator_dim, pac=10):

        super(Discriminator, self).__init__()

        dim = input_dim * pac

        self.pac = pac

        self.pacdim = dim

        seq = []

        for item in list(discriminator_dim):

            seq += [Linear(dim, item), LeakyReLU(0.2), Dropout(0.5)]

            dim = item

        seq += [Linear(dim, 1)]

        self.seq = Sequential(*seq)

    def calc_gradient_penalty(self, real_data, fake_data, device='cpu', pac=10, lambda_=10):

        """Compute the gradient penalty."""

        alpha = torch.rand(real_data.size(0) // pac, 1, 1, device=device)

        alpha = alpha.repeat(1, pac, real_data.size(1))

        alpha = alpha.view(-1, real_data.size(1))

        interpolates = alpha * real_data + ((1 - alpha) * fake_data)

        disc_interpolates = self(interpolates)

        gradients = torch.autograd.grad(

            outputs=disc_interpolates, inputs=interpolates,

            grad_outputs=torch.ones(disc_interpolates.size(), device=device),

            create_graph=True, retain_graph=True, only_inputs=True

        )[0]

        gradients_view = gradients.view(-1, pac * real_data.size(1)).norm(2, dim=1) - 1

        gradient_penalty = ((gradients_view) ** 2).mean() * lambda_

        return gradient_penalty

    def forward(self, input_):

        """Apply the Discriminator to the `input_`."""

        assert input_.size()[0] % self.pac == 0

        return self.seq(input_.view(-1, self.pacdim))
</code></pre>
<p>Best regards!</p>

username: alexandresablayrolle
cooked: <p>Hi,</p>
<p>Thanks for reaching out. Unfortunately, Opacus does not yet support advanced computation graph manipulations (such as torch.autograd.grad()). We are currently looking at functorch to potentially support that kind of operations in the future.</p>

username: donarni
cooked: <p>Thank you for your quick reply. I guess I’ll try another GAN variant then.</p>
Notebook to demonstrate the exploding gradients for the RNN vs. LSTM

username: Thomas_Lees
cooked: <p>I am trying to demonstrate to myself the fact that the standard RNN architecture struggles to</p>
<p>I can read about the theory behind this and believe it to be true (Stack Exchange Answer -<a href="https://stats.stackexchange.com/a/370732/213097" class="inline-onebox" rel="noopener nofollow ugc">machine learning - Why do RNNs have a tendency to suffer from vanishing/exploding gradient? - Cross Validated</a> / arxiv paper - Pascanu et al 2012 / Nielsen 2019 book neuralnetworksanddeeplearningcom ). But I want to create an experiment to <strong>demonstrate</strong> that it is true.</p>
<p>I have tried mocking up something <a href="https://gist.github.com/tommylees112/79c7fa6f6a2161f2ac9679780c1ebe19" rel="noopener nofollow ugc">here</a>.</p>
<p>The main questions I have:</p>
<ol>
<li>
<p>My hypothesis is that the increased <code>seq_length</code> makes the gradient issues worse, however I don’t see much evidence for this, despite expecting it to be true based on my reading of the above literature</p>
</li>
<li>
<p>Perhaps my design of the experiment is wrong, I am using the following code to extract the gradients:</p>
</li>
</ol>
<pre><code class="lang-python">if "flat_grads" not in globals().keys():
# if True:
    lstm.zero_grad()

    for ix, data_dict in enumerate(tqdm(all_dl, desc="Calculating Parameter Gradients")):
        x_d = data_dict["x_d"]
        y = data_dict["y"]

        # run forward pass
        y_hat = lstm(data_dict)
        loss = loss_fn(y.squeeze(), y_hat.squeeze())
        loss.backward()
    
    # get the gradients for all of the parameters
    lstm_grads = [pm.grad for pm in lstm.lstm.parameters()]
    flat_grads = np.concatenate([p.flatten() for p in lstm_grads])
</code></pre>
<p>Expecting that <code>loss.backward()</code> and <code>pm.grad</code> are enough to calculate the gradients of each parameter.</p>
<p>Is this the correct way to go about this?</p>
Functorch.grad and vmap

username: Wesley_Neill
cooked: <p>I want to compute the laplacian of a batch satellite images of a single location, treating each channel of each image as it’s own scalar field. The Laplacian is to be taken with respect to a single shared frame of reference. In other words, I may have a set of 10 images taken from 10 different angles, so we have developed a way to rectify them all into a single coordinate system.</p>
<p>The function I want to take the gradient of is this (psuedo-code):</p>
<pre><code class="lang-auto">def get_pixel_values(local_coords, images, xforms):
    geo_coords = convert_local_to_lat_lon(local_coords)
    pixel_coords = get_pixel_locations(geo_coords, xforms)
    scaled_pixel_coords = scale(pixel_coords)
    pixel_values = grid_sample(images, grid=scaled_pixel_coords)
</code></pre>
<p>My problem is that this function outputs more than a singleton tensor required by the <code>functorch.grad</code> docs:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fa2d10cfd6eaa60083c5a15e11ba8d393405c7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/3fa2d10cfd6eaa60083c5a15e11ba8d393405c7f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa2d10cfd6eaa60083c5a15e11ba8d393405c7f_2_690x122.png" alt="image" data-base62-sha1="94WUDTxLYzNvsER29JQde1jbUJ1" width="690" height="122" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3fa2d10cfd6eaa60083c5a15e11ba8d393405c7f_2_690x122.png, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fa2d10cfd6eaa60083c5a15e11ba8d393405c7f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3fa2d10cfd6eaa60083c5a15e11ba8d393405c7f.png 2x" data-dominant-color="F3F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1016×180 14.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’ve created a version of the above function that takes only a single coordinate, single image and single transform to meet <code>vmap</code> requirements:</p>
<pre><code class="lang-auto">def img_grad(local_coords, images, xforms)

    def single_input(single_coord, single_image, single_transform):
        return get_pixel_values(single_coord[None], single_image[None], single_xform[None])
</code></pre>
<p>but I’m stuck at meeting the <code>grad</code> requirements of a single element tensor output…</p>
<p>Any help appreciated. I’m sorry I can’t provide more than pseudocode.</p>
Flat training and validation error

username: Istvan_Ferenc_Toth
cooked: <p>Hello,</p>
<p>I am using 64x64 images (just to test my model 10000, split as 80%, 10%, 10%) to obtain a vector of 64 elements. I have tried different learning rates, weight decay, different architecture, different normalization of input, even normalization of the target vectors. The best I obtain for the MSE error is in the image below, where  the training and validation (despite the legend) are plotted. However, when I compare the output of the model to some vectors in the test set, the result is bad. Here the input is zero centralized and the target vectors are scaled to [-0.2, 0.2].</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/01a1a7e6ba83eb57b8b8e1f3e905177c92725b9c.png" data-download-href="https://discuss.pytorch.org/uploads/default/01a1a7e6ba83eb57b8b8e1f3e905177c92725b9c" title="denseFrog_adam_phase_img64_epochs9001_batch64_random_noise_loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01a1a7e6ba83eb57b8b8e1f3e905177c92725b9c_2_666x500.png" alt="denseFrog_adam_phase_img64_epochs9001_batch64_random_noise_loss" data-base62-sha1="eqP3M1gfzv6ah75pncaMgSbtla" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01a1a7e6ba83eb57b8b8e1f3e905177c92725b9c_2_666x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/1/01a1a7e6ba83eb57b8b8e1f3e905177c92725b9c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/1/01a1a7e6ba83eb57b8b8e1f3e905177c92725b9c.png 2x" data-dominant-color="F0F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">denseFrog_adam_phase_img64_epochs9001_batch64_random_noise_loss</span><span class="informations">800×600 24.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried for a single image and the error goes well down well beyond what is seen in the plot. What can cause this flattening of the errors after a few epochs? I welcome any help . Many thanks.</p>
Use pre-trained ViT as a backbone

username: whatevername
cooked: <p>Hi, I’m very new to this. I want to use the ViT B 16 pre-trained on ImageNet as backbone for the task of image classification on a different dataset. Given this trained backbone, the image representation is consequently used in combination with a kNN classifier. My code looks like this:</p>
<p>Initializing the model:</p>
<pre><code class="lang-auto">net = Embedder("vit_b_16", pretrained_flag = True)
</code></pre>
<p>The Embedder class:</p>
<pre><code class="lang-auto">class Embedder(nn.Module):

    def __init__(self,architecture,pretrained_flag = True):

        super(Embedder, self).__init__()

        #load the base model from PyTorch's pretrained models (imagenet pretrained)
        network = getattr(models,architecture)(pretrained=pretrained_flag)
        
        if architecture.startswith('vit'):
            self.backbone = network

    def forward(self, img):
        '''
        Output has shape: batch size x descriptor dimension
        '''
        x = self.backbone(img).squeeze(-1).squeeze(-1)

        return x
</code></pre>
<p>And here I extract the embeddings using torchvision’s feature_extraction tool and stopping at node getitem_5 (before the heads node):</p>
<pre><code class="lang-auto">def extract_embeddings(net,dataloader,ms=[1],msp=1,print_freq=None,verbose = False):

    feature_extractor = create_feature_extractor(net, return_nodes=['backbone.getitem_5'])

    if verbose:

        if len(ms) == 1:
            print("Singlescale extraction")
        else:
            print("Multiscale extraction at scales: " + str(ms))
    
    net.eval()
    
    with torch.no_grad():

        vecs = np.zeros((768, len(dataloader.dataset))) #768 -&gt; vb16; 1024 -&gt; vl16
    
        for i,input in enumerate(dataloader):

            if len(ms) == 1 and ms[0] == 1:
                vecs[:, i] = feature_extractor(net,input[0])['backbone.getitem_5'] #.cuda()
            
            else:
                vecs[:, i] = feature_extractor(input[0])['backbone.getitem_5'] 

            if print_freq is not None:
                print("image: "+str(i))

    return vecs.T

</code></pre>
<p>Am I using the ViT correctly? Do I need to add anything to the forward function?</p>
Use generalized_box_iou_loss

username: unofficial-Jona
cooked: <p>Hi all,<br>
I’m training a model to fit bounding boxes on images. All Bounding boxes are defined by two coordinates (x1,y1,x2,y2). To fit these bounding boxes I first used mse_loss. The loss converges to 0, but the results are still not great enough.<br>
I therefore tried to use <a href="https://pytorch.org/vision/stable/generated/torchvision.ops.generalized_box_iou_loss.html" rel="noopener nofollow ugc">generalized_box_iou_loss</a>. Unfortunately this does not work (the loss converges to 1). Any idea what the problem may be?</p>
<p>Best, Jona</p>
Automate code conversion between different ML framework

username: Allaye
cooked: <p>This is just a question, Currently, it is not possible to convert code directly between different ML frameworks, such as JAX and Pytorch. what do you think are the biggest challenges? and how can this be solved. just need peoples views on this.</p>

username: InnovArul
cooked: <p>I have come across <code>ivy</code> package which tries to achieve this.<br>
I have not tried it yet.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/unifyai/ivy">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/unifyai/ivy" target="_blank" rel="noopener">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/7e57eab0a75135e664262aa8e7e98a62efeec3f4085fa8a5a0e7e4137caafeda/unifyai/ivy" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/unifyai/ivy" target="_blank" rel="noopener">GitHub - unifyai/ivy: The Unified Machine Learning Framework</a></h3>

  <p>The Unified Machine Learning Framework. Contribute to unifyai/ivy development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Allaye
cooked: <p>i have come across it, i just want to know the technical problem one might face trying to achieve this.</p>
Different batch size on different gpus in ddp

username: divinit7
cooked: <p>I am training a classification model using 4 gpus. I see there are 3 extra processes running on gpu 0 so it wont fit a batch size larger than 2 but the others can still  accept a batch size of 4. I dont understand what the 3 extra processes are and is manually setting batch size for each gpu correct?</p>
<pre><code class="lang-auto">batch_size=4
if gpu == 0:
    batch_size = 2
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b16e41ec89109b607e74d0ebaa486de92d03def5.png" data-download-href="https://discuss.pytorch.org/uploads/default/b16e41ec89109b607e74d0ebaa486de92d03def5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b16e41ec89109b607e74d0ebaa486de92d03def5.png" alt="image" data-base62-sha1="pjCJKvThym2y4pEzkOwx0hlQnPL" width="690" height="428" data-dominant-color="161515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">946×588 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>It seems you might have created multiple CUDA contexts on the default device (GPU0). Are you launching the script via <code>torchrun</code>? If so, make sure that each script sets the proper device e.g. via <code>torch.cuda.set_device</code>.</p>

username: mrshenli
cooked: <p>+1 to <a class="mention" href="/u/ptrblck">@ptrblck</a>’s answer</p>
<p>You can also set <code>CUDA_VISIBLE_DEVICES</code> to make sure that each process only sees one device, so that they won’t unintentionally create CUDA context on <code>cuda:0</code></p>

username: divinit7
cooked: <p>I am using mp.spawn as suggested in other thread for single machine multi-gpu setup.</p>
Dtype=int for classification labels

username: LB97
cooked: <p>Hi,<br>
I would like to know, for a multi class classification task, if it is required to pass labels’ dtype as ‘int’. If yes, what doesn’t it signify?</p>

username: KFrank
cooked: <p>Hi LB!</p>
<aside class="quote no-group" data-username="LB97" data-post="1" data-topic="151200" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lb97/48/36823_2.png" class="avatar"> LB97:</div>
<blockquote>
<p>for a multi class classification task, if it is required to pass labels’ dtype as ‘int’.</p>
</blockquote>
</aside>
<p>For a multi-class classification task, one would typically use<br>
<code>CrossEntropyLoss</code> as the loss criterion.  In recent versions of pytorch,<br>
<code>CrossEntropyLoss</code> supports two types of labels.</p>
<p>Let’s say that your prediction (the <code>input</code> passed to <code>CrossEntropyLoss</code>)<br>
has shape <code>[nBatch, nClass]</code>.  Then the labels (the <code>target</code>) can be<br>
either integer categorical class labels of shape <code>[nBatch]</code> and type<br>
<code>torch.int64</code> (which is <code>long</code>) (but not type <code>torch.int32</code>, which is <code>int</code>).</p>
<p>Or they can be “soft” probabilistic labels of shape <code>[nBatch, nClass]</code><br>
and of the floating-point type that matches the type of <code>input</code> (typically<br>
<code>float32</code> or <code>float64</code>).</p>
<p>Best.</p>
<p>K. Frank</p>
Optimizer : TypeError: __init__() got an unexpected keyword argument 'momentum'

username: Leonmac
cooked: <p>Hello,<br>
I am trying to make a common train model for both MNIST and CIFAR10, the optimizer is SGD.<br>
However, I just find the below setup only works for CIFAR10:</p>
<pre><code class="lang-auto">optimizer(self.model.parameters(), lr=cmd_lr, momentum=0.9, weight_decay=1e-4)
</code></pre>
<p>if using MNIST, looks like I can not set the momentum (remove it will be OK), otherwise, I will get errors like:<br>
why does that happen? I can not figure out why the SGD for MNIST does not support momentum setup?</p>
<pre><code class="lang-auto">TypeError: __init__() got an unexpected keyword argument 'momentum'
</code></pre>
<p>The MNIST model is as below, which is a quite common model to be used.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self) -&gt; None:
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.dropout1 = nn.Dropout(0.25)
        self.dropout2 = nn.Dropout(0.5)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x) -&gt; torch.Tensor:
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = self.dropout1(x)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout2(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output
</code></pre>

username: Matias_Vasquez
cooked: <p>The initialization of the optimizer should not depend on which dataset you are using.</p>
<pre><code class="lang-python"># Example
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
</code></pre>
<p>Could you post the line where you get this error or a reproducible code to get this error?</p>

username: Leonmac
cooked: <aside class="quote no-group" data-username="Leonmac" data-post="1" data-topic="151185">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leonmac/48/30837_2.png" class="avatar"> Leonmac:</div>
<blockquote>
<p><code>momentum=0.9</code></p>
</blockquote>
</aside>
<p>Sorry, just take a closer examination–I actually passed the different optimizer to MNIST (Adadelta) other than to CIFAR10 (SGD) – that looks the Adadelta does not support the momentum setup…</p>

username: Leonmac
cooked: <p>Just find a good article to explain what is momentum and why SGD has it and other does not.</p>
Retain Graph- Branch Convolutional Neural Network

username: ganLover
cooked: <p>I trying to implement B-CNN: Branch Convolutional Neural Network for Hierarchical Classification.-<a href="https://arxiv.org/pdf/1709.09890.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1709.09890.pdf</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0a7cea3bc54349cbb09a5559ef690f5e3032ea1a.png" alt="image" data-base62-sha1="1uMp2dAAR4bUgxIoxhQtrWY1ULw" width="632" height="347"></p>
<p>I have a doubt-<br>
Should I backpropagate losses from the branches individually (using retain graph)</p>
<p>OR</p>
<p>Should i simply add the losses from all the branches and then perform backpropagation.</p>

username: Naruto-Sasuke
cooked: <p>Both is ok, while the latter seems better.</p>

username: ganLover
cooked: <p>I wanted to know why 2nd is better.</p>
<p>Is it because- in 2nd case I need only one backprop compared to 3 backprops in 1st case?</p>

username: k0pch4
cooked: <p>By better, I think <a class="mention" href="/u/naruto-sasuke">@Naruto-Sasuke</a> means that your code would “look” better and would have less lines, therefore less surface area for software bugs. AFIK, Pytorch internally is able to see what are the variables that have been worked upon and is able to find the required grads, without us explicitly asking to calculate the grads for each of the branches.</p>

username: moad_alami
cooked: <p>Hi <a class="mention" href="/u/ganlover">@ganLover</a> is it possible to share the code used to implement the B-CNN architecture?</p>
Paralelize on Google Colab with ProcessPoolExecutor

username: maria.solyanik
cooked: <p>I am trying to parallelize my code on Google Colab like this:</p>
<pre><code>    from concurrent.futures import ProcessPoolExecutor
    from time import sleep, time
    from torch.autograd import Variable

    def f(x):
        chromosome_init = np.random.rand(30000)
        return single_run(images, labels, num_epochs, chromosome_init, fitness)

    L = list(range(1))

    if __name__ == '__main__':
        
        begin = time()
        with ProcessPoolExecutor(max_workers=1) as pool:

            # result = sum(pool.map(f, L))
            result = np.sum(np.fromiter(pool.map(f, L), float))
        end = time()
        
        print(f"result = {result} and time = {end-begin}")
</code></pre>
<p>The function single_run is a training function and right now I simplified it to this:</p>
<pre><code> def single_run(images, labels, num_epochs, chromosome, fitness):

  with torch.no_grad():
    for layer in cnn.state_dict():
      cnn.state_dict()[layer].data.fill_(2)
return 1
</code></pre>
<p>When I run the code in Colab I get this error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64b8950da30d4482dbe161bd943a11668e9fc7bb.png" data-download-href="https://discuss.pytorch.org/uploads/default/64b8950da30d4482dbe161bd943a11668e9fc7bb" title="Screen Shot 2022-05-09 at 4.13.20 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64b8950da30d4482dbe161bd943a11668e9fc7bb_2_690x45.png" alt="Screen Shot 2022-05-09 at 4.13.20 PM" data-base62-sha1="en1dbuVLudycfiz6gUQnVIvOQKD" width="690" height="45" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64b8950da30d4482dbe161bd943a11668e9fc7bb_2_690x45.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64b8950da30d4482dbe161bd943a11668e9fc7bb_2_1035x67.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64b8950da30d4482dbe161bd943a11668e9fc7bb_2_1380x90.png 2x" data-dominant-color="F1F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-09 at 4.13.20 PM</span><span class="informations">2142×142 32.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have identified that the error shows up when I assign weights as “cnn.state_dict()[layer].data.fill_(2)”. I have tried to reset CUDA calls. Any ideas how to debug/cure it are appreciated!</p>
<p>Thanks in advance!</p>
Failed to convert torchscript model use torch.jit.script

username: mertonmeng
cooked: <p>Hi Community,</p>
<p>I have came across a issue that throw me an error when convert model:<br>
AssertionError: Unsupported annotation typing.Union[torch.FloatTensor, NoneType] could not be resolved because &lt;class ‘torch.FloatTensor’&gt; could not be resolved.</p>
<p>This looks to me that the script does not import torch package correctly, but I checked all the code I see torch is imported correctly</p>
<p>system                     ubuntu-18.04<br>
torch                         1.8.0<br>
torchvision                0.9.0</p>
<p>Anyone come across similar issue?</p>
<p>Thanks</p>
Prefetch_factor and num_workers

username: Andre_Amaral_IST
cooked: <p>Hey,</p>
<p>In the dataloader what means “prefetch_factor” and how it will affect the batch size and the way the data is loaded?</p>
<p>What is the best approach to choosing the num_workers?</p>
<p>This question came up because when I initialize my lstm hidden and cell states the parameters are (num_layers, batch_size, hidden_size), and if I set a batch size of 10000 (for example) and when i go to initialize the hidden and cell state in the init method it gives me dimensions error saying that the batch_size parameter should be half of the batch size i established (5000 in this example)</p>
<p>Regards!<br>
André</p>

username: nivek
cooked: <p>From the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">documentation</a>, <code>prefetch_factor</code> defines the number of samples loaded in advance by each worker. <code>2</code> means there will be a total of <code>2 * num_workers</code> batches prefetched across all workers. (default: <code>2</code> ).</p>
<p>It should not affect your <code>batch_size</code>.</p>
<p>As for choosing the value for <code>num_workers</code>, it varies depending on your setup. You should see better performance when increasing from a small value to a slightly larger one. At the same time, it will increase your memory usage due to overhead and you will need sufficient memory for that. At some point, a larger <code>num_workers</code> value will not boost your performance anymore and you should use a value around that point.</p>

username: Andre_Amaral_IST
cooked: <p>Thanks for your answer.<br>
I am using 8 workers, I saw on the web people saying that a good way to choose the num_workers is<br>
4 * number of gpus</p>
Train a generator: trying to backward through the graph a second time

username: imxtx
cooked: <p>I want to train a generator to simulate the distribution of the output of a ResNet18. But pytorch gives an error <strong>in the second iteration</strong> of the for loop:</p>
<pre><code class="lang-nohighlight">RuntimeError: Trying to backward through the graph a second time (or directly access saved 
tensors after they have already been freed). Saved intermediate values of the graph are freed
 when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to 
backward through the graph a second time or if you need to access saved tensors after 
calling backward.
</code></pre>
<p>Here is my code for training:</p>
<pre><code class="lang-python">def noise(batch_size):
    """
    Generates a 1-d vector of gaussian sampled random values
    """
    return torch.randn(batch_size, 100, requires_grad=True)

def noise_loss(self, noise_data):
    n0 = noise_data[0].reshape((1, -1))
    n1 = noise_data[1].reshape((1, -1))
    n_loss0 = torch.linalg.norm(n0)
    n_loss1 = torch.linalg.norm(n1)
    return n_loss0 + n_loss1

def verification_loss(self, fake_pred):
    # target_feature is computed by ArcFace with image input and saved.
    v_loss0 = F.cosine_similarity(fake_pred[0], self.target_feature[0], dim=0)
    v_loss1 = F.cosine_similarity(fake_pred[1], self.target_feature[1], dim=0)
    return v_loss0 + v_loss1

def optimize(self, num_epochs):
    # freeze ArcFace
    self.predictor.eval()

    batch_size = 2
    # start to train the generator
    for epoch in range(num_epochs):
        # generate batch of noises
        noise_data = self.generator(noise(batch_size).to(self.device))
        fake_data = noise_data * self.imposter_mask + self.imposter_image

        # get feature from ArcFace
        fake_pred = self.predictor(fake_data)

        # noise loss
        n_loss = self.noise_loss(noise_data)
        # verification loss
        v_loss = self.verification_loss(fake_pred)

        g_error = n_loss + v_loss

        # Update the generator
        self.g_optimizer.zero_grad()
        g_error.backward()
        self.g_optimizer.step()
</code></pre>

username: imxtx
cooked: <p>I figured out. I didn’t detach the target_feature from the computation graph. But I still don’t know why do I need to detach it.</p>
<pre><code class="lang-auto"># load data
self.target_image = torch.tensor(load_image("data/train/target.jpg")).to(self.device)
# target feature, why need to detach?
self.target_feature = self.predictor(self.target_image).detach()
</code></pre>
RuntimeError: The size of tensor a (28) must match the size of tensor b (76) at non-singleton dimension 3 for YOLOv4

username: imran
cooked: <p>Hi, I changed the image size in Cfg file of the YOLOv4 <a href="https://github.com/Tianxiaomo/pytorch-YOLOv4" rel="noopener nofollow ugc">this repo</a> to</p>
<pre><code class="lang-auto">Cfg.width = 224
Cfg.height = 224
</code></pre>
<p>I got the below error. I am not able to resolve it. Can you help me to find out the problem.</p>
<pre><code class="lang-auto">Epoch 1/3:   0%|           | 0/2 [00:03&lt;?, ?img/s]
Traceback (most recent call last):
  File "train.py", line 634, in &lt;module&gt;
    train(model=model,
  File "train.py", line 383, in train
    loss, loss_xy, loss_wh, loss_obj, loss_cls, loss_l2 = criterion(bboxes_pred, bboxes)
  File "C:\ProgramData\Anaconda3\envs\yolov4t\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "train.py", line 249, in forward
    pred[..., 0] += self.grid_x[output_id]
RuntimeError: The size of tensor a (28) must match the size of tensor b (76) at non-singleton dimension 3
</code></pre>
<p><a href="https://github.com/Tianxiaomo/pytorch-YOLOv4/issues/523" rel="noopener nofollow ugc">I raised an issue for this repot but no one is answering</a>.</p>
How to concatenate tensors without copy?

username: dbsx
cooked: <p>I am using torchlib for reinforcement learning. One thing frequently doing is to sample a batch of data from replay buffer, in which a large amount of tensor data stored on gpu, and concatenate them as a whole big tensor as input for model training.</p>
<p>The action “sample” generally, would copy sampled data. If it is just used as reference, then:</p>
<ol>
<li>training would be more efficient, since there is no copy;<br>
2.low probability of  “out of memory”, since gpu memory is fixed (or pre allocated) as replay buffer size is fixed.</li>
</ol>
<p>I wonder whether there is a way to do so. If not, would this be a good feature to post?</p>
Docker vs source for inferencing at the edge in real time application

username: Miguel_Campos
cooked: <p>Hello!</p>
<p>I have a model that I need to run in real time scenarios using a Jetson. In my computer it runs very fast with JIT and now is time to move on a Jetson for production.</p>
<p>I wonder if running this model through using a Docker Image will be as fast as running it from source, installing all the libs from scratch.</p>
<p>Thank you!</p>

username: tom
cooked: <p>I think the speed difference is small enough for the main question to be which gets you better support / a better workflow for your life-cycle, so likely you want to go with what NVidia recommends unless you have some other support arrangement.</p>
<p>I’m saying this as someone who always builds his own PyTorch from source and does offer support for bespoke installations on a commercial basis.</p>
<p>Best regards</p>
<p>Thomas</p>
TensorFloat MatMul speed up issue in 3090

username: Tanmoy
cooked: <p>I am trying to run TensorFloat operations on 3090 to test the speed up using these code snippet. But in TF32 enabled execution time is more than TF32 disabled mode. The first matmul takes <code>5.173683166503906e-05</code> in TF32 disabled, whether in TF32 enabled its taking <code>0.0006937980651855469</code>. Am I doing anything wrong ?</p>
<p>System spec:<br>
GPU - RTX 3090FE,<br>
torch -  1.11.0+cu113,<br>
cuda-11.3</p>
<pre data-code-wrap="python3"><code class="lang-nohighlight">a_full = torch.randn(10240, 10240, dtype=torch.double, device='cuda')
b_full = torch.randn(10240, 10240, dtype=torch.double, device='cuda')
ab_full = a_full @ b_full
mean = ab_full.abs().mean()  

a = a_full.float()
b = b_full.float()

# Do matmul at TF32 mode.
ab_tf32 = a @ b  
error = (ab_tf32 - ab_full).abs().max()  
relative_error = error / mean 

# Do matmul with TF32 disabled.
torch.backends.cuda.matmul.allow_tf32 = False
ab_fp32 = a @ b 
error = (ab_fp32 - ab_full).abs().max() 
relative_error = error / mean 
</code></pre>
<p>Also one more follow up question, is there any way to detect ampere vs non ampere architecture in Pytorch? Or anything similar(not taking about <code>torch.cuda.get_device_name()</code>?</p>

username: tom
cooked: <p>How do you measure that?<br>
I’m asking because it is easy to forget to sync the GPU before taking (the start and end times) when doing manual timing. With the autograd profiler, you should be OK with the measurements by default.</p>
<p>Best regards</p>
<p>Thomas</p>
Higher-Dimensional Batch Norm

username: rambo
cooked: <p>Hi,</p>
<p>I have a batch of k-dimensional data, meaning I have tensors of size <code>(batch_size, n1, n2, ..., nk)</code>.<br>
I would like to normalize the data after the batch axis using a batch norm module, meaning I would like to learn the k-dimensional mean of my data, subtract it from my tensors along the batch axis, and learn the centered L2-norm of my data, and scale my tensors just like a batch norm does.<br>
It would be nice to not need to flatten my data and then reshape it after the batch axis all the time.</p>
<p>Is there a pytorch functional class that implements this functionality such that I can implement an appropriate batch norm module myself?<br>
It would be nice to just have a single batch norm module and pass it an axis (axes) along which to operate, instead of having the current zoo of batch norms…</p>
<p>Thanks!</p>

username: Andrei_Cristea
cooked: <p>Any chance <a href="https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html#torch.nn.SyncBatchNorm" rel="noopener nofollow ugc">SyncBatchNorm</a> could help you? Note that one is for tensors of size <code>(batch_size, channels, n1, n2, ...)</code> so it has the extra <code>channels</code> dimension vs your example, but you could just unsqueeze a length-1 dimension in there, and it would work for your use case.</p>
<p>Though I’ve never used it, reading through the description seems promising.</p>

username: rambo
cooked: <p>Thanks, this looks promising!</p>
My Pytorch Reinforcement learning AI doesn't react to reward

username: PringleOrange
cooked: <p>Hello. I have this python pytorch reinforcement learning AI. It’s goal is to choose any random character from a-z and if it chooses ‘d’ it gets rewarded. it isnt rewarded or punsihed for anything else. it currently just chooses a random letter from a-z and constantly chooses that, the only time it isnt sticking with the same character is when its epsilon value forces it to choose one at random but no matter how many times it stumbles on ‘d’ it doesn’t react to the reward it was given. Below, there is a link attatched which takes you to the code which calculates what move it does based on its reward and gamma and such. Is there anything here which may be the reason why the bot won’t choose the rewarding character?</p>
<p><a href="https://paste.pythondiscord.com/hekedehepe" class="onebox" target="_blank" rel="noopener nofollow ugc">https://paste.pythondiscord.com/hekedehepe</a></p>

username: InnovArul
cooked: <pre><code class="lang-auto">target = pred.clone()
for idx in range(len(d  one)):
       Q_new = reward[idx]
       if not done[idx]:
           Q_new = reward[idx] + self.gamma * torch.max(self.model(next_state[idx]))
  
       target[idx][torch.argmax(action[idx]).item()] = Q_new
    
</code></pre>
<p>When calculating the target, I guess you have to use the model with <code>eval()</code> mode (if there is <code>batchnorm</code> involved) and under <code>torch.no_grad()</code>. Also, while cloning <code>pred</code> to <code>target</code> variable, you may have to detach it?</p>
<pre><code class="lang-auto">target = pred.detach().clone()
with torch.no_grad():
     for idx in range(len(done)):
         Q_new = reward[idx]
         if not done[idx]:
             Q_new = reward[idx] + self.gamma * torch.max(self.model(next_state[idx]))

         target[idx][torch.argmax(action[idx]).item()] = Q_new
    
</code></pre>

username: PringleOrange
cooked: <p>Hello. Thank you for your response although that didn’t seem to affect the output. I’ve attatched the other 2 files which use model.py and have all of the logic for what it outputs incase that’s what is causing the error.<br>
<a href="https://paste.pythondiscord.com/rowotaboho" class="onebox" target="_blank" rel="noopener nofollow ugc">https://paste.pythondiscord.com/rowotaboho</a><br>
<a href="https://paste.pythondiscord.com/pujewikiyi" class="onebox" target="_blank" rel="noopener nofollow ugc">https://paste.pythondiscord.com/pujewikiyi</a></p>
A computational graph with a common node in PyTorch is it wasting memory?

username: ali_hejrati
cooked: <p>This image shows my computational graph(Something like a DCGAN). And I first call backward on the last intermediate node of G1 with retain_graph=true. And then call backward on the last intermediate node of G2 with retain_graph=false.</p>
<p>My question is graph G1 still in memory?</p>
<p>I want to train my network in N epochs. How many G1 graphs are still in memory?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fe55cf8c61862c92a07e9f9f1f29787ab906a09.png" data-download-href="https://discuss.pytorch.org/uploads/default/8fe55cf8c61862c92a07e9f9f1f29787ab906a09" title="IMG_20220508_153029_648"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fe55cf8c61862c92a07e9f9f1f29787ab906a09_2_689x331.png" alt="IMG_20220508_153029_648" data-base62-sha1="kwXGzipt5pfjt1696lbPf2hnU3n" width="689" height="331" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fe55cf8c61862c92a07e9f9f1f29787ab906a09_2_689x331.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fe55cf8c61862c92a07e9f9f1f29787ab906a09_2_1033x496.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fe55cf8c61862c92a07e9f9f1f29787ab906a09.png 2x" data-dominant-color="F8F5EB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20220508_153029_648</span><span class="informations">1231×592 96.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: tom
cooked: <p>When the values all go out of scope and are collected, the autograd graph should be released, too.<br>
That said: If you are concerned about it and you cannot do a joint backward, you could manually split the graph (with <code>intermediate_new = intermediate.detach().requires_grad()</code>, use that in your computation and then do the first backward and do <code>intermediate.backward(intermediate_new.grad, retain_graph=True)</code>.</p>
<p>Best regards</p>
<p>Thomas</p>