akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is it possible to load a model which is stored with model.module.state_dict() but load with model.state_dict()

JZArray: <p>Hello guys,</p>
<p>I want to ask a question, I have trained a model with two gpus and stored this model with model.module.state_dict(), now I want to load this model in one gpu, can I directly load this trained model with model.state_dict()?</p>
<p>Thanks in advance!</p>

ptrblck: <p>Yes, you should be able to load the <code>state_dict</code> via <code>model.load_state_dict()</code>.<br>
Are you seeing any issues using this approach?</p>
Some difference between pytorch version 1.0.0 and pytorch version 1.7.1

IricsDo: <p>I have two code with difference pytorch version<br>
In torch 1.0.0<br>
torch.randn(4, 4).view(-1)&gt;0. The result of this is tensor([0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1], dtype=torch.uint8)<br>
In torch 1.7.1<br>
torch.randn(4, 4).view(-1)&gt;0. The result of this is tensor([ True, True, False, False, False, True, False, True, False, True, True, True, False, True, False, False])<br>
I don’t know what is happen?<br>
Note: This is no problem for me, i can solve with key word “convert true flase to 0 1 python”. But i want to know how to torch work.<br>
Thanks for reading.</p>

albanD: <p>Hi,</p>
<p>This is expected yes.<br>
Comparison operations now return the new boolean dtype instead of the old uint8 dtype.<br>
This should not be a problem as both can be used for masking and regular ops. But the using uint8 as a boolean is deprecated an will be removed in the future.</p>

IricsDo: <p>Thanks your answer. It so great!!</p>
Adjust contrast in the transformation pipeline

beto16: <p>Hi I’m trying to modify contrast in the transformation pipeline for data augmentation but I have this error</p>
<p><strong>TypeError: adjust_contrast() missing 1 required positional argument: 'img’</strong></p>
<p>This is my code</p>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.Grayscale(num_output_channels=1),
    transforms.RandomRotation(degrees= (-15, 15)),
    transforms.ToTensor(),
    transforms.functional.adjust_contrast( contrast_factor= 0.5)
    ])
</code></pre>

eqy: <p>The issue is that the functional version of contrast expects to take in the input directly rather than returning a callable function. You can work around this by using <a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ColorJitter" rel="noopener nofollow ugc">ColorJitter</a> and only changing the contrast (e.g., <code>torchvision.transforms.ColorJitter(brightness=0, contrast=0.5, saturation=0, hue=0)</code>).</p>
<p>If you want a constant adjustment you can just wrap the current function:</p>
<pre><code class="lang-auto">def my_adjust_contrast():
    def _func(img):
        return  transforms.functional.adjust_contrast(img, contrast_factor= 0.5)
    return _func
transform = transforms.Compose([
    transforms.Grayscale(num_output_channels=1),
    transforms.RandomRotation(degrees= (-15, 15)),
    transforms.ToTensor(),
    my_adjust_contrast()
    ])
</code></pre>
Softmax activation function

gaza_palestine: <p>Hi, I want to know how Soft max() function works, we get probabilities vector length equal to number of classes but how it knows 4 index is for coat in fashion Mnist and 5 for sandle.<br>
My question is how we define which column is for which label can we change it.?</p>

ptrblck: <p>Neither the <code>softmax</code> method nor the model “knows” anything about the label.<br>
You, as the researcher, create the dataset and create the input-output mapping, which the model tries to learn. This would also mean that you are free to remap any labels, as long as it’s consistent for all samples in the dataset.<br>
The predictions of the model would also give you the logits (or probabilities assuming you apply <code>softmax</code> on them), which can then be mapped to the class index via <code>torch.argmax</code>. The interpretation of this class index to a class name also defined by the input-output mapping.</p>
Indexing documentation

sh0416: <p>Hi,</p>
<p>Is there any comprehensive guideline for indexing tensor?<br>
I’m trying to find the pytorch version of this <a href="https://numpy.org/doc/stable/user/basics.indexing.html" rel="noopener nofollow ugc">link</a>.<br>
Or, is the indexing supported by numpy also supported in pytorch?</p>
<p>Thanks,</p>

ptrblck: <p>I’m unsure about the relative parity between numpy’s and PyTorch’s indexing, but I’m usually also referring to the numpy docs in necessary and wasn’t running into an unsupported use case so far.</p>
Manually adding biases to convolution gives wrong results

andrea_bragagnolo: <p>Hi, given a Conv2d module I’d like to manually perform the biases addition, i.e. given a module <code>M</code> with weights <code>w</code> and biases <code>b</code>, I can compute the output <code>y</code> given and input <code>x</code> as <code>y = x * w + b</code>.</p>
<p>What I’d like to do is to evaluate <code>x * w</code> using the module’s  forward method <code>M(x)</code> and then manually add the biases <code>b</code>. Unfortunately I cannot just override the module’s biases since the one I want to add might be N-D tensors while PyTorch expects only 1-D tensors.</p>
<p>The problem I’m facing is that the results I obtain with the manual addition are different from the results obtained using the full module’s forward method when working with float32 tensor but the same results when working in float64 format. Is there any way this manual addition of the biases can give the same results as the forward method working with float32? Is the underlying convolution code operating differently from a python-side <code>module(x) + bias</code>?</p>
<p>Here how to reproduce the issue:</p>
<pre><code class="lang-auto">def test_conv_manual_bias_float32(self):
    module = nn.Conv2d(3, 64, 3, padding=1)
    x = torch.randn((64, 3, 128, 128))
    
    y_src = module(x)
    
    bias = module.bias.data.clone()
    module.bias.data.mul_(0)
    y_prop = module(x) + bias[:, None, None]
    
    print(torch.allclose(y_src, y_prop))


def test_conv_manual_bias_float64(self):
    module = nn.Conv2d(3, 64, 3, padding=1).double()
    x = torch.randn((64, 3, 128, 128)).double()
    
    y_src = module(x)
    
    bias = module.bias.data.clone()
    module.bias.data.mul_(0)
    y_prop = module(x) + bias[:, None, None]
    
    print(torch.equal(y_src, y_prop))
</code></pre>
<p>The difference can also be tested ussing <code>torch.max(y_src.abs() - y_prop.abs())</code>, that returns a value grater than <code>0</code> for the first function and <code>0</code> for the second.</p>

ptrblck: <p>The max. absolute error for the FP32 implementation is <code>tensor(8.3447e-07, grad_fn=&lt;MaxBackward1&gt;)</code> on my system, which would be explained by the limited floating point precision.<br>
I don’t know which path is taken for FP64 on the CPU, but assume that less or no optimizations are applied internally, which could explain the 0 difference.</p>

andrea_bragagnolo: <p>Is it possible that the C implementation of the convolution bias differs from a simple addition in python? Because I’m simply adding the same exact tensor. This problem doens’t seem to exist when using Linear layers.</p>

ptrblck: <p>Yes, the internal conv implementation (using <code>MKL</code>?) could use another approach than the native addition used in PyTorch, which could thus yield these small numerical mismatches.</p>
Embedding Layer

TheOraware: <p>I just started NN few months ago , now playing with data using Pytorch. I learnt how we use embedding for high cardinal data and reduce it to low dimensions. There is one thumb of role i saw that for reducing high dimensional categorical data in the form of embedding you use following formula</p>
<pre><code class="lang-auto">embedding_sizes = [(n_categories, min(50, (n_categories+1)//2)) for _,n_categories in embedded_cols.items()]
embedding_sizes
[(69, 35), (11, 6)]
</code></pre>
<p>After creating this embedding layer how do we know these embedding layers are appropriate for MLP ? Do we check score with different size of embedding layer or do we visualise this layer? If we visualise then what are the ways to visualise , in short how can i validate that my embedding layers are good in terms of its reduced size from 69 to 35 and 11 to 6?</p>

Kushaj: <p>The embedding layer is just a look up table. So you pass an index and an embedding vector is returned. When you initialize the embedding layer, these are just random values. After training the embeddings, you can try the following to check the quality of the embeddings</p>
<ul>
<li>Check the metric. As everything kept same, the metric value of 69 dim and 35 dim embedding can give you some idea of the quality of embeddings</li>
<li>You can use PCA to visualize the embeddings in 2D space. Although this is not a good approach.</li>
</ul>

TheOraware: <p><a class="mention" href="/u/kushaj">@Kushaj</a> thanks Kushaj , i think i need to extract those layers after training and then extract those to visualize with target variable , how can i extract these embeded layers weights after trainig?</p>

Kushaj: <p>Use <code>layer.weight</code> to get the embedding matrix. In general whenever you want to extract something from any layer in pytorch just look up at the <code>__init__</code> function in the source code.</p>
2nd order differential for PINN model

sunny1: <p>X is [n,2] matric which compose x and t. I am using Pytorch to compute differential of u(x,t) wrt to X to get du/dt and du/dx and du/dxx. Here is my piece of code</p>
<p>X.requires_grad = True<br>
p = mlp(X)<br>
grads, = torch.autograd.grad(p, X, grad_outputs=p.data.new(p.shape).fill_(1),create_graph=True, only_inputs=True)<br>
grads1, = torch.autograd.grad(grads, X,  grad_outputs=grads.data.new(grads.shape).fill_(1), create_graph=True, only_inputs=True)<br>
dpdx, dpdt = grads[:,0], grads[:,1]<br>
dpdxx = grads1[:,0]</p>
<p>I am not getting satisfactory result. I think something is wrong when i am calculating 2 differential. can anyone help me?</p>

Yanis.Zatout: <p>Hello there !</p>
<p>I see you’re using an MLP, depending on the activation functions you have, if you use 2nd order derivatives for ReLU activations, you should have all zeros, be careful with that. Tanh or sigmoid activation should do the trick.</p>
<p>In addition, if you want to differentiate your neural net wrt t or x, I advise you to be a bit more careful with how you write your derivatives.</p>
<p>Here’s a little example:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.l1=nn.Linear(2,80)
        self.mi=nn.Linear(80,80)
        self.mi1=nn.Linear(80,80)
        self.mi2=nn.Linear(80,40)
        self.ol=nn.Linear(40,1)
        self.th=nn.Tanh()
            
    def forward(self,x,t):
        u=torch.cat((x,t),1)
        u=self.th(self.l1(u))
        u=self.th(self.mi(u))
        u=self.th(self.mi1(u))
        u=self.th(self.mi2(u))
        u=self.th(self.ol(u))
        
        return u
def f(x,t):
    u = net(x,t)
    u_t = grad(u, t, 
                     create_graph=True,
                     grad_outputs=torch.ones_like(u), 
                     allow_unused=True)[0]
    u_x = grad(u, x, 
                     create_graph=True,
                     grad_outputs=torch.ones_like(u), 
                     allow_unused=True)[0]
    u_xx = grad(u_x, x, 
                     create_graph=True,
                     grad_outputs=torch.ones_like(u), 
                     allow_unused=True)[0]
    w = torch.tensor(0.01/np.pi)
    f = u_t + u*u_x - w*u_xx
    return f
</code></pre>
<p>Here’s an example for Burgers’ equation. I got the original code from <a href="https://github.com/dbgannon/NNets-and-Diffeqns/blob/master/burgernet.py" class="inline-onebox" rel="noopener nofollow ugc">NNets-and-Diffeqns/burgernet.py at master · dbgannon/NNets-and-Diffeqns · GitHub</a><br>
It’s explained here: <a href="https://cloud4scieng.org/2020/06/10/notes-on-deep-learning-and-differential-equations/" class="inline-onebox" rel="noopener nofollow ugc">Notes on Deep Learning and Differential Equations. – Cloud Computing For Science and Engineering</a></p>
<p>Hope it helps !</p>
Convert tensor to numpy is very slow

Amr_Elsersy: <p>I am trying to convert a tensor to numpy array using numpy() function</p>
<p>it is very slow ( takes 50 ms !)</p>
<pre><code class="lang-auto">semantic = semantic.cpu().numpy()
</code></pre>
<p>semantic is a tensor of size “torch.Size([512, 1024])”  and it’s device is cuda:0</p>

albanD: <p>Hi,</p>
<p>I think the slow part is the <code>.cpu()</code> here, not the <code>.numpy()</code>.<br>
Sending the Tensor to the CPU requires to sync with the GPU (if there are outstanding computations, that will be extra slow, make sure to torch.cuda.synchronize() before timing) and copy the memory to ram.</p>
<p>The  numpy conversion itself should be very fast though as no memory copy occurs.</p>

Amr_Elsersy: <p>it is solved ! thanks <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
can you explain what did the synchronization do ?</p>

albanD: <p>The short story is that most of the cuda API is asynchronous and so it will just queue up the work to be done and return immediately. Only when you try to access the value (by asking to copy it on cpu here) will it actually wait for the computations to be done.<br>
the synchronize force this waiting to happen and so is very convenient when measuring CPU time to know if a slow call is because of outstanding ops still running asynchronously on the GPU or the call itself.</p>
The behavior of forward hook

phantom90: <p>Hi there,</p>
<p>Suppose we are using a forward hook to analyze a mid layer:</p>
<pre><code class="lang-auto">class Hook():
    def __init__(self,m):
        self.hook = m.register_forward_hook(self.hook_func)
    def hook_func(self,m,i,o):
        self.stored = o.detach().clone()
    def __enter__(self, *args): return self
    def __exit__(self, *args):
        self.hook.remove()
</code></pre>
<p>for instance layer4 in resnet. Once we get the feature map feature_map.features, can we further use this to calculate the gradient of <code>torch.norm(feature_map.features)</code> w.r.t model parameters?</p>
<p>Will the computation graph be retained? What is the correct way to accomplish this task?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>If you use <code>.detach()</code> in the hook, then no the graph will not be kept.<br>
But if you don’t do that, then yes, the Tensors will require gradients just like the ones in the forward and you can do whatever you want there <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

phantom90: <p>Thank you. I will remove the .detach() method.</p>
Batching `torch.tensordot` using `torch.vmap`

rambo: <p>Hi,</p>
<p>I want to implement batched linear combination of batches <code>bs</code> of tensors <code>b</code> of arbitrary shape with batches <code>as</code> of coefficients <code>a</code>. This means <code>bs</code> has size <code>[batch_size, n0, n1, ..., nk ]</code> and <code>as</code> has size <code>[batch_size, n0]</code>. The output is of size <code>[batch_size, n1, ..., nk]</code><br>
The following does what I want:</p>
<pre><code class="lang-auto">def cust_td(a,b):
   return torch.tensordot(a, b, dims=1)

batch_td = torch.vmap(cust_td)
batch_td(as, bs)
</code></pre>
<p>Is this fine (from an efficiency perspective?).</p>
<p>On another note this question maybe motivates allowing <code>torch.vmap</code> to get keyword args (e.g. <code>dims=1</code> in my case) and pass them to the function <code>func</code> before vectorizing it.</p>
Select By Index

Hieu_Phan: <p>Hi,<br>
I have segmentation maps with size = [5,64,64] where batch size is 5, the spatial size is 64x64.<br>
Given a feature map F with the same size 64x64 and N channels.</p>
<p>Let select locations in the segmentation map whose labels are <em>v</em>. Now I want to select features in F based on the selected locations whose labels are <em>v</em> in the segmentation map.</p>
<p>Here’s how it looks like:</p>
<pre><code class="lang-auto">seg = torch.randint(0,5,size=(5,64,64))

F = torch.rand(5,25,64,64)

locations = seg==1
F[locations]??
</code></pre>
<p>Examples:</p>
<pre><code class="lang-auto">seg with batch_size=2, spatial_size=2x2=4
seg = [[[0,1,2,3],[1,2,1,2]
F with 3 channels, the size is (2,3,4)
F = [[2,3,4,3],
     [4,1,2,4],
     [4,0,5,1]],

     [1,2,0,4],
     [0,1,10,3],
     [1,2,0,5]],
location = seg == 1

output = F[location]= [[3,1,0],[1,0,1],[0,10,0]]
</code></pre>

the-dharma-bum: <p>Your example isn’t coherent with your explanation above. Given your explanation, seg and F size should be respectively <code>(2, 2, 2)</code> and <code>(2, 3, 2, 2)</code>:</p>
<pre><code class="lang-python">seg = [[[0, 1],
        [2, 3]],
       [[1, 2],
        [1, 2]]]

mask = np.array(seg)

print(mask.shape)  # (2, 2, 2)

F = [[[[2, 3],
       [4, 3]],
      [[4, 1],
       [2, 4]],
      [[4, 0],
       [5, 1]]],
     [[[1, 2],
       [0, 4]],
      [[0, 1],
       [10, 3]],
      [[1, 2],
       [0, 5]]]]

features = np.array(F)

print(features.shape)  # (2, 3, 2, 2)

output = features[mask == 1] # yields an error because mask and features haven't the same dimension

</code></pre>
<p>But in that case I don’t understand what you want to do. If you have a batch of segmentation masks, and a batch of features maps, don’t you want to treat each sample from the batch independantly ? Based on your example, it seems you want the positive indices across all the batch, which is implicitely a fusion of the batched segmentation masks. Is it really what you want to do ?</p>
<p>For instance let’s forget this batch issue and assume you have one segmentation mask and one set of features:</p>
<pre><code class="lang-python">mask     = torch.randint(0, 5, size=(64, 64))
features = torch.rand(25, 64, 64)
</code></pre>
<p>You can then use <a href="https://pytorch.org/docs/stable/generated/torch.masked_select.html" rel="noopener nofollow ugc">torch.masked_select</a>:</p>
<pre><code class="lang-python">class_mask = torch.where(mask == 1, True, False)
output = features.masked_select(class_mask)
</code></pre>

Hieu_Phan: <p>Hi, that is exactly what I’m looking for. Thank you!</p>
GAN doesn't train

paozaf: <p>Hi all,<br>
I wrote this GAN (in the example it uses fake input):</p><aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/pzaffino/7c3714ffe8eb867eb45b721ac4d2d808" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/pzaffino/7c3714ffe8eb867eb45b721ac4d2d808" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/pzaffino/7c3714ffe8eb867eb45b721ac4d2d808</a></h4>
<h5>pytorch_error.py</h5>
<pre><code class="Python">import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.models
import torch.nn.functional as F
 
class Generator(nn.Module):
    def __init__(self, initial_features, num_channels=1, dropout=0.1):
        super(Generator, self).__init__()
 </code></pre>
This file has been truncated. <a href="https://gist.github.com/pzaffino/7c3714ffe8eb867eb45b721ac4d2d808" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Basically, the generative accuracy is quantified in terms of image difference.</p>
<p>The generator net and the discriminator net get trained (individually), but it looks like they don’t “communicate”. I mean, the discriminator judgment doesn’t affect the generative training…the training goes parallel, without affecting each other.</p>
<p>Any idea? Am I missing something?<br>
I multiply the discriminator score by a float factor in order to make it comparable to the image difference.</p>
<p>Thank you in advance.</p>
<p>Best,<br>
Paolo</p>

paozaf: <p>No idea?<br>
Thanks in advance</p>
Gradient flow and device

cbd: <p>I used “grad1 = grad.clone()” in my hook function. Other tensors are on GPU but it shows “grad1” is on CPU as “grad.clone()” is returning value in CPU. Even when function return the value, it gives error if return tensor is on GPU so need to convert in CPU again.</p>
<p>Is gradient normal flow on CPU? What can i do to convert all grad on GPU so that don’t require to switch from GPU and CPU.</p>

ptrblck: <p>I don’t know how <code>grad</code> and <code>grad1</code> are calculated, but PyTorch will stick the the specified devices in your script and won’t change the device behind your back.<br>
Of course this doesn’t apply to any 3rd party utilities, which explicitly could push data to the CPU to save GPU memory, but I also don’t know if you are using any of these.</p>
<p>Could you describe your use case a bit and post a minimal code snippet, which would reproduce this issue?</p>

cbd: <p>Sorry it was my mistake. Tensor was on CPU.</p>
Converting own custom dataset into MNIST numpy form

batman: <p>Does anyone have an idea how can I convert my own custom dataset in the form of an MNIST dataset for the below code? I am stuck at this point like MNIST has its own <code>train.target.numpy</code>,<code>test.targets.numpy</code>. How should I do it for my own custom dataset.</p>
<p>I have around 16000 images that are in .png form, so converting all of them into NumPy array will result in out of memory or the system is killing my script.</p>
<pre><code class="lang-auto">batchsize=128
train = dsets.MNIST(root='../data/',train=True,download=True)
test=dsets.MNIST(root='../data/',train=False,transform=transforms.Compose([transforms.ToTensor(),]))
indices= np.random.choice(len(train.targets.numpy()), 2000, replace=False)
indices_test= np.random.choice(len(test.targets.numpy()), 100, replace=False)
</code></pre>
<p>Any suggestion are welcome?</p>
How to match two ImageFolder?

phantom90: <p>Hi there,</p>
<p>Please consider the scenario:</p>
<p>We have the first ImageFolder for training set:</p>
<p><code>./train/&lt;CLASSES&gt;/*.jpg</code></p>
<p>We also have another ImageFolder:</p>
<p><code>./tr/&lt;CLASSES&gt;/*.jpg</code></p>
<p>The classes of these two ImageFolder should be identical. However, some folders in <code>./tr/&lt;CLASSES&gt;/*.jpg</code> are empty. Then how to match the class index for these two ImageFolder.</p>
<p>Thanks~</p>

eqy: <p>If the class folder names of both <code>./train/</code> and <code>/.tr/</code> are the same then the indices for each class should be identical by default. This is because the class names are sorted before the class to index map is built:</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/b2d4d8a24e221b3d084184fee89a8e1f51e1bcaf/torchvision/datasets/folder.py#L40" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/b2d4d8a24e221b3d084184fee89a8e1f51e1bcaf/torchvision/datasets/folder.py#L40" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/b2d4d8a24e221b3d084184fee89a8e1f51e1bcaf/torchvision/datasets/folder.py#L40</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="30" style="counter-reset: li-counter 29 ;">
<li>        bool: True if the filename ends with a known image extension</li>
<li>    """</li>
<li>    return has_file_allowed_extension(filename, IMG_EXTENSIONS)</li>
<li>
</li>
<li>
</li>
<li>def find_classes(directory: str) -&gt; Tuple[List[str], Dict[str, int]]:</li>
<li>    """Finds the class folders in a dataset.</li>
<li>
</li>
<li>    See :class:`DatasetFolder` for details.</li>
<li>    """</li>
<li class="selected">    classes = sorted(entry.name for entry in os.scandir(directory) if entry.is_dir())</li>
<li>    if not classes:</li>
<li>        raise FileNotFoundError(f"Couldn't find any class folder in {directory}.")</li>
<li>
</li>
<li>    class_to_idx = {cls_name: i for i, cls_name in enumerate(classes)}</li>
<li>    return classes, class_to_idx</li>
<li>
</li>
<li>
</li>
<li>def make_dataset(</li>
<li>    directory: str,</li>
<li>    class_to_idx: Optional[Dict[str, int]] = None,</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How multiply a 4-dimensional sparse tensor to a dense tensor?

Boltzmachine: <p>suppose A is a sparse tensor, B is a dense tensor,<br>
A’s size is (12, 250, 92, 768), B’s size is (250, 768, 768). How to multiply the two tensors.</p>

KFrank: <p>Hi Weikang!</p>
<aside class="quote no-group" data-username="Boltzmachine" data-post="1" data-topic="122087" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/boltzmachine/48/32323_2.png" class="avatar"> Boltzmachine:</div>
<blockquote>
<p>How to multiply the two tensors.</p>
</blockquote>
</aside>
<p>Pytorch supports only certain operations between sparse and dense<br>
tensors.  But you can always convert your sparse tensor to dense:</p>
<pre data-code-wrap="python"><code class="lang-python">dense_result = my_multiply_op (A.to_dense(), B)
</code></pre>
<p>Further, from the shapes of your tensors, it’s not clear how you wish<br>
to “multiply” them together.  Could you post explicit pytorch code<br>
that shows – for two dense tensors of your given shapes – how you<br>
would like them multiplied?</p>
<p>Best.</p>
<p>K. Frank</p>

Boltzmachine: <p>Hi! I posted the question only for asking for sparse operation. so <code>to_dense</code> is not what I want.<br>
The multiplication behavior is the same as <code>A @ B</code>, which multiplies the 3th and 4th dimension and broadcast in the 1st dimension.</p>

KFrank: <p>Hi Weikang!</p>
<aside class="quote no-group" data-username="Boltzmachine" data-post="3" data-topic="122087" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/boltzmachine/48/32323_2.png" class="avatar"> Boltzmachine:</div>
<blockquote>
<p><code>to_dense</code> is not what I want.</p>
</blockquote>
</aside>
<p>Having ruled out using <code>to_dense()</code>, you will have to drill down into<br>
<code>A.indices()</code> and write your own sparse-reshape function.</p>
<p>At issue is that most of pytorch’s <em>tensor</em> (as opposed to matrix)<br>
multiplication functions do not support sparse tensors.  <code>torch.bmm()</code><br>
does, but it only supports 3-d tensors, so it doesn’t quite fit your use<br>
case.</p>
<p>Furthermore, pytorch also does not support reshaping sparse tensors.</p>
<p>In your use case, if you write a sparse-reshape function, you could<br>
use the following expression:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.bmm (my_sparse_reshape (A, (12 * 250, 92, 768)), B.expand (12, 250, 768, 768).reshape (12 * 250, 768, 768)).reshape (12, 250, 92, 768)
</code></pre>
<p>(Note the use of <code>expand()</code> to perform the broadcasting that <code>bmm()</code><br>
does not do automatically.)</p>
<p>This would preserve the benefits of sparsity in the tensor multiplication<br>
(but final result would still be dense), but you would have to write<br>
some not-fully-trivial code to implement it.</p>
<p>Best.</p>
<p>K. Frank</p>
What is the initial ~1.3GB allocated VRAM when first using .cuda()?

lalord: <p>Hi, I was playing with pytorch trying to find how much space some things occupy in VRAM using this <a href="https://github.com/joaqo/silly_pytorch_memory_thingy/blob/master/memory.py" rel="noopener nofollow ugc">script</a> and found that when I first send anything to the GPU, I get a fixed memory usage (seen through nvidia-smi) of around <a href="https://github.com/joaqo/silly_pytorch_memory_thingy/blob/master/memory.py#L50" rel="noopener nofollow ugc">1284MB</a>.</p>
<p>Was wondering what it is due to? I would’ve guessed the cuda runtime sits in normal RAM, but it seems to also occupy space in VRAM? Though I am not understanding how that would work.</p>
<p>Thanks!</p>

eqy: <p>Great question. There are several things going on here. First, a bare CUDA context on a GPU will use about 300MiB VRAM (this gets initialized the first time you call <code>.cuda()</code>). Then <code>libtorch_cuda.so</code> (containing all GPU kernels for various tensor operators, etc.) will be loaded into VRAM taking another 500MiB or so. Finally if you use any cuDNN or cuBLAS functions loading those libraries will also take memory. Additionally, the <a href="https://pytorch.org/docs/stable/notes/cuda.html#memory-management" rel="noopener nofollow ugc">caching allocator</a> may reserve additional GPU memory ahead of time even before it is required for tensors to save time when more allocations are needed.</p>

lalord: <p>Hi thanks a lot for the answer! That makes perfect sense.</p>
<p>The way I originally thought about VRAM was that it only stored the tensors the GPU would operate on, but that the instructions were stored in RAM and fed into the GPU live on each step. From your answer I can see that I was totally wrong, and you actually do store libraries in VRAM. If you don’t mind me asking, is there any resource I could read to get a decent overall idea of how this flow works?</p>
Best way to calculate a neuron size of a model?

pomcho555: <p>I am implementing transfer learning model from NVIDIA SoTA model: <a href="https://github.com/NVIDIA/semantic-segmentation" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/semantic-segmentation: Nvidia Semantic Segmentation monorepo</a></p>
<p>This model was learned from cityscape original image, the resolution is (1024 * 2048). The quality is extraordinary for me. So I need to calculate GPU memory that the model demands. I just write <code>print(model)</code> and calculate neurons in each layer manually. Holy moly!  I know that task is nightmare. Is there a smart way to do?</p>
Packed vs unpacked RNN sequence format for CUDA

bennofs: <p>I am exploring the different ways that RNNs can be implemented for batches of variable-length sequences. I found that PyTorch uses the packed sequence format for variable length batches.</p>
<p>Looking at the documentation for cuDNN 8, I noticed that the packed format appears to be supported only for backwards compatibility:</p>
<blockquote>
<p>This function initializes a previously created RNN data descriptor object. This data structure is intended to support the unpacked (padded) layout for input and output of extended RNN inference and training functions. A packed (unpadded) layout is also supported for backward compatibility.</p>
</blockquote>
<p>(<a href="https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnSetRNNDataDescriptor" class="inline-onebox" rel="noopener nofollow ugc">API Reference - NVIDIA Docs</a>)</p>
<p>This seems to suggest that cuDNN prefers unpacked, padded sequences instead of packed sequences. Why is this the case? Aren’t packed sequences better, since they consume less memory?</p>

googlebot: <p>overhead from extra indirections may be not less significant than minor memory savings. it is better to measure this with tensor shares used in practice.</p>
Numerical stable and overflow problem with torch.exp compared with numpy.exp

Hong_Dinh: <p>I have this tensor/numpy array: aaa = torch.tensor([123, 456, 789]).float()</p>
<p>If using pytorch, the following code has result: &gt; tensor([0., 0., 1.])</p>
<blockquote>
<p>torch.exp(aaa-torch.max(aaa))/ torch.sum(torch.exp(aaa-torch.max(aaa)))</p>
</blockquote>
<p>However, if i use exp function from numpy, the same code leads to the result:</p>
<blockquote>
<p>array([5.75274406e-290, 2.39848787e-145, 1.00000000e+000])</p>
</blockquote>
<p>How can i solve this problem when using pytorch.</p>

googlebot: <p>torch float == float32, you have to explicitly use double() or torch.float64<br>
there is also set_default_dtype global override</p>
In different pytorch version, dropout performs differently(I set the same random seed)

ZhaoweiWang: <p>Hi,<br>
recently, I am migrating code from PyTorch 1.4 to PyTorch 1.7. I set the random seeds of these two pieces of code to 42. When I test my code, I find that torch.nn.Dropout in PyTorch 1.7 dropped different elements compared to torch.nn.Dropout in 1.4.</p>
<p>My function to set random seed:</p>
<pre><code class="lang-auto">def seed_all(num: int = 42):
    random.seed(num)
    np.random.seed(num)
    torch.manual_seed(num)
    if torch.cuda.device_count() &gt; 0:
        torch.cuda.manual_seed_all(num)
        torch.backends.cudnn.deterministic = True
</code></pre>

ptrblck: <p>The internal implementations might have changed (e.g. vectorization could have been added) and I don’t think there is a guarantee for determinism between different PyTorch releases.</p>

ZhaoweiWang: <p>Oh, I get it. Thx a lot</p>
Pyro does not learn linear function

Micha_Christ: <p>Hi there!<br>
I’m trying to use the pyro framework to learn the linear function <code>f(x)=5x+2</code> - for this I’ve worked through the Bayesian Regression Tutorial.</p>
<p>First I’ve designed some noisy data points of the function with</p>
<pre><code>x_data = torch.rand((500, 1))
y_data = 5*x_data +2. + torch.normal(0, 0.1, size=(500, 1))
</code></pre>
<p>Then I’ve build the Bayesian Regression Model. For a sanity check, I’ve set the priors of the weight and the bias to the true values and set a very low variance.</p>
<pre><code>class BayesianRegression(PyroModule):
    def __init__(self, in_features, out_features):
        super().__init__()
        self.linear = PyroModule[nn.Linear](in_features, out_features)
        self.linear.weight = PyroSample(dist.Normal(5, 1).expand([out_features, in_features]).to_event(2))
        self.linear.bias = PyroSample(dist.Normal(2, 1).expand([out_features]).to_event(1))

    def forward(self, x, y=None):
        sigma = pyro.sample("sigma", dist.Uniform(0., 1))
        mean = self.linear(x).squeeze(-1)
        with pyro.plate("data", x.shape[0]):
            obs = pyro.sample("obs", dist.Normal(mean, sigma), obs=y)
        return mean
</code></pre>
<p>Then I’ve ran the SVI for 1500 iterations:</p>
<pre><code>model = BayesianRegression(1, 1)
guide = AutoDiagonalNormal(model)

adam = pyro.optim.Adam({"lr": 0.03})
svi = SVI(model, guide, adam, loss=Trace_ELBO())
pyro.clear_param_store()
for j in range(num_iterations):
    # calculate the loss and take a gradient step
    loss = svi.step(x_data, y_data)
    if j % 100 == 0:
        print("[iteration %04d] loss: %.4f" % (j + 1, loss / len(x_data)))
</code></pre>
<p>Evaluating this model <code>x=torch.arange(0, 1, 0.01</code>) gives the following plot:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/504c222a315bf0f41531a531e8f60733901cd049.png" alt="image" data-base62-sha1="bsljoshGXPrj3ettqGyi9FbtKzL" width="547" height="428"></p>
<p>After the training the mean of the weight is <code>0.0011</code> and the bias is <code>4.5674</code>.<br>
I do not understand why these Means do not correspond to the correct values after the training, although the Priors already contain the desired value? Of course if I decrease the learning rate, the deviation from <code>5</code> and <code>2</code> is smaller after training, but this is obviously not the right way, because this only works because I already have the correct values in the prior.</p>
<p>Moreover if I e.g. choose other prios like</p>
<pre><code>self.linear.weight = PyroSample(dist.Normal(0, 10).expand([out_features, in_features]).to_event(2))
self.linear.bias = PyroSample(dist.Normal(0, 10).expand([out_features]).to_event(1))
</code></pre>
<p>no learning effect occurs,even after 10.000 training iterations, although the variance in the priors is pretty high.</p>
<p>What am I doing wrong?</p>
V1.7.0 Reducer buckets have been rebuilt in this iteration logs

space1panda: <p>Hello,</p>
<p>While using the latest version of DDP during training with 4 GPUs I am getting log “Reducer buckets have been rebuilt in this iteration” 3 times at the beginning of traning, while in the doc string it is stated that the rebuilding of the buckets should be done only once. My understanding is that I see 3 logs because there are 3 additional GPUs on which the buckets should be built but it doesn’t seem to be a trustworthy explanation. Thank you!</p>

mrshenli: <p>Hey <a class="mention" href="/u/space1panda">@space1panda</a>, does the same process print that log 3 times or does the log come from different processes?</p>

space1panda: <p>Hi, Shen, thank you for replying. I’ve actually found out that the extra logs were caused by the arch of my model. I have 3 models and 3 optimizers in my framework, so it makes sense now, why ddp called allocation of buckets 3 times.</p>

wmweng: <p><a class="mention" href="/u/space1panda">@space1panda</a> Hi! I have 3 models and 3 optimizers in my framework as well. Actually, I implemented three torch.nn.parallel.DistributedDataParallel classes as follows, instead of ONE. And also the log ‘Reducer buckets have been rebuilt in this iteration.’  repeated three times. I only use one GPU to train. So it is ok to use multiple DistributedDataParallel models in Pytorch DDP?</p>
<p>model1 = ddp(model1, device_ids=[local_rank], output_device=local_rank)<br>
model2 = ddp(model2, device_ids=[local_rank], output_device=local_rank)<br>
model3 = ddp(model3, device_ids=[local_rank], output_device=local_rank)</p>
Gradient of Loss w.r.t two groups of parameters?

phantom90: <p>Hi There,</p>
<p>Please consider the following case:</p>
<pre><code class="lang-auto">aa = tensor.FloatTensor([10,10])
aa.requires_grad = True
y = net(x)
optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)
loss = aa * creterion(y)
......
</code></pre>
<p>Suppose we want to minimize loss by alternating update aa and parameters in net(). How to implement with backward() and step() function?</p>
<p>Thank you so much!</p>

googlebot: <p>something like:</p>
<pre><code class="lang-auto">aa.requires_grad = bool(epoch % 2)
if aa.requires_grad:
  with torch.no_grad():
    y = net(x)
else:
  y = net(x)
</code></pre>
<p>you may also want to assign aa to a distinct optimizer or parameter group, as this procedure only makes sense if you want to adjust training dynamics of “aa”</p>
Is it possible to calculate the loss over slices of samples within a mini-batch and aggregate that loss?

seankala: <p>Hi. I’m currently using a loss function that causes some resource problems for me (i.e., GPU memory).</p>
<p>Samples within a mini-batch often are often in the hundreds, and each minibatch has a different number of samples. Calculating the loss over these in the “normal” way causes memory issues, so one way that I thought of would be to further divide the data into slices, calculate the loss over those slices, and later aggregate that loss for the backward pass. I’m not completely sure if this would solve the fundamental issues, but thought it’d be worth a try.</p>
<p>Right now, what I have looks something like this:</p>
<pre><code class="lang-auto">num_samples = embeddings.shape[0]
num_batches = num_samples // batch_size

# Include remaining samples.
if num_samples / batch_size != 0:
    num_batches += 1

losses = []
current_idx = 0
for _ in range(num_batches):
    embeddings_batch = embeddings[current_idx:current_idx + batch_size]
    labels_batch = labels[current_idx:current_idx + batch_size]

    loss = self.loss_function(embeddings_batch, labels_batch)
    losses.append(loss)

    current_idx += batch_size
</code></pre>
<p>When I check the value for <code>losses</code> after running this, it looks like:</p>
<pre><code class="lang-auto">tensor(7.2189, device='cuda:0', grad_fn=&lt;MeanBackward0&gt;)
tensor(0., device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
tensor(0., device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
tensor(0., device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
tensor(0., device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
tensor(0., device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
tensor(0., device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>I’m not sure why the remaining values other than the first one are the way they are; I’d assume that they should all be pretty similar. Does anyone know why this might be happening, and how I could fix it?</p>
<p>Thanks.</p>

eqy: <p>That the <code>grad_fn</code> is different between the first loss and the rest is a hint that something is unexpected. Can you check the shapes of <code>embeddings_batch</code> and <code>labels_batch</code> (maybe check few values too) before they are passed to the loss function?</p>
<p>However, a more fundamental issue is that the current problem setup will still leave all of the intermediate tensors from the forward pass during the backward pass in memory. Unless your loss function has some esoteric scaling properties (e.g., nonlinear memory increase with batch size), I’m not sure this will actually save GPU memory. There are fancy methods to save GPU memory at the cost of a little extra recomputation (e.g., <a href="https://arxiv.org/abs/2006.09616" class="inline-onebox" rel="noopener nofollow ugc">[2006.09616] Dynamic Tensor Rematerialization</a>) but I don’t think that is something that is easy to do in upstream PyTorch out of the box.</p>
Parallelizing transforms

ekarais: <p>Hi all,</p>
<p>I have a conceptual question (which I can back up with code if necessary). Let’s say I’m dealing with vector data where part of each vector is a cartesian coordinate (x,y), and I want to write a transform that maps each cartesian coordinate in each vector to its corresponding polar coordinate.</p>
<p>As far as I understand, the common practice is for transforms to operate on samples one-by-one.<br>
Can I write this transform in a vectorized way, i.e. in a way that it transforms a batch of vectors at a time?</p>
<p>Thank you!</p>

Kushaj: <p>You can use broadcasting.</p>
<pre><code class="lang-auto">x = torch.randn(10, 2)

r = torch.sqrt(torch.sum(x**2, dim=1))
t = torch.atan2(x[:,1], x[:0])
</code></pre>

ekarais: <p>Dear Kushajveer,</p>
<p>Thank you for your response.</p>
<p>I know how to use broadcasting; that isn’t my question.</p>
<p>My question is, what is the preferred way to write a <strong>transform</strong> (e.g. the transforms in torchvision that are passed to dataset constructors) such that it operates on a batch of inputs at a time.</p>
<p>It seems that the convention is to apply the transform within the <code>__getitem__()</code> function of the dataset class, but doing it this way means it can’t be parallel.</p>

Kushaj: <p>Oh, I see what you mean. torchvision transforms are just nn.Modules. If you look at the implementations of these transforms like <a href="https://pytorch.org/vision/stable/_modules/torchvision/transforms/transforms.html#Resize" rel="noopener nofollow ugc">Resize</a>. So you only need to define the forward pass. In your case, the input to the transform is the batch itself and you can implement your transform just like any other nn.Module.</p>
Dataloader Make Sure One sample at least for each class in each batch

Dadatata-JZ: <p>Hi folks,</p>
<p>I am meeting one question when comparing the model performances between class-balance and instance-balance. Since the dataset is a little bit skewed, a class has way imbalanced and few samples than others. How can I set the weights to make sure at least one sample of this few-shot class can be batched each time while maintaining the other heavy classes still dominate the quantity in each batch.</p>
<p>In other words, some batches have 0 entities for this few-shot class that makes the loss computes as nan.</p>
<p>Thank you in advance.</p>

eqy: <p>Does writing a custom sampler that is guaranteed to sample from the smaller class every x samples work? Alternatively, a more crude but simpler solution is to just set weights for the desired balance you want and to simply throw out batches that have 0 entities for that class. Note that if you set <code>shuffle=True</code> this will still use all of your data over time, just not over every epoch.</p>

Dadatata-JZ: <p>Thanks for the note. I think the simpler one is surely working but I am not sure whether it would guarantee the smaller class will be drawn while the larger classes maintain the majority. UNLESS I set the smaller set significantly heavily, somehow make it close to the larger set. But in that case, it will be more like an instance-balance sampling than class-balance sampling. Do you happen to recommend any reference customized sampler sample codes that serve similar tasks ?</p>
What does torch.utils.bottleneck do on its two runs?

pytorcher: <p>I am trying to do some profiling with torch.utils.bottleneck.  I see it gets run twice, once with the line:</p>
<blockquote>
<p>Running your script with cProfile</p>
</blockquote>
<p>and a second time with</p>
<blockquote>
<p>Running your script with the autograd profiler…</p>
</blockquote>
<p>The first one runs to completion and the second one crashes. Based on my code my suspicion is that it doesnt reset the entire python call, just runs the script twice.  I have some initialization items that might be doing funky things if this is the case, i.e. checking if variables have been initialized so the second time they will have been.  Just want to confirm this is how it works before diving into how to ‘uninitialize’ my variables again.</p>
Round of floating point value and backpropagation

cbd: <p>I have round of the floating point value using below code but its not allow to backpropagate the gradient. Is there anyway to round off value with two decimal point so that it backpropagate the gradient?</p>
<pre><code class="lang-auto"> x=torch.FloatTensor([2.123])
 n_digits=2
 x_r = (x * 10**n_digits).round() / (10**n_digits)
 print(x_r)</code></pre>

eqy: <p>Does something like a custom round function that actually uses the gradient of the original tensor work? <a href="https://discuss.pytorch.org/t/torch-round-gradient/28628/6" class="inline-onebox">Torch.round() gradient - #6 by Zhongzhi_Yu</a></p>