akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I want to add the normalize term in augmentation transform

username: Jake_Eum
cooked: <p>Here is my mapper for augmentation</p>
<pre><code class="lang-auto">def mapper2(dataset_dict):
    dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
    #image = utils.read_image(dataset_dict["file_name"], format="BGR")    
    image = utils.read_image(dataset_dict["file_name"], format="RGB")    

    transform_list = [
                   
                    T.RandomFlip(prob=0.5, horizontal=False, vertical=True)
                    ,T.ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice')
                    #,T.RandomCrop('relative_range', (0.4, 0.6))
                    #,T.GridSampleTransform()
 
                    
                      ]
    image, transforms = T.apply_transform_gens(transform_list, image)
    dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))

  
    annos = [
        utils.transform_instance_annotations(obj, transforms, image.shape[:2])
        for obj in dataset_dict.pop("annotations")
        if obj.get("iscrowd", 0) == 0
    ]

    instances = utils.annotations_to_instances(annos, image.shape[:2])
    dataset_dict["instances"] = instances
    return dataset_dict
</code></pre>
<p>I want to put normalize option like T.GridSampleTransform()<br>
but when I perform the above code</p>
<p><code>TypeError: __init__() missing 2 required positional arguments: 'grid' and 'interp'</code></p>

username: ptrblck
cooked: <p><code>GridSampleTransform</code> doesn’t seem to be a built-in transformation and based on the raised error message it requires two input arguments: <code>grid</code> and <code>interp</code>, so you would have to provide them.<br>
Since I cannot find the implementation, I can just guess that the arguments might be similar to the <code>grid_sample</code> method.</p>
Image input and mask target does not have the same size

username: MA_CASANDRA_QUILANG
cooked: <p>Hi, good day!</p>
<p>I am training a neural network and my input image is a 4D tensor [batch_size, channel, height, width]. I also had a 4D tensor target but since I got an error saying my target size should only be 3D, i tried mask.squeeze(1) to get rid of the channel index. Now after I did that, I had another error, this time it says my input size and target size mismatch.</p>
<p>This is my code</p>
<pre><code class="lang-auto">epochs = 1
for epoch in tqdm(range(epochs)):
    for batch_idx, (img, mask) in tqdm(enumerate(train_gen)):
      img = torch.Tensor(img).view(-1, 3, 500, 500)
      mask = torch.Tensor(mask).view(-1, 1, 500, 500)
      mask = mask.squeeze(1)

      model.zero_grad()

      outputs = model(img)
      loss = loss_function(outputs, mask)
      loss.backward()
      optimizer.step()    # Does the update 

    print(f"Epoch: {epochs}. Loss: {loss}")
</code></pre>
<p>and my error is</p>
<pre><code class="lang-auto">0%|          | 0/1 [00:00&lt;?, ?it/s]
0it [00:06, ?it/s]
  0%|          | 0/1 [00:06&lt;?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-13-ad775b024a80&gt; in &lt;module&gt;()
     12       outputs = model(img)
     13       outputs = outputs.squeeze(1)
---&gt; 14       loss = loss_function(outputs, mask)
     15       loss.backward()
     16       optimizer.step()    # Does the update

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2264         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2265     elif dim == 4:
-&gt; 2266         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2267     else:
   2268         # dim == 3 or dim &gt; 4

RuntimeError: size mismatch (got input: [4, 60, 504, 504] , target: [4, 500, 500]

</code></pre>
<p>does anyone have any idea how to solve this? Thank you!</p>

username: ptrblck
cooked: <p>Your model seems to increase the spatial size of the input from 500 to 504, so you should take a look at the layers and make sure the output size is as expected.</p>
THFloat_gemm implementation

username: a_d
cooked: <p>Hi,<br>
I was trying to use the DCNv2 which uses the <code>THFloatBlas_gemm</code>. Now since that has been removed in Pytorch 1.7, I was wondering where I could find the source code of the same function so that I can define the said function and make the model.<br>
TIA</p>

username: ptrblck
cooked: <p>You could (and should) use <code>torch::addmm</code> instead as described <a href="https://github.com/pytorch/pytorch/issues/47750#issuecomment-725554682">here</a>.</p>

username: a_d
cooked: <p>Thanks a lot for replying <a class="mention" href="/u/ptrblck">@ptrblck</a>, will do that</p>
How does nn.optimizer work? How do they locate the parameters?

username: yuanqi_chau
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn

class LinearModel:

    def __init__(self, train_x, train_y):
        self.train_x = train_x
        self.train_y = train_y
        self.W = torch.tensor([0.5], requires_grad=True)
        self.b = torch.tensor([0.5], requires_grad=True)
        self.params = [self.W, self.b]
    
    def forward(self, x):
        self.params = [self.W, self.b]
        return self.W * x + self.b
    
    def train(self):
        criterion = nn.MSELoss()
        optimizer = torch.optim.SGD(self.params, lr=0.01)
        for i in range(10):
            print(self.W, self.b)
            optimizer.zero_grad()
            y_pred = self.forward(self.train_x)
            loss = criterion(y_pred, self.train_y)
            loss.backward()
            optimizer.step()

train_x = torch.Tensor([0,1,2,3])
train_y = torch.Tensor([1,3,5,7])

model = LinearModel(train_x, train_y)
model.train()
</code></pre>
<pre><code class="lang-auto">tensor([0.5000], requires_grad=True) tensor([0.5000], requires_grad=True)
tensor([0.6200], requires_grad=True) tensor([0.5550], requires_grad=True)
tensor([0.7300], requires_grad=True) tensor([0.6053], requires_grad=True)
tensor([0.8307], requires_grad=True) tensor([0.6513], requires_grad=True)
tensor([0.9230], requires_grad=True) tensor([0.6933], requires_grad=True)
tensor([1.0076], requires_grad=True) tensor([0.7318], requires_grad=True)
tensor([1.0851], requires_grad=True) tensor([0.7669], requires_grad=True)
tensor([1.1561], requires_grad=True) tensor([0.7990], requires_grad=True)
tensor([1.2212], requires_grad=True) tensor([0.8284], requires_grad=True)
tensor([1.2809], requires_grad=True) tensor([0.8552], requires_grad=True)
</code></pre>
<p>These are the inputs and outputs from my simple linear model. I want to ask why the values of self.W and self.b get updated?</p>
<p>Only the values of self.W and self.b are passed  into self.params, how can the optimizer locate self.W and self.b to update their values?</p>

username: ptrblck
cooked: <p>You are passing <code>self.params</code> to the optimizer in:</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD(self.params, lr=0.01)
</code></pre>
<p>which is why they get updated.</p>
<p>As a side note: you could create trainable parameters via <code>nn.Parameter</code>, which would properly register them inside the module and then pass all parameters via <code>model.parameters()</code> to the optimizer, which would avoid creating the list manually.</p>
How does PyTorch detect the CUDA installation?

username: lykius
cooked: <p>Hi, a question mainly out of curiosity: how does PyTorch detect the CUDA installation on the pc?<br>
For instance, on my pc I have CUDA-10.1 installed in /usr/local/cuda-10.1 but I have not added it neither to PATH or to LD_LIBRARY_PATH. Nevertheless, PyTorch detects its seamlessly and everything works out fine. How is this done?</p>
<p>Thanks in advance for any hint!</p>

username: ptrblck
cooked: <p>The conda binaries and pip wheels ship with their own CUDA runtime and your local CUDA toolkit will not be used.<br>
You would only need to provide a sufficiently new NVIDIA driver and it should work.<br>
Your local CUDA toolkit would be used, if you are building PyTorch from source or are building custom CUDA extensions.</p>

username: lykius
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a>, that makes a lot of sense!<br>
And also, it’s a great deploy strategy, a big thumbs up for PyTorch once again!</p>
`forward` is calling a strange attribute in custom autograd.Functon class

username: Nikronic
cooked: <p>Hi all,</p>
<p>Let’s say I have a class called <code>NonLinearProblem</code> which solves a specific type of problem, give different values to its arguments. After instantiating a object from this class, one only needs to pass a numpy array to this object like to do the computation. I have also access to analytically obtained gradients</p>
<pre><code class="lang-auto">nlp = NonLinearProblem(constraints, #other_args)
input = np.randn(5, )  # any 1D array

# forward function
nlp.__objective(input)  # returns is a scalar 
# backward function
nlp.__gradient(input)  # returns array with the same shape of input
</code></pre>
<p>Now, I want to extend <code>autograd.Function</code> to use this class and its corresponding <code>.__objective</code> and <code>.__gradient</code> in <code>forward</code> and <code>backward</code> respectively.</p>
<p>This is what I have done so far:</p>
<pre><code class="lang-auto"># I init `nlp` outside of Function

class TopoptFunction2(autograd.Function):
    @staticmethod
    def forward(ctx, densities: torch.Tensor, nlp):
        
        # tps.setOverallDensities()
        output = nlp.__objective(densities.data.numpy())

        ctx.save_for_backward(densities, nlp)

        return torch.FloatTensor(output)

    @staticmethod
    def backward(ctx, grad_output):
        densities, nlp = ctx.saved_tensors
        # tps.setOverallDensities(densities.data.numpy())
        objective_gradient = nlp.__gradient(densities.data.numpy())
        return torch.FloatTensor(objective_gradient * grad_output), None
</code></pre>
<p>I am trying to <code>gradcheck</code> by:</p>
<pre><code class="lang-auto">topopt = TopoptFunction2.apply
dummy_in = torch.empty(100, dtype=torch.float64).fill_(0.6)
dummy_in.requires_grad = True

test = autograd.gradcheck(topopt, (dummy_in, nonLinearProblem), eps=1e-6, atol=1e-4)
print(test)
</code></pre>
<p>But in <code>forwad</code>, I get a strange error where an attirbute combined with name of class and method is trying to be called which obviously does not exist:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/nikan3/diff-voxelfem-optimization-problem/VoxelFEM/python/demo_1.py", line 219, in &lt;module&gt;
    test = autograd.gradcheck(topopt, (dummy_in, nonLinearProblem), eps=1e-6, atol=1e-4)
  File "/home/nikan3/.local/lib/python3.6/site-packages/torch/autograd/gradcheck.py", line 323, in gradcheck
    func_out = func(*tupled_inputs)
  File "/home/nikan3/diff-voxelfem-optimization-problem/VoxelFEM/python/demo_1.py", line 187, in forward
    output = nlp.__objective(densities.data.numpy())
AttributeError: 'cyipopt.problem' object has no attribute '_TopoptFunction2__objective'

</code></pre>
<p>As you can see, somehow <code>_[ClassName][attributeName]</code> is being called!</p>
<p>Some further details:</p>
<ol>
<li>
<code>cyipopt.problem</code> is <code>type(nlp)</code>
</li>
<li>I have created python bindings for this package and it works fine except in this issue</li>
</ol>
<p>Thanks for your suggestions and comments<br>
Bests</p>
How dose slice assignment work in pytorch

username: pytorch_lzwhard
cooked: <p>b = a[:, 0:2], I guess slice of <strong>a</strong> will copy to tensor <strong>b</strong><br>
a[: 0:2] = a[:, 2:4], but how this work behind?</p>

username: googlebot
cooked: <aside class="quote no-group" data-username="pytorch_lzwhard" data-post="1" data-topic="102493">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorch_lzwhard/48/28763_2.png" class="avatar"> pytorch_lzwhard:</div>
<blockquote>
<p>b = a[:, 0:2], I guess slice of <strong>a</strong> will copy to tensor <strong>b</strong></p>
</blockquote>
</aside>
<p>nope, b and a share memory, though b is a new python wrapper object with different strides</p>
<aside class="quote no-group" data-username="pytorch_lzwhard" data-post="1" data-topic="102493">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorch_lzwhard/48/28763_2.png" class="avatar"> pytorch_lzwhard:</div>
<blockquote>
<p>a[: 0:2] = a[:, 2:4],</p>
</blockquote>
</aside>
<p>here python’s <code>object.__setitem__</code> mechanism is used instead, making partial assignment with copying possible</p>

username: pytorch_lzwhard
cooked: <aside class="quote no-group" data-username="googlebot" data-post="2" data-topic="102493">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ba8739/48.png" class="avatar"> googlebot:</div>
<blockquote>
<p>here python’s <code>object.__setitem__</code> mechanism is used instead,</p>
</blockquote>
</aside>
<p>tkx，and how does torch tensor implement this a[: 0:2] = a[:, 2:4]?</p>

username: googlebot
cooked: <p>Should be the same as a[:,0:2].copy_(a[:,2:4]), i.e. strided mem. copy</p>

username: tom
cooked: <aside class="quote no-group" data-username="googlebot" data-post="5" data-topic="102493" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ba8739/48.png" class="avatar"> googlebot:</div>
<blockquote>
<p>Should be the same as a[:,0:2].copy_(a[:,2:4]), i.e. strided mem. copy</p>
</blockquote>
</aside>
<p>Indeed, in particular, the RHS is created and then copied over to the slice.</p>

username: pytorch_lzwhard
cooked: <p>tkx, so in <code>a[:,0:2].copy_(a[:,2:4])</code>, <code>a[:,0:2]</code> is just a reference not a clone?</p>

username: googlebot
cooked: <p>That’s correct. ~~~~~~~~</p>
How can I run out of memory when most memory is available?

username: Anders_G
cooked: <p>Im doing some computations (basically using torch as numpy with gpu) and I ran out of memory when there is plenty available. How is that possible?</p>
<p>I get the following error:<br>
RuntimeError: CUDA out of memory. Tried to allocate 144.00 MiB (GPU 0; 11.00 GiB total capacity; 972.39 MiB already allocated; 8.57 GiB free; 986.00 MiB reserved in total by PyTorch)</p>
<p>So basically I have 8.57GB free but run out of memory trying to allocate 144MB. Is it that the GPU memory can suffer from very severe heap fragmentation or something in that regard?</p>

username: ptrblck
cooked: <p>Could you post a code snippet to reproduce this issue?<br>
I doubt it’s memory fragmentation, so would like to debug it.</p>

username: Anders_G
cooked: <p>I found the reason for the low memory. I was simply using much more memory than I expected. And the problem seem to be that the error message was missleading. The reason for that I don’t know.</p>
Trouble with NaN

username: oma219
cooked: <p>I am trying to train an autoencoder and I using torch.nn.MSELoss() as the loss function. I keep getting this error below:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-76-67910671b729&gt; in &lt;module&gt;()
     20 
     21         optimizer.zero_grad()
---&gt; 22         loss.backward()
     23         optimizer.step()
     24 
RuntimeError: Function 'MseLossBackward' returned nan values in its 0th output.
</code></pre>
<p>I have tried reading many of the other posts on this topic and I have tried various approaches such as adjusting the learning rate, gradient clipping. The weird thing is the loss doesn’t seem to diverge at all. I am not totally sure why this error keeps occurring for me, any suggestions or help would be appreciated? I am very new to pytorch.</p>

username: pchandrasekaran
cooked: <p>Could you check if you have any nan’s in your input? And if there aren’t any in it, could you check if the outputs are producing nan’s?</p>
How to apply pruning to a model with weight normalization?

username: eddie
cooked: <p>I have trained a model for waveform synthesis and would now like to apply global pruning using an iterative pruning schedule. However, the model uses weight normalization in most of its layers, which means the weights are split into two components, <code>weight_g</code> and <code>weight_v</code>, from which the actual weights are computed on each forward pass. It doesn’t make sense to apply pruning on either <code>weight_g</code> or <code>weight_v</code> only, so I need to find some way to temporarily get the un-normalized weights back.</p>
<p>So far I have considered two approaches:</p>
<ol>
<li>For each pruning step, disable weight normalization, conduct the pruning, and re-enable weight normalization. This doesn’t work because pruning will create a parameter <code>weight_orig</code> while after re-enabling weight normalization there will be no <code>weight</code> anymore. Also re-enabling weight normalization seems to create a new set of parameters which are not tracked by the optimizer.</li>
<li>For each pruning step, compute the actual weights by calling the <code>_weight_norm</code> function, replace <code>weight_v</code> with the result, conduct the pruning and undo the swapping. The problem here is that the pruning function will write into <code>weight_v</code> the values of <code>weight</code> with the pruning mask applied, which means you can’t just swap the former values back in.</li>
</ol>
<p>How could I go about this? Could I make one of the described approaches work in an elegant way or is there another possibility that I didn’t think of?</p>
Using nn.EmbeddingBag with a CNN in PyTorch model

username: doruk
cooked: <p>I am trying to build a text classifying model in PyTorch using <code>nn.EmbeddingBag</code> and a CNN.<br><br>
I know there are a bunch of NLP CNN models using <code>nn.Embedding</code> out there but due to my hardware constraints I do not want to use <code>nn.Embedding</code>.</p>
<p>I have this simple model setup for starters (16 is the batch size):</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, vocab_size, embed_dim, num_class):
        super().__init__()
        self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=True)
        self.conv1 = nn.Sequential(
            nn.Conv1d(in_channels=embed_dim, out_channels=16, kernel_size=1),
            nn.ReLU(),
            nn.MaxPool1d(1)
        )
        self.fc = nn.Linear(16, num_class)

    def forward(self, text, offsets):
        x = self.embedding(text, offsets).unsqueeze(2)
        x = self.conv1(x)
        x = x.view(-1, 16)
        x = self.fc(x)
        return x
</code></pre>
<p>This <code>CNN</code> model “compiles” (the dimensions and everything matches) but training does not accuracy stays at ~7% (random guessing). In other words, this model doesn’t work.</p>
<p>I should also add that the reason I’m trying to do this is to improve the following fully connected model (which works fine):</p>
<pre><code class="lang-auto">class FC(nn.Module):
    def __init__(self, vocab_size, embed_dim, num_class):
        super().__init__()
        self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=True)
        self.fc = nn.Linear(embed_dim, num_class)

    def forward(self, text, offsets):
        embedded = self.embedding(text, offsets)
        return self.fc(embedded)
</code></pre>
<p>How can I implement a CNN model using the <code>nn.EmbeddingBag</code>? <br><br>
If that’s not possible, how can I improved the <code>FC</code> model?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Based on your code snippet it seems your temporal dimension has only a single value which would also explain the kernel size of <code>1</code> in the conv layer. In that case, this operation should be equal to a linear layer. Also the pooling layer won’t have any effect with a kernel size of <code>1</code> as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 16, 24)
pool = nn.MaxPool1d(1)
out = pool(x)
print((x == out).all())
&gt; tensor(True)
</code></pre>
<p>so you can remove it.</p>
Apply nn.PixelShuffle() in training a network with a 3-channel input

username: Xinyi
cooked: <p>My input tensor size is (72, 72) and the channel is 3. I have used <code>nn.conv2d()</code> in my network. And I want to use <code>nn.PixelShuffle()</code> after the convolution layers to get a HR ouput. I have learned that <code>nn.PixelShuffle()</code> gets a input of (4, 72, 72) and give an output in (1, 144,144). I have several issues.</p>
<ol>
<li>
<p>I think the tensor dimension cannot be expanded during training. So is my input of my network supposed to be in (3, 4, 72, 72)?</p>
</li>
<li>
<p>If so, how to set the channel parameters in <code>nn.conv2d()</code> ? Like <code>nn.conv2d(in_channels=3, out_channels=64)</code> ?</p>
</li>
</ol>

username: ptrblck
cooked: <p><code>nn.PixelShuffle</code> expects an input in the shape <code>[*, C * r**2, H, W]</code> and outputs <code>[*, C, H*r, W*r]</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.PixelShuffle.html?highlight=pixelshuffle#torch.nn.PixelShuffle">docs</a>.<br>
If you want to use an input of <code>[*, 4, 72, 72]</code>, <code>r</code> would correspond to <code>2</code> and thus <code>C</code> to <code>1</code>.<br>
In that case your conv layer should probably output <code>4</code> channels.</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: PyTorch error

username: Li-Yun
cooked: <p>I am trying to run some code in PyTorch but I am got stacked at that point:</p>
<p>At first iteration, both backward operations, for Discriminator and Generator are running well</p>
<pre><code>....

self.G_loss.backward(retain_graph=True)

self.D_loss.backward()

...
</code></pre>
<p>At the second iteration, when <code>self.G_loss.backward(retain_graph=True)</code> executes i got this error:</p>
<pre><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8192, 512]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>According to <code>torch.autograd.set_detect_anomaly</code>, the last of the following lines in the Discriminator network, is responsible for this:</p>
<pre><code>    bottleneck = bottleneck[:-1]
    self.embedding = x.view(x.size(0), -1)
    self.logit = self.layers[-1](self.embedding)
</code></pre>
<p>The strange thing is that I have used that network architecture in other code and worked properly. Any suggestion?</p>
<p>the full error:</p>
<pre><code>  &gt;   site-packages\torch\autograd\__init__.py", line 127, in backward
</code></pre>
<blockquote>
<pre><code>    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8192, 512]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
</blockquote>

username: ptrblck
cooked: <p>It’s unclear what exactly is causing this error, since you haven’t posted a code snippet to reproduce this issue.<br>
However, based on the description and since you are using a GAN, you might be facing a similar issue as described <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">here</a>.</p>
Upsampling input with nearest neighbors' pixel value

username: navid_mahmoudian
cooked: <p>Hello,<br>
When I am trying to upsample my input using the <code>nn.functional.interpolate</code> function (nearest mode), I get the following error</p>
<blockquote>
<p>return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors)<br>
RuntimeError: input tensor has spatial dimension larger than the kernel capacity</p>
</blockquote>
<p>I think it is mainly because my input tensor is big, but I don’t have any memory issues. Is there any way to solve this problem without shrinking the input tensor?</p>

username: ptrblck
cooked: <p>This is a limitation in the CUDA launch config for the current algorithm. As a workaround you could use the CPU for these large shapes and create a feature request on GitHub.</p>
Tan of shear factor in get_shear_matrix2d

username: pgp
cooked: <p>Tan of shear factors is taken in <code>get_shear_matrix2d</code> function. It changes the shear factor from what is expected. What is the reason behind using tan in shear?</p>

username: ptrblck
cooked: <p>I assume it’s the general formula for the shear matrix as described in <a href="http://www.sci.sdsu.edu/visualstructure/vss/htm_hlp/simpl_s.htm">this simple example</a>.</p>
Why does a model definition have both a __init__ and forward function?

username: Stanley_C
cooked: <p>Could someone explain to me why pytorch has both <strong>init</strong> and forward function in a model definition, and what each one does, and why do we need 2 function rather than just one?</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html">This tutorial</a> might be a good starter to understand why <code>nn.Modules</code> are built in this way.<br>
The TL;DR: all parameters and submodules are defined and registered in the <code>__init__</code> and used in the <code>forward</code> method.</p>
How to add activation histogram in tensorboard?

username: stroncea
cooked: <p>Currently this is how i add a histogram for my layer’s weights:</p>
<pre><code class="lang-auto">writer.add_histogram(f'Classifier/p/Weights',model.fc[-1].weight, epoch)
</code></pre>
<p>How can I add the activation histogram in a similar manner?</p>
<p>My assumption would be that I have to add it in the forward function after it passes through the relevant ReLU, softmax, etc. However what do I do when i don’t have access to the forward function directly (model imported from torchvision.models, or defined as a <code>nn.Sequential</code>).</p>
<p>Example of what i’m looking for taken from <a href="https://tensorboard.dev/experiment/EDZb7XgKSBKo6Gznh3i8hg/#histograms" rel="noopener nofollow ugc">tensorboard.dev</a>:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d96daf4b39b858305c8451535b9d27684c0ac79.png" alt="image" data-base62-sha1="b4nYdjG2QmON0UbYf55Mni5tzoR" width="321" height="231"></p>

username: ptrblck
cooked: <p>You can use forward hooks to get the internal activation as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time

username: moreshud
cooked: <p>I get this error when training on a CPU machine but not on GPU machine i.e google colab though I am computing two losses.</p>
<p><code>if phase == 'train':         loss1.backward()          loss2.backward()           optimizer.step()</code></p>

username: ptrblck
cooked: <p>Which PyTorch version are you using? If not the latest one, could you update and rerun the code again?<br>
If one backend detects this error, it should be raised by all backends, so it might be an internal bug.</p>

username: moreshud
cooked: <p>My Pytorch version is 1.7.0+cpu. I am computing two losses and the error message when training. However, once I pass the retain_graph=True in the loss1, the training is completed successfully.</p>

username: ptrblck
cooked: <p>Since this error is only raised on the CPU, while the GPU run seems to be working, I would still consider it a potential issue.<br>
Could you post an executable code snippet to reproduce this issue, please?</p>
Program hangs at os.waitpid

username: hudsontek
cooked: <p>I just ran into a very weird bug. When using multiprocessing to create a network, the program hangs indefinitely. Code is as below:</p>
<pre><code class="lang-auto">import torch
from torch import nn
import torch.multiprocessing as mp

def normalized_columns_initializer(weights, std=1.0):
    out = torch.randn(weights.size())
    out *= std / torch.sqrt(out.pow(2).sum(1, keepdim=True))
    return out

class ACNet(nn.Module):
    def __init__(self, max_addrs, num_loc, hidden_dim = 64):
        super().__init__()
        self.loc_linear = nn.Linear(hidden_dim, num_loc)

        print('start init')
        self.loc_linear.weight.data = normalized_columns_initializer(
            self.loc_linear.weight.data, 0.01)    #todo: this line causes deadlock somehow
        self.loc_linear.bias.data.fill_(0)

        print("init done")

def create_model(dim):
    mdl = ACNet(100, dim)
    print('model is created')

if __name__ == '__main__':
    shared_model = ACNet(100, 512)
    p = mp.Process(target=create_model, args=(512,))
    p.start()
    p.join()
</code></pre>
<p>Interrupting the program will show that the program hangs at os.waitpid(). If either model’s second parameter is changed to a smaller number than 512, the program will pass. The same thing will happen if the two models are both created by directly invoking the constructor, or by two child processes. Does anyone have any idea about this? My environment is as below:</p>
<p>PyTorch version: 1.7.0+cpu<br>
Is debug build: True<br>
CUDA used to build PyTorch: None<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Linux Mint 19.1 Tessa (x86_64)<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
Clang version: 7.0.0 (tags/RELEASE_700/final)<br>
CMake version: version 3.12.3</p>
<p>Python version: 3.6 (64-bit runtime)<br>
Is CUDA available: False<br>
CUDA runtime version: No CUDA<br>
GPU models and configuration: No CUDA<br>
Nvidia driver version: No CUDA<br>
cuDNN version: No CUDA<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.4<br>
[pip3] torch==1.7.0+cpu<br>
[pip3] torchaudio==0.7.0<br>
[pip3] torchvision==0.8.1+cpu<br>
[conda] Could not collect</p>
Torchvision.transforms.functional.pad with pillow

username: nastaran
cooked: <p>Padding pillow image with <code>torchvision.transforms.functional.</code>pad` fails.</p>
<pre><code class="lang-python">w, h = image.size
max_wh = np.max([w, h])
hp = int((max_wh - w) / 2)
vp = int((max_wh - h) / 2)
padding = (hp, vp, hp, vp)
print(F.pad(image, padding, 0, 'constant'))
</code></pre>
<p>This gives me this error:</p>
<pre><code class="lang-auto">AttributeError                            Traceback (most recent call last)
&lt;ipython-input-49-022af53bc969&gt; in &lt;module&gt;
      5 vp = int((max_wh - h) / 2)
      6 padding = (hp, vp, hp, vp)
----&gt; 7 print(F.pad(image, padding, 0, 'constant'))

/nas/env/work/lib/python3.8/site-packages/torch/nn/functional.py in _pad(input, pad, mode, value)
   3548                 _pad, (input,), input, pad, mode=mode, value=value)
   3549     assert len(pad) % 2 == 0, 'Padding length must be divisible by 2'
-&gt; 3550     assert len(pad) // 2 &lt;= input.dim(), 'Padding length too large'
   3551     if mode == 'constant':
   3552         return _VF.constant_pad_nd(input, pad, value)

AttributeError: 'PngImageFile' object has no attribute 'dim'
</code></pre>
<p>Per `torchvision.transforms.functional. pad’ docs, pad should take in pillow image. What is going wrong here?</p>

username: SimonW
cooked: <p>look at trace. you are calling torch.nn.functional.pad</p>
Loading data and splitting

username: ManishSharma
cooked: <p>Help needed</p>
<p>I have 20000 images with some id assigned to it<br>
Ex : snake1.jpg ( ye id hogya )</p>
<p>And for these 20k images , i have a csv file for which each id is assigned a class</p>
<p>Ex : Snake1.jpg ( id ) , class ( Rattlesnake)<br>
Basically ( 20000 , 2) shape of dataframe<br>
Total num of unique classes is 10 ( in the sense 20k images are from these 10 classes )</p>
<p>Now , i want to segregate these 20k images into 10 diff folder based on their respective class present in .csv file</p>
<p>Or is there any func in pytorch (like in tensorflow callled as flow_from_dataframe)</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ManishSharma" data-post="1" data-topic="103424">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/manishsharma/48/31374_2.png" class="avatar"> ManishSharma:</div>
<blockquote>
<p>Now , i want to segregate these 20k images into 10 diff folder based on their respective class present in .csv file</p>
</blockquote>
</aside>
<p>You could load the <code>.csv</code> file via e.g. <code>pandas</code> and copy the files using the class id, if I understand your use case correctly. PyTorch doesn’t provide utilities to copy or move data.<br>
Alternatively, you could also write a custom <code>Dataset</code> and load the files using your custom logic as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.</p>

username: ManishSharma
cooked: <p>Thank you for your quick response <a class="mention" href="/u/ptrblck">@ptrblck</a>. I shall do the same now. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Will nn.transformer take 3D masks?

username: JMoo
cooked: <p>I noticed that nn.multiheadattention does take a 3D mask of (N*num_heads, L, S) but that the transformer module only gives a 2D mask of (L,S) as a possible attention mask shape.</p>
<p>However, in its documentation, nn.transformer uses multiheadattention (“from .activation import MultiheadAttention”). Could I expect a transformer given a 3D mask to work properly?</p>
Libtorch C++ tensor output to a file

username: John_J_Watson
cooked: <p>I have some libtorch code which runs a forward and gets me the result like so:</p>
<p><code>torch::Tensor output = module.forward(inputs_f).toTensor();</code></p>
<p>the problem I am facing is that I am not able to write this to an output in a clean way. The best I could do is this:</p>
<pre><code class="lang-auto">std::fstream res("result.out", std::ios::out | std::ios::binary);
res &lt;&lt; output;
res.close();
</code></pre>
<p>and while this works. it writes me a bit of nonsense like so:</p>
<pre><code class="lang-auto">Columns 1 to 10-0.0474 -0.0582  0.0047 -0.0048 -0.0018 -0.0153  0.0084 -0.0178 -0.0336 -0.0082

Columns 11 to 20 0.0119 -0.0617 -0.0218  0.0003  0.0205 -0.0086  0.0285 -0.0193  0.0052 -0.0009

Columns 21 to 30 0.0014 -0.0140 -0.0333  0.0931  0.0562  0.0246  0.0674 -0.0256  0.0018  0.0125

Columns 31 to 40 0.0015 -0.0564 -0.0016  0.0185 -0.0028 -0.0436 -0.0289 -0.0729 -0.0255  0.0550

Columns 41 to 50 0.1132  0.0346 -0.0030 -0.0842  0.0433  0.1089  0.0023 -0.0148 -0.0411  0.0094
</code></pre>
<p>It looks like this is a string represnetation of some sort - I REALLY dont want these <code>Columns 11 to 20</code> etc… I just want the tensor in a file line after line <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
How does one go about doing this?</p>
Gradient is None after backward

username: BGjeroska
cooked: <p>Hey!</p>
<p>I recently started with Pytorch and right now I am facing a problem with computing the gradients.</p>
<p>I have a list of Tensors (L) with <code>required_grad = True</code>, that is stucked in one Tensor: <code>T=torch.stack(L,dim=0)</code>.</p>
<p>So, when I am trying to compute the gradients from all elements from the Tensor T with <code>out.backward()</code> , <code>T[0].grad</code> is always <code>None</code>.</p>
<p>How can I get the gradients in this case?</p>

username: pascal_notsawo
cooked: <p>Where does <code>out</code> come from in your example?</p>

username: Henry_Chibueze
cooked: <p>Where exactly is out coming from?</p>
<p>If u r training a model then I’m assuming that out is the output of ur model which in that case the code is supposed to be:</p>
<pre><code class="lang-auto">loss.backward()
</code></pre>
<p>Not</p>
<pre><code class="lang-auto">out.backward()
</code></pre>
Fixing manual_seed for validation but not in training

username: Ahmad_Nedal
cooked: <p>I want to validate my model using only a fixed subset of the whole dataset  without affecting the training loop. I want my code to look something like</p>
<blockquote>
<p>for epoch in range(num_epochs):</p>
<pre><code>for batch in train_dataloader:
     train_step()  #  loss, optimize, etc ..

torch.manual_seed(0) # This fixes the training data too!

for repetition in range(segments_per_speaker) # sampling multiple segments per speaker
    for batch in valid_dataloader:
         valid_step() # choose best model
</code></pre>
</blockquote>
<p>Placing torch.manual_seed(0) after the training loop somehow fixes the training data in different epochs. What am I missing? Any recommendations to solve my issue?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p><code>torch.manual_seed</code> is used globally, so you would reseed the code each time you are calling this method.<br>
I think the proper way would be to create the fixed subset for the validation once and reuse it in each iteration.</p>

username: Ahmad_Nedal
cooked: <p>Thanks a lot for your answer <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>I was hoping to find a workaround not to fix a subset of the validation set. But it seems there aren’t. At least now I can proceed with it in peace.</p>
Using unmodified input

username: ardiii
cooked: <p>Hello, I am trying to learn two distributions with my network. The first one is a categorical distribution and the second one a normal distribution. Below is my pseudocode of the forward() function which illustrates the problem. From input2 i only take the value at that index, which the categorical distribution outputs and then use this value as the mean of the normal distribution. But autograd returns <em>Nones</em> since somehow the output is not connected to input. I hope there is a workaround, for which I would be very thankful! The <em>action_std</em> is a parameter defined in <em>init</em>.</p>
<p><code>def forward(input1, input2):</code></p>
<p><code>x=softmax(layer1(input1))</code><br>
<code>dist1 = Canonical(x) </code><br>
<code>index = dist1.sample() </code><br>
<code>action_mean = input2[index]</code><br>
<code>dist2 = Normal(action_mean, action_std)</code><br>
<code>action = dist2.sample()</code></p>
<p><code>return action</code></p>
Creating custom pytorch dataset class for n number of target columns

username: IamSparky
cooked: <p>I have a tabular dataset , where I have to perform multi-label classfication . For that , I am unable to figure out how to write its custom dataset class for 100 target columns</p>
<p>e.g. if its like 5 - 10 target classes I can write like this --&gt;</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset
import torch

class for_5_target_columns(Dataset):
    def __init__(self, tabular_data , is_valid):
        self.tabular_data = tabular_data
        self.1st_target_value = tabular_data.1st_target_value.values
        self.2nd_target_value = tabular_data.2nd_target_value.values
        self.3rd_target_value = tabular_data.3rd_target_value.values
        self.4th_target_value= tabular_data.4th_target_value.values
        self.5th_target_value = tabular_data.5th_target_value.values
        
    def __len__(self):
        return len(self.tabular_data)
    
    def __getitem__(self, index):
        tabular_data = self.tabular_data.iloc[:,:]
        
        X = tabular_data[training_input.columns()] #training_input.columns() represents column names of X_train
        X = X.values[index]
       
        return {
            'tabular_data' : torch.tensor(X, dtype = torch.float) , 
            '1st_target_value' : torch.tensor(self.1st_target_value[index], dtype = torch.float), 
            '2nd_target_value' : torch.tensor(self.2nd_target_value[index], dtype = torch.float), 
            '3rd_target_value' : torch.tensor(self.3rd_target_value[index], dtype = torch.float),  
            '4th_target_value' : torch.tensor(self.4th_target_value[index], dtype = torch.float),  
            '5th_target_value' : torch.tensor(self.5th_target_value[index], dtype = torch.float)
        }
</code></pre>
<p><strong>How to achieve the same for 100 target columns having different column names ?</strong></p>

username: IamSparky
cooked: <p>this thread has been not categorized yet , <a class="mention-group" href="/groups/admins">@admins</a> plz hel me to  categorize it , so  that It can reach out to required people which will eventually help me in getting solution asap.</p>

username: ptrblck
cooked: <p>Which category would you like to use for this topic?</p>

username: IamSparky
cooked: <p>Last few days I was googling about this issue which I created here , finally <a href="https://discuss.pytorch.org/t/multi-label-classification-in-pytorch/905/45">this post</a> helped me in resolving my problem . Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
How to find individual class Accuracy

username: RajAt_SiNha
cooked: <p>I am using Transfer Learning for Classification of my Dataset.</p>
<p>How to calculate Classification accuracy of each class?</p>

username: ice
cooked: <p>I would just have an array <code>correct</code> filled with zeros and size of number of total classes. Then I would classify data point <code>j</code>, if it matches the target <code>label[j]</code> then just increment the array with that class index, <code>correct[j] += 1</code>. If <code>num_class</code> is an array that contains the number of points for each class <code>c</code>, then the accuarcy is <code>correct[c] / num_class[c]</code>.</p>

username: Mona_Jalal
cooked: <p>Answer given by  <a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a lot!</p>
<pre><code class="lang-auto">nb_classes = 9

confusion_matrix = torch.zeros(nb_classes, nb_classes)
with torch.no_grad():
    for i, (inputs, classes) in enumerate(dataloaders['val']):
        inputs = inputs.to(device)
        classes = classes.to(device)
        outputs = model_ft(inputs)
        _, preds = torch.max(outputs, 1)
        for t, p in zip(classes.view(-1), preds.view(-1)):
                confusion_matrix[t.long(), p.long()] += 1

print(confusion_matrix)
</code></pre>
<p>To get the per-class accuracy:</p>
<pre><code class="lang-auto">print(confusion_matrix.diag()/confusion_matrix.sum(1))
</code></pre>

username: sherlock
cooked: <p>how to calculate overall precision and recall here</p>

username: Mona_Jalal
cooked: <p>Yes, you should calculate the accuracy on your test images. I also suggest you create stratified 5 or 10 fold experiment.</p>

username: KanZa
cooked: <p>Thank you so much Mona for your reply.</p>

username: SebastianRiechert
cooked: <p>This doesn’t calculate accuracy. The true negatives are missing from the numerator of your fraction:</p>
<pre><code class="lang-auto">confusion_matrix.diag()/confusion_matrix.sum(1)
</code></pre>
<p>You are either calculating precision or recall. I don’t know which of the two, because I can’t tell which of the axis is prediction and which is ground truth.</p>

username: Moein_Shariatnia
cooked: <p>Really nice! Thanks…</p>
How to mask a sequence with randomly placed blocks of random length?

username: greaber
cooked: <p>I’m looking to mask the input to a sequence model with independently randomly placed blocks of random length.  Here is a function prototype with pseudo-code for what I want:</p>
<pre><code class="lang-auto">def mask(x, prob, max_length, batch_dim=0, seq_dim=1, mask_value=0):
    """Returns a new tensor like x but possibly with certain elements masked to mask_value.

    More precisely,
      ret = x.clone()
      for each i &lt; x.size(batch_dim)
        for each j &lt; x.size(seq_dim)
          with probability prob
            l = randint(1, max_length)
            for each index tuple idx of ret
              if idx[batch_dim] == i and j &lt;= idx[seq_dim] &lt; j+l
                ret[idx] = mask_value
      return ret
    """
    pass
</code></pre>
<p>I am looking for a way to do this without actually looping through the tensor, which is at least inelegant and possibly would cause a speed issue for the model (didn’t profile it though).</p>
<p>I’m also open to suggestions of variant masking strategies that might work equally well but be easier to implement if this one is hard.</p>

username: ptrblck
cooked: <p>Your use case sound similar to <a href="https://github.com/pytorch/vision/blob/4521f6d152875974e317fa247a633e9ad1ea05c8/torchvision/transforms/transforms.py#L1389-L1507"><code>torchvision.transforms.RandomErasing</code></a>, which is used to randomly selects rectangle regions in an image and erases its pixels, so maybe you could reuse their approach for your temporal data.</p>

username: greaber
cooked: <p>Thanks.  It looks like this code masks a single rectangle in a single image, which is easy to do.</p>
<p>More specifically:</p>
<p>Looping through the whole sequence to choose points to start blocks at is probably fine if it is done in C++, but it is probably slow in python, so I thought there might be some vectorized alternative.  Another approach would be to draw the number of blocks from the binomial distribution and then choose that number of indices (not sure what function does this) and then loop through these indices, which is probably faster but still not nice to have to do in python.  In either case, the actual masking can be done using array index notation so long as we arrange for the batch and time dimensions to come first (which can be done by transposing, which is pretty fast I guess).</p>
<p>The <code>torch.Transforms</code> code only works on one image at a time; it is meant for data loaders.  That might be OK for my case, but ideally masking would work as a layer.</p>
Batching character-level tensors with token-level tensors

username: JustCunn
cooked: <p>Hi,</p>
<p>I’m trying to train an NLP model that has both a character-level LSTM and a token-level LSTM. My embeddings that result from these two LSTMs are concatenated together and then passed to another LSTM.</p>
<p>Would anyone know if there’s any efficient way to batch the training examples so that each batch iterator have the same sentences and words in the sample place?</p>
<p>Thanks in advance</p>
How to get features out of timeseries data

username: granth_jain
cooked: <p>Hi,<br>
I have many timeseries of varying crest and troughs amplitudes, and I want to get features out of it from a window of size 50.</p>
<p>I am sampling a continuous batch of 500 timesteps from the timeseries and training them.<br>
Since all the series have different amplitudes, to avoid scaling I am using percentage change to scale the data, and then taking window of size 50.</p>
<p>Now, is it ok to apply nn.conv1d on this percentage window to get features from the window.</p>
<p>Is there a better way I can features out of window without using scaling.</p>
<p>Below is my current implementation.</p>
<h1><a name="data-is-a-dataframe-1" class="anchor" href="#data-is-a-dataframe-1"></a>data is a dataframe</h1>
<blockquote>
<pre><code>data = data[['Value']]
data['ret'] = data['Value'].pct_change()
data = data.dropna().reset_index()[['Value','ret']]
data = torch.FloatTensor(data.values)
</code></pre>
</blockquote>
<p>and now I take window of size 50 , and calculate the reward based on ‘Value’ , but pass ‘ret’ in Neural network to get features.</p>