akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
libTorch C++ model not updating

username: cdcdcd
cooked: <p>I have done a very straight forward transpiling of a PyTorch implementation I had into a C++ libTorch project. Everything seems to work fine. The forward modelling is running as expected but the model does not seemingly update any part of the model using “loss.backward()”. The offending code:</p>
<blockquote>
<p>for (auto&amp; batch : data_loader) {<br>
auto data = batch.data.to(device), targets = batch.target.to(device);</p>
<pre><code>    auto output = model.forward(data);

    auto loss = torch::binary_cross_entropy_with_logits(output, targets);

    optimizer.zero_grad();
    loss.backward();
    optimizer.step();
    cout &lt;&lt; loss.item() &lt;&lt; std::endl;
}
</code></pre>
</blockquote>
<p>As I say everything seems to be working as is but the backward propagation isn’t either happening within the batch loop (or epoch loop). I have tried different learning rates using Adam optimisation.</p>
<p>Any help would be much appreciated.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Try to check the gradients after the <code>backward()</code> call and make sure the gradients are calculated and check the magnitude of the values.<br>
If you are (accidentally) detaching the computation graph, no gradients would be calculated. On the other hand, even if gradients are calculated they might be tiny and thus it might look as if no parameters are updated.</p>

username: cdcdcd
cooked: <p>Cheers I was “calling” into a module defined outside the main model structure. When a registered all my modules within the main model definition all worked fine.</p>
CNN for face recognition created for microcontroller?

username: Aduka_27
cooked: <p>Hello everyone,</p>
<p>I want to build CNN that can enable face recognition and deploy it on the microcontroller. I am using the board runs on MicroPython and based on that I will need to find a way how to convert CNN into some Python code that can be integrated into my MCU. Does anyone know how can I solve this issue? Any kind of help is welcome! Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to properly use affine_grid for batch tensor?

username: mzimmerman
cooked: <p>I was following this <a href="https://brsoff.github.io/tutorials/intermediate/spatial_transformer_tutorial.html#depicting-spatial-transformer-networks" rel="noopener nofollow ugc">official (I think?) tutorial</a> on Spatial Transformer Network, but the <code>affine_grid</code> seems to be error-nous. I get error saying</p>
<pre><code class="lang-auto">RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [10, 3] but got: [4000, 3].
</code></pre>
<p>on the line</p>
<pre><code class="lang-auto">grid = F.affine_grid(theta, x.size())
</code></pre>
<p>by simply running:</p>
<pre><code class="lang-auto">from torchinfo import summary
model = Net()
summary(model, input_size=(10, 1, 256, 256))
</code></pre>
<p>What’s the catch here? Thanks in advance.</p>
Custom Loss Function doesn't work

username: TeDataPro
cooked: <p>Hey ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Here’s a description of my problem:<br>
<strong>I want to create a Custom Loss to be able to take into account only some predictions in the loss calculation. But I have a problem with the backward function which doesn’t work and prints:</strong></p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Here is the code:</p>
<pre><code class="lang-auto">def my_custom_loss(output, target):
    # Transform tensor to np.array
    sub_labels = np.array([item for sublist in target.tolist() for item in sublist])
    sub_outputs = np.array([item for sublist in output.tolist() for item in sublist])

    # Get index of list of labels with '-1'
    ind = [index for index,value in enumerate(sub_labels) if value != -1]

    # Remove elements with '-1'
    sub_labels = torch.tensor(sub_labels[ind], dtype=torch.float64)
    sub_outputs = torch.tensor(sub_outputs[ind], dtype=torch.float64)

    # Compute the loss
    the_loss = F.binary_cross_entropy_with_logits(sub_outputs, sub_labels)

    return the_loss
</code></pre>
<p>Do you know what is the problem with my custom loss ?<br>
Here’s what I have already tried:</p>
<ul>
<li>I tried to <strong>remove transformations made to tensor</strong> (ie: Create the same function which only return the loss of <em>output</em> and <em>target</em>. This makes the “<em>loss.backward()</em>” work but it’s not the custom function I want to code.</li>
<li>I tried to to add “<strong>the_loss.requires_grad = True</strong>” in “<em>my_custom_loss</em>” before the return, it makes the “<em>loss.backward()</em>” work but my model doesn’t learn at all (loss doesn’t change with epoch).</li>
</ul>
<p>I wish I was clear, if you want more details you can ask me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thank you and have a nice day !</p>

username: ptrblck
cooked: <p>In your custom loss function you are detaching <code>output</code> from the computation graph as you are using numpy operations and are then re-creating the tensors.<br>
You would need to either use PyTorch tensors in all operations so that Autograd can track these ops or you could alternatively implement a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/polynomial_custom_function.html">here</a> in case you really need to use a 3rd party library (such as numpy).</p>
TypeError: Input type float32 is not supported from transform.toPilImage

username: kieutrongthien
cooked: <p>Hey guy! i have a problem when i trying to transform image and i’m stucking here.</p>
<p>First parse image:</p>
<pre><code class="lang-auto">parse = Image.open(parse_obj)
parse = np.array(parse)
parse_channel = 20
parse_emb = []

for i in range(parse_channel):
    parse_emb.append((parse == i).astype(np.float32).tolist())

input_image = np.array(parse_emb).astype(np.float32)
</code></pre>
<p>Second transform image</p>
<pre><code class="lang-auto">transform['1'] = transforms.Compose([
        transforms.ToPILImage(),
        transforms.RandomAffine(degrees=10, translate=(
            0.1, 0.1), scale=(0.8, 1.2), shear=20),
        transforms.ToTensor()])
])

transform['2'] = transforms.Compose([
        transforms.ToTensor()])
])

manualSeed = random.randint(1, 10000)
random.seed(manualSeed)
torch.manual_seed(manualSeed)

num_channel_image = input_image.shape[0]
tform_input_image_np = np.zeros(
shape=input_image.shape, dtype=input_image.dtype)
  
for i in range(num_channel_image):
    if i != 1 and i != 2 and i != 4 and i != 5 and i != 13:
        # if i != 0 and i != 1 and i != 2 and i != 4 and i != 13:
        tform_input_image_np[i] = transforms['1'](input_image[i])  //Error from here
    else:
        tform_input_image_np[i] = transforms['2'](input_image[i])
</code></pre>
<blockquote>
<p>TypeError: Input type float32 is not supported</p>
</blockquote>
<p>But if i try to convert numpy array to uint8, it’ll show another errror:</p>
<blockquote>
<p>ValueError: could not broadcast input array from shape (3,1101,750) into shape (1101,750,3)</p>
</blockquote>
<p>How i can fix this <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> ?</p>
About return None in torch.autograd.Function's backward in C++ API(libtorch)?

username: babuya
cooked: <p>In python, if I want to realize a custom ops which is extended by autograd.Function, I may code as following. The idx’s grad is not need for updating.</p>
<pre><code class="lang-auto">class MyOps(torch.autograd.Function):
    @staticmethod
    def forward(ctx, features: torch.Tensor, idx: torch.Tensor)  -&gt; torch.Tensor:
        # some operations, and output is a Tensor
        ......
        return output
    def backward(ctx, grad_out) :
        # some operations, and idx's grad is not needed for updating.
        ......
        return features_grad, None
</code></pre>
<p>In C++ API(libotrch), I code as follwing:</p>
<pre><code class="lang-auto">using namespace torch::autograd;

class MyOps : public Function&lt;MyOps&gt; {
    public:
    static torch::Tensor forward(AutogradContext *ctx, torch::Tensor features, torch::Tensor idx) {
        # some operations, and output is a Tensor
        return output;
    }
}
</code></pre>
<p>The problem is, I don’t know how to return None and define the return type in backward.<br>
I have read the C++ API and find a value named c10::nullopt. Is that equal to None in Python?<br>
Could anybody give me a backward example?<br>
Thanks.</p>

username: ptrblck
cooked: <p>I think returning a <code>torch::autograd::variable_list</code> object with empty tensors for the <code>None</code> equivalents could work as seen in <a href="https://github.com/pytorch/pytorch/blob/2e2cfa2ef77ff95741f7ebc21a24846567d8630b/torch/csrc/lazy/ts_backend/ts_autograd_functions.cpp#L9-L52">this example</a>.</p>

username: babuya
cooked: <p>Thank you very much!</p>
Access to weights after quantization

username: mkcho
cooked: <p>Hi there! Just wondering how to access the weight parameter of a quantized NN model which is loaded using torch.jit.load().<br>
I already read this post (<a href="https://github.com/pytorch/pytorch/issues/33013" class="inline-onebox" rel="noopener nofollow ugc">Acces to weights after quantization · Issue #33013 · pytorch/pytorch · GitHub</a>), but I’d like to know when the model is saved by using torch.jit.save()</p>
<p>I tried to use the following code; however, the results show only scale and zero_point, not weight and bias.<br>
q = torch.jit.load(‘xxxxxxx.pt’)<br>
q.state_dict()</p>

username: ddang
cooked: <p>Can you provide a minimal reproducible example so I can take a look?</p>
Creating simple NN for

username: Kirill_Radcenko
cooked: <p>Hi, I’m a total beginner in PyTorch and NN.</p>
<p>I am trying to create a simple Sentiment Analysis NN with 3 classes (negative, positive, neutral) for my school project, but I am still getting errors.</p>
<p>Recently I got this error:<br>
<strong>RuntimeError: Given groups=1, weight of size [50, 1, 2, 50], expected input[1, 64, 1, 50] to have 1 channels, but got 64 channels instead</strong></p>
<p>and I do not have a clue on how to solve it.</p>
<p>This is my <strong>Convolutional Neural Network</strong>, I forked it from this <a href="https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/4%20-%20Convolutional%20Sentiment%20Analysis.ipynb" rel="noopener nofollow ugc">Github project</a></p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, filter_sizes, output_dim, 
                 dropout, pad_idx):
        
        super().__init__()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, 
                                              out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):
        #text = [batch size, sent len]
        
        embedded = self.embedding(text)
                
        #embedded = [batch size, sent len, emb dim]
        
        embedded = embedded.unsqueeze(1)
        
        #embedded = [batch size, 1, sent len, emb dim]
        
        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
            
        #conv_n = [batch size, n_filters, sent len - filter_sizes[n]]
        
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]
        
        #pooled_n = [batch size, n_filters]
        
        cat = self.dropout(torch.cat(pooled, dim = 1))

        #cat = [batch size, n_filters * len(filter_sizes)]
            
        return self.fc(cat)
</code></pre>
<p>Can anybody help me? I would be really grateful</p>

username: ptrblck
cooked: <p>Could you post the model initialization as well as random input shapes to reproduce the issue?<br>
Based on the error message it seems as if</p>
<pre><code class="lang-python">conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
</code></pre>
<p>would be failing, but the channel dimension of <code>embedded</code> should be set to <code>1</code> via the <code>unsqueeze(1)</code> and I can’t see the obvious issue so far.</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x16 and 1024x64)

username: yuxin_yang
cooked: <p>import torchvision<br>
from torch.utils.data import DataLoader<br>
train_data = torchvision.datasets.CIFAR10(root="…/data", train=True, transform=torchvision.transforms.ToTensor(),<br>
download=True)<br>
test_data = torchvision.datasets.CIFAR10(root="…/data", train=False, transform=torchvision.transforms.ToTensor(),<br>
download=True)</p>
<p>train_data_size = len(train_data)<br>
test_data_size = len(test_data)</p>
<p>train_dataloader = DataLoader(train_data, batch_size=64)<br>
test_dataloader = DataLoader(test_data, batch_size=64)</p>
<p>class Tudui(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Tudui, self).<strong>init</strong>()<br>
self.model = nn.Sequential(<br>
nn.Conv2d(3, 32, 5, 1, 2),<br>
nn.MaxPool2d(2),<br>
nn.Conv2d(32, 32, 5, 1, 2),<br>
nn.MaxPool2d(2),<br>
nn.Conv2d(32, 64, 5, 1, 2),<br>
nn.MaxPool2d(2),<br>
nn.Flatten(),<br>
nn.Linear(64<em>4</em>4, 64),<br>
nn.Linear(64, 10)<br>
)</p>
<pre><code>def forward(self, x):
    x = self.model(x)
    return x
</code></pre>
<p>tudui = Tudui()</p>
<p>loss_fn = nn.CrossEntropyLoss()<br>
learning_rate = 1e-2<br>
optimizer = torch.optim.SGD(tudui.parameters(), lr=learning_rate)</p>
<p>total_train_step = 0</p>
<p>total_test_step = 0</p>
<p>epoch = 10</p>
<p>for i in range(epoch):<br>
print(f"第{format(i+1)}轮训练开始")</p>
<pre><code>for data in test_data:
    imgs, targets = data
    output = tudui(imgs)
    loss = loss_fn(output, targets)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    total_train_step = total_train_step + 1
    print(f"训练次数format{(total_train_step)},loss:{format(loss)}")</code></pre>

username: ptrblck
cooked: <p>The error is most likely raised in <code>nn.Linear(64*4*4, 64)</code> as the input activation seems to use <code>64</code> features instead of the expected <code>1024</code>.<br>
Change the <code>in_features</code> to <code>64</code> and it should work.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>
ADAM optimizer not applying gradient correctly

username: amink
cooked: <p>I’m using the ADAM optimizer for a SNN simple XOR test appl, although the   “loss.backward()” computes valid gradient values, the “optimizer.step()” is not applying the expected parameter updates of “gradient * Learning_rate”. The updated values are much larger than the gradient value by itself. What I see is for the 1st batch update, is 0.1 (the learning rate) for all non-zero gradients. The following updates are larger than the gradient values and sometimes are the opposite direction. Has anyone else seen this behavior ?</p>
<p>Thanks,<br>
Alan</p>

username: ptrblck
cooked: <p><code>Adam</code> uses running internal states and does not use the plain SGD update as <code>gradient * learning_rate</code>.<br>
The update rule is explained in the <a href="https://pytorch.org/docs/stable/generated/torch.optim.Adam.html">docs</a> as well as the <a href="https://arxiv.org/abs/1412.6980">paper</a>.</p>
Is the dataloader parallelizing __getitem__ or batch sampling?

username: Felipe_Mello
cooked: <p>Hi,</p>
<p>I tried to google the answer, but couldn’t find it.<br>
i) Is each worker in the the dataloader calling __ getitem __ <strong>within</strong> the same batch, and then collating it,<br>
ii) or is each worker loading and collating each batch <strong>independently</strong>?</p>
<p>Imagine this scenario:<br>
16 batches per epoch, each with 100 examples.<br>
24 workers available.<br>
Will 8 workers be iddle if I set num_workers = 24?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>The second case is currently used, i.e. each worker creates a full batch and pushes it to the queue.<br>
Once it’s ready it’ll start creating the new batch until <code>prefetch_factor*num_workers</code> batches are available.</p>
How to calculate MSE loss with respect to input gradient?

username: altsaad
cooked: <p>What is the best way to optimize \theta for the the following loss function:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17f9c2341d21eb0c4a9808b5ed14ca2ec48df09a.png" alt="Screenshot from 2022-04-05 15-08-29" data-base62-sha1="3q65p4UOFATZh1CxHTClJZHzwi6" width="249" height="43"></p>
<p>I tried the following but does not work:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
Input.requires_grad_()
Output = Model(Input)
Output_max = Output[0,target1]
Output_max.backward(retain_graph = True)
loss = criterion(Input.grad, target)
loss.backward()
optimizer.step()
</code></pre>
<p>I get the following error for loss.backword()</p>
<pre><code class="lang-auto">element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
Ignoring large losses when backpropagating

username: Andrei_Cristea
cooked: <p>I’m working on an image classification problem. I’m using a collection of transforms that occasionally distort certain input images beyond recognition (e.g. randomly cropping a nondescript portion of the input) and training on these transformed inputs leads to an erroneous learning step (it essentially updates the parameters in a random direction). More generally, you can think of this as having noisy labels, in a way that’s difficult to correct.</p>
<p>I’d like to deal with this by essentially ignoring (for the purposes of backprop) datapoints with large losses from within each minibatch after a certain training step. Acceptable solutions would include ignoring losses above / capping losses at a certain cutoff or ignoring the largest K losses in the minibatch.</p>
<p>Does anyone have any suggestions about how best to implement this?</p>
<p>Much appreciated.</p>

username: InnovArul
cooked: <p>Here are my attempts for your use case.</p>
<p>Hard-threshold capping:</p>
<pre><code class="lang-auto">import torch
loss = torch.rand(6, requires_grad=True) # lets say we have a loss tensor
torch.clip(loss, max=0.5).sum().backward() # capping loss to 0.5

# loss.grad contains 0 for loss values greater than 0.5
</code></pre>
<p>Ignoring top-k losses:</p>
<pre><code class="lang-auto">import torch
loss = torch.rand(6, requires_grad=True)
k = 3
sorted_loss, indices = torch.sort(loss)
sorted_loss[:-k].sum().backward()

# loss.grad contains 0 for top-k losses
</code></pre>
Loss propagation of outputs of two mdoels

username: abbasi7
cooked: <p>Hello</p>
<p>I am traiing an unsupervised network. I have two modes and two outputs in supervise part of network.</p>
<p>output1=model1<br>
Loss1<br>
output2=model2<br>
Loss2</p>
<p>For the un-supervised i am using the output of above two models, and plug them in an equation and estimate output3, and find its loss3</p>
<p>output3= f(output1,output2)<br>
Loss3</p>
<p>Now my question is how to backpropagate loss3, which is based on the above two models.</p>
<p>thanks</p>

username: mxahan
cooked: <p>Is your case similar to <a href="https://stackoverflow.com/questions/53994625/how-can-i-process-multi-loss-in-pytorch" rel="noopener nofollow ugc">this</a>?</p>

username: abbasi7
cooked: <p>Thanks for the response. But I do not think is quite what I like to have. I am more interested in how loss3(in my case), is backpropagated to two models. The third output of the function(not the model), takes in the 2 outputs of two models.</p>
<p>What should I do here:</p>
<ol>
<li>Final Loss=(Loss1+Loss3)+(Loss2+Loss3)<br>
2.Final Loss=(Loss1+Loss3)+Loss2</li>
<li>Final Loss=(Loss2+Loss3)+Loss1</li>
<li>Final Loss=Loss1+Loss2+Loss3(using optim of Loss1 or Loss2)</li>
</ol>
<p>thanks</p>

username: mxahan
cooked: <p>Oh, thanks for the clarification.</p>
<p>Can your problem be thoughts from a metric learning perspective? e.g. Let’s assume, there is a text and corresponding image.  We have a CNN model to encode images and we have a Language model for the texts. And our goal maybe is to maximize the model’s agreement in the final layers.</p>
<p>In that case, assuming using a euclidean distance (L2) loss between the model’s output. The CNN model will see the language model output as a constant target and update the weights as if it is trying to get close to the LM outputs. The vice-versa is true for the LM too.</p>
<p>It is generalizable for any other loss, instead of providing a fixed target, we are generating a target from another model. The baseline model has no idea where is the output coming from.</p>

username: abbasi7
cooked: <p>Thanks for your response. In my case the target is not generated from the  model. The target for the supervised part of the code are fixed and known.</p>
<p>For the unsupervised part(where we do not have the target to estimate loss directly-thus using unsupervised), we plug in the outputs(from model 1 and model 2) in a formula. If the outputs1 and outputs2 are correct it should give me correctoutput3, for which i have target to estimate loss.</p>
<p>Actually I am estimating output1 , that satisfies the loss for output3. But output3 according to the formula depends on otput1 and output2. I hope I was able to clarify.</p>

username: mxahan
cooked: <p>It’s a bit confusing for me. Can you please provide more about the optimizers, loss functions, and models?</p>
Backpropogate through the leaf node in make_functional()

username: wasabi
cooked: <p>I’m trying to implement HyperNetwork. The idea is that we have a hypernetwork A and it produces the weight of network B. We use B to perform classification and backpropogate through B to update the weights of A. The issue with vanilla pytorch implementation is that A’s weight is usually nn.Module() or nn.Parameter() and they are leaf nodes. Autodiff will stop at leaf nodes.<br>
It seems to me that make_functional() in functorch is a workaround: it returns a functional version of your network and their params. See the following code:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.nc = 8
        self.conv1 = nn.Conv2d(1, self.nc, (3, 3), bias=False)
        self.conv2 = nn.Conv2d(self.nc, self.nc, (3, 3), bias=False)
        self.conv3 = nn.Conv2d(self.nc, self.nc, (3, 3), bias=False)
        self.fc = nn.Linear(484*self.nc, 10, bias=False)
        nn.init.normal_(self.conv1.weight)
        nn.init.normal_(self.conv2.weight)
        nn.init.normal_(self.conv3.weight)
        nn.init.normal_(self.fc.weight)

    def forward(self, x):
        x = self.conv1(x)/(torch.sqrt(torch.tensor(1*3*3)))
        x = x.relu()
        x = self.conv2(x)/(torch.sqrt(torch.tensor(self.nc*3*3)))
        x = x.relu()
        x = self.conv3(x)/(torch.sqrt(torch.tensor(self.nc*3*3)))
        x = x.flatten(1)
        x = self.fc(x)/(torch.sqrt(torch.tensor(self.nc*484)))
        return x

net = CNN().to(device)
fnet, params = make_functional(net)
x = torch.rand(1,1,28,28).cuda()
A = nn.Linear(1, 72)
temp = A(torch.rand(1))
params[0].data = A(torch.rand(1)).view(8,1,3,3).cuda()
o=fnet(params, x)
o.sum().backward()
print(params[0].grad)
print(A.weight.grad)
</code></pre>
<p>However, A.weight.grad is still None. Is this expected?</p>
Almost Convolution Computation

username: FI_PytorchUser
cooked: <p>Hi there,</p>
<p>I’m new to PyTorch (and deep learning) and I need to do a  computation that’s not straight forward (to me) to do with provided functions.</p>
<p>It would be easy to do using “for” but clearly I should avoid that.</p>
<p>The computation is:<br>
For tensor A of dimension (20, 40, 256) and tensor B of dimension (20, 40, 256X300) do an “almost convolution” that results in a tensor C of dimension (20, 40, 300).</p>
<p>Here “almost convolution” means that for each element in the feature map of dim [20, 40] the convolution is done for channels, so the 256 channels from A get filtered through the 300 filters of 256 channels. But at the end, instead of adding all of the results to get a (1,1,300) tensor, the results stay separate for each of the (20,40) elements of the feature maps to get a (20, 40, 300) tensor.</p>
<p>Backpropagation should also work when this computation is part of the computation of the loss function.</p>

username: Matias_Vasquez
cooked: <p>Hi,</p>
<p>You can try something like this:</p>
<pre><code class="lang-python">A = torch.rand(20, 40, 256)
B = torch.rand(20, 40, 256*300)

C = torch.einsum("ijk, ijkl -&gt; ijl", [A, B.view(20, 40, 256, 300)])

# This is a more generic way
#C = torch.einsum("ijk, ijkl -&gt; ijl", [A, B.view([*A.shape, -1])])

print(C.shape)
</code></pre>
<pre><code class="lang-auto"># Output: 
torch.Size([20, 40, 300])
</code></pre>
<p>Please let me know if this is not the behavior that you expected.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

username: FI_PytorchUser
cooked: <p>Thank you!</p>
<p>This is the first time I heard of torch.einsum. I worry a bit about how optimal it is, but it should certainly be better than for loops.</p>
<p>I’ll also look into <a href="https://optimized-einsum.readthedocs.io/en/stable/" class="inline-onebox" rel="noopener nofollow ugc">opt_einsum — opt_einsum v3.3.0 documentation</a></p>
<p>Thank you again.</p>
Is better way to perform gather behavior?

username: John1231983
cooked: <p>I have an original code as using torch.gather</p>
<pre><code class="lang-auto">#outputs_x size of  68x64 and max_ids size of 68x1     
outputs_x_select = torch.gather(outputs_x, 1, max_ids) #68x1    
outputs_x_select = outputs_x_select.squeeze(1) # 68
</code></pre>
<p>I want to find an alternative way that does not use torch.gather function. Currently, I found a way that use index_select such as</p>
<pre><code class="lang-auto">#outputs_x size of  68x64 and max_ids size of 68  
max_ids = max_ids.squeeze()   
outputs_x_select = torch.index_select(outputs_x, 1, max_ids)
outputs_x_select = outputs_x_select[range(len(max_ids)), range(len(max_ids))]       
</code></pre>
<p>The behavior of index_select is that we will select a column/row for each index in indices. It is memory consuming than gather. Do we have any better way than my way to perform gather behavior?</p>
When using torch.distributed.algorithms.join, I got the `gloo::EnforceNotMet` error

username: zheng-da
cooked: <p>When I split the training set, the training data cannot be split evenly among all trainers. Without using <code>torch.distributed.algorithms.join</code>, the training script hangs. But when I start to use <code>torch.distributed.algorithms.join</code>, I got another error.</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'gloo::EnforceNotMet'
  what():  [enforce fail at ../third_party/gloo/gloo/transport/tcp/pair.cc:510] op.preamble.length &lt;= op.nbytes. 24640 vs 4
Traceback (most recent call last):
  File "/home/dzzhen/.local/lib/python3.9/site-packages/torch/distributed/algorithms/join.py", line 274, in __exit__
  File "/home/dzzhen/m5-gnn/python/m5gnn/model/rgcn_node_base.py", line 179, in fit
    loss.backward()
  File "/home/dzzhen/.local/lib/python3.9/site-packages/torch/_tensor.py", line 307, in backward
    join_hook.main_hook()
  File "/home/dzzhen/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 193, in main_hook
    ddp._match_all_reduce_for_bwd_pass()
  File "/home/dzzhen/.local/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1070, in _match_all_reduce_for_bwd_pass
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/dzzhen/.local/lib/python3.9/site-packages/torch/autograd/__init__.py", line 154, in backward
    work.wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:589] Read error [10.2.23.254]:1439: Connection reset by peer
    Variable._execution_engine.run_backward(
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:589] Read error [10.2.23.254]:49550: Connection reset by peer
</code></pre>
<p>My code is a bit complex. It has multiple <code>nn.module</code>. Here is a snippet of the code.</p>
<pre><code class="lang-auto">            with Join([model, embed_layer] + list(bert_model.values())):
                for i, (input_nodes, seeds, blocks) in enumerate(loader):
                    total_steps += 1
                    blocks = [blk.to(device) for blk in blocks]
</code></pre>
<p>Is the error related to how many modules are added to <code>Join</code>? I tried different options and always got the same error.</p>

username: H-Huang
cooked: <p>cc <a class="mention" href="/u/awgu">@awgu</a> for help with join context manager</p>

username: agu
cooked: <p>Is <code>model</code> a <code>DistributedDataParallel</code> instance? Do the modules <code>embed_layer</code> and <code>list(bert_model.values())</code> include any collective communications?</p>
<p>If the answers are yes and no, respectively, then can you try <code>with Join([model]):</code> (i.e. do not pass in the other modules)?</p>
How to fix finetuned model?

username: Klink
cooked: <p>I finetuned gpt-j model by fp16.</p>
<p>After the trainning, the model makes abnormal memory usage.</p>
<p>Long tokens make much memory use.</p>
<p>Short tokens need double memory usage too.</p>
<p>So I extracted the files before and after, But they aren’t have big difference.</p>
<p>it was little volume, few MB.</p>
<p>I think it needs some converting, or edit options for light model.</p>
<p>How can I find cause of problems?</p>
Adjacency Matrix to Edge Index (Solution)

username: Andre_Amaral_IST
cooked: <p>Hey,</p>
<p>For those who have this question, here you have a way to solve it!</p>
<pre><code class="lang-auto">adj_t = torch.tensor([[0,1,0,0],[1,0,0,0], [1,0,0,1], [0,0,1,0]])
edge_index = adj_t.nonzero().t().contiguous()

&gt;&gt;&gt; tensor([[0, 1, 2, 2, 3],
            [1, 0, 0, 3, 2]])
</code></pre>
<p>Thanks for your attention,<br>
André Amaral</p>
Getting: RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead

username: hilde
cooked: <p>Hi I get this runtime error and I am not sure how to solve the problem. I have tried a few different things but it is not working.  I am using PINN to solve a hyperbolic PDE Kvu</p>
<p>I have a function:     def eps1(self, x):<br>
return np.sin(x)<br>
That i derive by:       def eps1_grad(self, points):<br>
p = points.clone()<br>
p.requires_grad_(True)<br>
eps1xi = self.eps1(p)<br>
eps1_xi = autograd.grad(outputs=eps1xi, inputs=p,<br>
grad_outputs=torch.ones_like(eps1xi), create_graph=True)[0]<br>
return eps1_xi</p>
<p>Then I use the derived function in:<br>
def loss_Kvu(self, points):<br>
p = points.clone()<br>
p.requires_grad_(True)<br>
KvuKvv = self.forward(p)<br>
Kvu = KvuKvv[:, [0]]<br>
Kvv = KvuKvv[:, [1]]<br>
Kvu_grad = <br>
autograd.grad(outputs=Kvu, inputs=p, grad_outputs=torch.ones_like(Kvu),<br>
create_graph=True)[0]<br>
self.Kvu_x = Kvu_grad[:, [0]]<br>
self.Kvu_xi = Kvu_grad[:, [1]]<br>
x = points[:, [0]]<br>
xi = points[:, [1]]<br>
eps1_xi = self.eps1_grad(xi)<br>
f1 = self.eps2(x) * self.Kvu_x - self.eps1(xi) * self.Kvu_xi - eps1_xi * Kvu -<br>
self.c2(xi) * Kvv<br>
loss_f1 = self.loss_function(f1, f1_hat)<br>
return loss_f1</p>
<p>When eps1 = np.sin(x) I get the RuntimeError: Can’t call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead. However when eps1 = x**2, there is no issue.<br>
I cannot understand when the problem arises, is it already in the object where the function is defined or is it when I am trying to derive the function? Any suggestions are appreciated.</p>

username: AlphaBetaGamma96
cooked: <p>Numpy is a different library to PyTorch and hence it can’t track operation outside PyTorch. The operation <code>**</code> is just the exponent which PyTorch understands. If you want to track operations such that you can calculate a derivative you need to use an equivalent PyTorch operation (in your case replace <code>np.sin(x)</code> with <code>torch.sin(x)</code>)</p>
<p>If you want to use a different library (say Numpy or SciPy) you need to remove the Tensor and convert it to Numpy (via <code>.detach().numpy()</code>)</p>
Keeping small batch_size while using GPU memory more efficiently?

username: jayz
cooked: <p>Hey,</p>
<p>I found that a batch_size = 32 during training is a good value for updating the model, but it largely underuses the available RAM in my GPU, as seen in this nvidia smi output:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 511.79       Driver Version: 511.79       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   70C    P8     3W /  N/A |   1284MiB /  4096MiB |     18%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
</code></pre>
<p>Just about 30% of the memory is used. Also, if I understand correctly, it seems just 18% of the available GPU power are used? This seems like a batch_size = 32 is very inefficient in terms of training time. Nevertheless, increasing the batch_size much further would also not be good. Are there typical things one can do in this situation to increase the efficiency, like some form of parallelization, etc?</p>
<p>Thanks!</p>
<p>Best, JZ</p>
Finding the derivative of a function belonging to an equation, that is to be solved by a neural network

username: hilde
cooked: <p>Hi, I am using PINN(Raissi et. al) to solve a set of equations.<br>
The equations consist of functions and the derivative of the functions. Like this:<br>
PDE1 = func1 + func2’<br>
PDE2 = func1’ + func3<br>
I am wondering if I can use autograd to do the derivation of the functions, and at the same time use autograd to find the gradients of the network. The reason for wanting to use autograd for the derivation of the functions is because I want to have the option to change the function e.g. it could be func1 = x<strong>2 in certain cases, but in other cases it could be e.g. func1 = x</strong>3. I dont have a lot of experience with programming and pytorch, so this might seem obvious to some, however I cannot seem to figure out how to do this.</p>

username: suraj.pt
cooked: <p>Hi <a class="mention" href="/u/hilde">@hilde</a>, can you share an example of what you’re trying to do? Please include the input variables and the outputs you expect. This will help us answer your question. Thanks!</p>

username: hilde
cooked: <p>I am such a noob. I was able to solve the problems I had. It helped thinking about what input and output to expect.</p>

username: suraj.pt
cooked: <p>I’m glad you solved it! I find reframing my question in a minimal example very helpful too.<br>
If you don’t mind sharing your solution, it might help someone in the future <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: hilde
cooked: <p>I used this:</p>
<p>I have a function:     def eps1(self, x):<br>
return x**2<br>
That i derive by:       def eps1_grad(self, points):<br>
p = points.clone()<br>
p.requires_grad_(True)<br>
eps1xi = self.eps1(p)<br>
eps1_xi = autograd.grad(outputs=eps1xi, inputs=p,<br>
grad_outputs=torch.ones_like(eps1xi), create_graph=True)[0]<br>
return eps1_xi</p>
<p>Then I use the derived function in:<br>
def loss_Kvu(self, points):<br>
p = points.clone()<br>
p.requires_grad_(True)<br>
KvuKvv = self.forward(p)<br>
Kvu = KvuKvv[:, [0]]<br>
Kvv = KvuKvv[:, [1]]<br>
Kvu_grad = <br>
autograd.grad(outputs=Kvu, inputs=p, grad_outputs=torch.ones_like(Kvu),<br>
create_graph=True)[0]<br>
self.Kvu_x = Kvu_grad[:, [0]]<br>
self.Kvu_xi = Kvu_grad[:, [1]]<br>
x = points[:, [0]]<br>
xi = points[:, [1]]<br>
eps1_xi = self.eps1_grad(xi)<br>
f1 = self.eps2(x) * self.Kvu_x - self.eps1(xi) * self.Kvu_xi - eps1_xi * Kvu -<br>
self.c2(xi) * Kvv<br>
loss_f1 = self.loss_function(f1, f1_hat)<br>
return loss_f1</p>
<p>Sadly it does not work when eps1 = np.sin(x). Only for function that I can define without the help of numpy.</p>
Dataset Processing not terminating correctly

username: jpbohl
cooked: <p>I have created a Pytorch Geometric InMemoryDataset and am running into some problems when loading this dataset. When the data has not been processed yet the Processing does not seem to finish correctly as I am not getting the usual “Done!” message in the terminal, and also my code does not continue as expected but just quits without an error message.<br>
If I then try to rerun the code, pytorch tries to load the processed dataset but throws the following error:<br>
<em>PytorchStreamReader failed reading zip archive: failed finding central directory</em>.</p>
<p>Here is the code for my dataset:</p>
<pre><code>class HurricaneDataset(InMemoryDataset):
    def __init__(self, root, transform=None, pre_transform=None, pre_filter=None):
        super().__init__(root, transform, pre_transform, pre_filter)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self):
        return os.listdir(self.root + "/raw")
    
    @property
    def processed_file_names(self):
        return ["data.pt"]
    
    def download(self):
        pass
    
    def len(self):
        """
        Reads length of .npz archive 
        """
        return len(self.raw_file_names)


    def process(self):
        data_list = [torch.load(self.root + "/raw/" + f) for f in self.raw_file_names]

        if self.pre_filter is not None:
            data_list = [data for data in data_list if self.pre_filter(data)]
        
        if self.pre_transform is not None:
            data_list = [self.pre_transform(data.item()) for data in data_list]

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])</code></pre>
The model.cuda() do not work

username: JansonYoung
cooked: <p>My computer configures GTX1080ti, PyTorch =1.0. The model can work. After changing the GPU to GTX3070ti, I run “torch.cuda.is_available()”, and it return True. But, the running code stops to “model = model.cuda()” for a long time, and not possible to keep going.</p>
<p>I have updated the GPU driver to 497.29, CUDA to 10.0, and reinstalled PyTorch to 1.4.0. But the situation has no changing.</p>

username: ptrblck
cooked: <p>Your Ampere GPU needs CUDA11, so install the latest binaries with the 11.3 or 11.5 CUDA runtime.</p>
Libtorch JNI loading

username: temerick
cooked: <p>This problem is very similar to <a href="https://discuss.pytorch.org/t/cppextension-loading-complains-undefined-symbol/27908">this issue</a>, but in a different context. I’m trying to use libtorch 1.0 on ubuntu in java via JNI. All of my stuff compiles and runs perfectly fine in C++, and there is no problem at compile time in java, but I’m getting the same error message as the above issue:</p>
<pre><code class="lang-auto">undefined symbol: _ZN2at19UndefinedTensorImpl10_singletonE
</code></pre>
<p>My gcc version is 7.3.0.</p>
<p>Since I’m not in python, the fix in the linked issue won’t work for me, so I was hoping for more details about what “import torch” is doing in python to fix this problem.</p>
<p>Thanks!</p>

username: mehmetcalikus
cooked: <p><a class="mention" href="/u/temerick">@temerick</a> I’m having the same problem as you, did you find a solution?</p>
Can Mask rcnn masks have shared pixels?

username: vahvero
cooked: <p>Hi!</p>
<p>I am currently doing a training with mask rcnn model and all output <code>masks</code> seem to be really bad compared to the label scores.</p>
<p>In my training set most of the masks have one annotation inside an another. For example:</p>
<pre><code class="lang-auto">  A = torch.zeros((5, 5)).to(int)
  A[1:4, 1:4] = 1

  B = torch.zeros(
      (
          5,
          5,
      )
  ).to(int)
  B[2, 2] = 1
# Is the correct mask for A == A ^ B
</code></pre>
<p>Masks A and B would share the center pixel.</p>
<p>Is this incorrect or could something else be currently wrong?</p>

username: KFrank
cooked: <p>Hi Vahvero!</p>
<aside class="quote no-group" data-username="vahvero" data-post="1" data-topic="148248" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/5e9695/48.png" class="avatar"> vahvero:</div>
<blockquote>
<p>In my training set most of the masks have one annotation inside an another.</p>
</blockquote>
</aside>
<p>This is perfectly reasonable for Mask R-CNN (although it may indicate<br>
that you are working on a problem that is inherently more difficult to train).</p>
<p>Mask R-CNN performs <em>instance segmentation.</em>  It is conceptually fine<br>
to have instance-1 of class-A  be contained in instance-2 also of class-A.<br>
You could also have instance-1 of class-A be enclosed in instance-1 of<br>
class B.</p>
<p>Mask R-CNN can be applied to such problems (assuming enough training<br>
data of good-enough quality, and so on), although if you told me that such<br>
a use case would tend to be harder to train, I would believe you.</p>
<p>Best.</p>
<p>K. Frank</p>

username: vahvero
cooked: <p>I found the error in my code. I was feeding the model with mask tensor [N, 1, X, Y] in size which is the output shape of the model and not [N, X, Y] that was required as the feed tensor shape.</p>
<p>It seems often peculiar with these models that they often accept oddly shaped inputs. I had a similar error a long time ago when I fed labels in [1, N] tensor and not [N] tensor as was required.</p>
Torch.save() pytorch runtimeerror: could not allocate bytes object!

username: Hcz-up
cooked: <p>During the training of the model, there was no problem with torch.save() for the previous several times, but I suddenly encountered this problem [runtimeerror: could not allocate bytes object!]. What is the reason for this and how to solve it?</p>

username: ptrblck
cooked: <p>Maybe you are running out of memory?<br>
Could you post a minimal, executable code snippet to reproduce this issue, please?</p>
Linux wheels built with MKL DNN?

username: Jakub_Adamczyk
cooked: <p>I tried installing PyTorch 1.11 on Linux Ubuntu 20.04, Python 3.8 with both pip and conda. I get:</p>
<pre><code class="lang-auto">PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.5.2 (Git Hash a9302535553c73243c632ad3c4c80beec3d19a1e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.11.0, USE_CUDA=0, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
</code></pre>
<p>Is there any wheel available with <code>USE_MKLDNN=ON</code>, or do I have to build from source? I don’t need particularly 1.11, earlier versions are also ok.</p>

username: hackcs
cooked: <p>+1, I had a few google searches on this but didn’t find anything useful, is MKLDNN somehow deprecated or we <em>have to</em> build from source now to gain support of MKLDNN?</p>

username: Jakub_Adamczyk
cooked: <p>It is definitely not deprecated, as Intel now ships it under the name oneDNN and develops it quite heavily as far as I know. However, for some reason in PyTorch it is not used in compiled wheels, I had to build from source. Shame.</p>