akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
C10::error "cudnn filters (a.k.a. weights) must be contiguous in memory_format

Timangar: <p>My code works fine on one thread, but as soon as I use multiple threads for inference, the c10 error from the title is thrown from time to time. I have no idea what it means and why my weights would not be contiguous. I am using torch::NoGradGuard during every evaluation.</p>
<p>The net I am using is a fairly standard convnet.</p>

ptrblck: <p>I assume the error is raised <a href="https://github.com/pytorch/pytorch/blob/f6a4c80a5acf7454e90814449c4d6f4b73f7fad9/aten/src/ATen/cudnn/Descriptors.cpp#L132-L136">here</a>, which would point to the input tensor, which isn’t contiguous in memory.<br>
I don’t know how you are using multiple threads but would check if you are passing valid tensors around.</p>

Timangar: <p>I’ve been doing some tests and I think I have been able to resolve the error. What I do now is calling .detach() on the output tensor before using .item&lt;&gt;() on it. This is very weird to me, as I used NoGradGuard AND as torch objects should be thread safe when reading anyways. The input tensors were valid and contiguous, I haven’t found any problems with them. I still don’t exactly know what the problem was, but at least it works now. Thank you for the tips <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to convert a pytorch nn module to float 64

Rami_Ismael: <p>I want a simple technique that will convert a pytorch nn.module to a float 64 model.</p>

ptrblck: <p>To transform all parameters and buffers of a module to <code>float64</code> tensors, use <code>model.double()</code>.</p>

Rami_Ismael: <p>What is the Pytorch buffer?</p>

ptrblck: <p>Buffers are tensors, which are registered to the parent module but don’t require gradients. E.g. the running stats in batchnorm layers are buffers. You can register buffers via <code>self.register_buffer</code> inside an <code>nn.Module</code>.</p>
RuntimeError: Trying to backward through the graph a second time, when we store the variable

Jiyao_Li: <p>Could you tell me why the error occurs here?</p>
<p>import torch</p>
<p>a = torch.rand(3,3, requires_grad=True)</p>
<p>L = []</p>
<p>for i in range(10):</p>
<pre><code>b = a * a

d = b * b

e = d.sum()

L.append(e)

sum(L).backward()</code></pre>

ptrblck: <p>Your code is unfortunately not formatted correctly, but I assume the <code>backward()</code> operation is performed inside the loop. The first <code>backward</code> pass would free the intermediate activations from the forward pass, so that you wouldn’t be able to call <code>backward</code> a second time (if this is needed, use <code>retain_graph=True</code>, but it’s usually not the case). Since you are storing the <code>e</code> tensor in the <code>L</code> <code>list</code> and calculating the <code>sum</code> afterwards, the next <code>backward</code> pass would then try to backpropagate through <em>both</em> <code>e</code> tensors (the one from the current iteration [<code>e1</code>] as well as the one from the previous iteration [<code>e0</code>]). However, since the computation graph from <code>e0</code> is already freed the error is raised.</p>

mMagmer: <p>I don’t know why you wanna do this:</p>
<pre><code class="lang-auto">import torch
a = torch.rand(3,3, requires_grad=True)

L = []
for i in range(10):
    b = a * a
    d = b * b
    e = d.sum()
    L.append(e)
    sum(L).backward()
</code></pre>
<p>the more normal way is to have <code>sum(L).backward()</code> outside of loop.(and i think it really is the case)<br>
But if your intention is to compute grad for what you asked, then you can go around using <code>retain_graph = True</code> with change in how you backward.</p>
<pre><code class="lang-auto">import torch
a = torch.rand(3,3, requires_grad=True)

L = []
for i in range(10):
    b = a * a
    d = b * b
    e = d.sum()
    e.backward(torch.tensor(10-i)) # if you really want to compute 
                                   # grad multiple times,
                                   # i.e. backward for 10 
                                   # times on first item of the list
    L.append(e.item())
sum(L)
# do some thing with a.grad and then
print(a.grad)

a.grad.fill_(0)
</code></pre>
<p>This way is more memory efficient.</p>
Pytorch training loop bug in updating loss

ish_07: <p>This is my training loop and my loss is not updating.</p>
<pre><code class="lang-auto">for epoch_i in range(5):
  total_loss = 0
  print(epoch_i,total_loss)

  model.train()
    
  for step, batch in enumerate(train_data,0):

      train_x, train_y = tuple(t.to(device)for t in batch)

      model.zero_grad()

      logits=model(train_x)

      loss_fn= nn.CrossEntropyLoss()

      #print(train_y.shape,"y",train_x.shape,"x")

      loss= loss_fn(logits,train_y)

      total_loss=loss+total_loss
    
      loss.backward()

      optimizer.step()

</code></pre>
<p>This is my output :<br>
0 0<br>
1 0<br>
2 0<br>
3 0<br>
4 0<br>
The input dimension of my tensor is :<br>
train_x= torch.Size([32, 300])<br>
train_y= [torch.Size([32, 20])</p>
<p>May be I am missing something. Can someone spot the error why the loss is not updating and</p>

mMagmer: <p>move print line to the end of for loop on epoch. this should fix 0 total loss<br>
and you sould change some other lines  for the code to be bug free.</p>
<ol>
<li><code>total_loss =  loss.item() + total_loss</code></li>
<li>move loss_fn definition out of loop body</li>
</ol>
Fast computation with pytorch

Luca_Pamparana: <p>I am doing this numpy computation at the moment which is very, very slow. Something like:</p>
<pre><code class="lang-auto">cutoff = self.theta
X_flat = X.reshape((X.shape[0], X.shape[1] * X.shape[2]))
weightfun = lambda x: 1.0 / np.sum(np.dot(X_flat, x) / np.dot(x, x) &gt; 1 - cutoff)
# This is a very expensive operation. 
N_list = np.array(list(map(weightfun, X_flat)))
</code></pre>
<p>Assuming I can move everything to pytorch tensors and use the GPU, how could one compute a faster version of this.<br>
Currently it computes the dot product between every column vector and is a N^2 operation and just takes many many hours on my laptop.</p>
How to prune filters permanently?

TigerVersusT: <p>I’m trying to do filter pruning fro efficientNet-b1, I think I’ve set unimportant filters to zeros using distiller, but how can I remove those filters permanently？Since the distiiler has some bugs, I have to do this myself, where should I start?</p>
<p>Thanks for your help.</p>

TigerVersusT: <p>Reply to myself: DEADLY simple, just navigate through the whole model find Conv layers and repalce them with new Conv module. I was using distiller and think this problem in an univresal view, It is quit hard if you want to prune filters for ALL CNN based model but deadly simple for a given CNN.</p>
Training Pre-trained Quantized Resnet 18 model

CHOONG_HONG_LEE: <p>Hello there. I am new to quantization topic and have been trying to explore this. I have a question about training a pre-trained, quantized Resnet-18 model. Is this possible? If yes, is there a sample on how to do this? Thanks in advance.</p>

tom: <p>To train this model, you would likely need to convert it back to floating point and then train (because the quantized operators typically do not have autograd support).<br>
I am unaware of an automated way to do this.</p>
<p>Best regards</p>
<p>Thomas</p>

CHOONG_HONG_LEE: <p>Thanks for the comment <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Multilayer LSTM per-layer hidden states behavior

Qwinpin: <p>I have a question about the behavior of hidden and cell states in multilayer LSTM module.</p>
<p>torch.nn.LSTM layer with <em>num_layers</em> &gt; 1 starts each internal layer with default <em>h_0</em> and <em>c_0</em> (which are zero matrix usually), so, we can simply replace:</p>
<blockquote>
<p>lstm_m = torch.nn.LSTM(…, num_layers=2)<br>
x = lstm_m(x)</p>
</blockquote>
<p>with</p>
<blockquote>
<p>lstm = torch.nn.LSTM(…, num_layers=1)<br>
lstm2 = torch.nn.LSTM(…, num_layers=1)<br>
x, (h, c) = lstm(x)<br>
x = lstm2(x)</p>
</blockquote>
<p>But in the second case, we can pass last states <em>(h, c)</em> as initial states of <em>lstm2</em> in order to keep the flow:</p>
<blockquote>
<p>x = lstm2(x, h_0=h, c_0=c)</p>
</blockquote>
<p>I tried to find any information about the chosen behavior of multilayer LSTM: start each internal layer with its own default states, but I couldn’t. Is there some theory behind that or practical reasons?</p>
Parallel execution of different modules with different input

surfer: <p>Hi,</p>
<p>I have about 8k one dimensional vectors of length 128, and corresponded to each vector I have a linear layer. I would like to pass each vector through its corresponded linear layer, and then concatenate their outputs, is there a way which I could take advantage of gpu, and do them in parallel ?</p>
<p>There is a similar question <a href="https://discuss.pytorch.org/t/parallel-execution-of-modules-in-nn-modulelist/43940/7">here</a> but I am not sure how to adjust that to my problem.</p>

eqy: <p>If you have different “linear layers,” (one layer for each input) then this sounds like it would be a batched-matrix multiply:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.bmm.html" rel="noopener nofollow ugc">torch.bmm — PyTorch 1.10.0 documentation</a><br>
b = 8000,<br>
n = 1,<br>
m = 128,<br>
p = output dim of linear layer.<br>
It should also be expressible as an einsum:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.einsum.html" rel="noopener nofollow ugc">torch.einsum — PyTorch 1.10.0 documentation</a></p>

surfer: <p>Thank you so much I will try it.</p>

surfer: <p>my apologies this may be a dumb question; I am not sure how to create a matrix of modules, should I just copy the parameters into a tensor of size (b×m×p)?<br>
If I only use torch.tensor to create a new matrix of (b×m×p) dimension, then how to initialize and set the bias of that?</p>
<p>Here is what I tried, will back propagation be applied on output_layers later in training?</p>
<p>self.output_layers = torch.rand([8000, 128, 2], requires_grad=True)<br>
torch.nn.init.normal_(self.output_layers, mean=0, std=5e-3)</p>

eqy: <p>Generally parallelization across modules is difficult, so you may want to write your custom module that does everything in a single layer, as it looks like you are currently doing. What you have looks OK, and if you want bias you can add a bias tensor to your custom module.</p>
Multi gpu inference pytorch

marcel_Gibier1: <p>I trained a model in multigpu thanks to accelerate from HuggingFace <a href="https://github.com/huggingface/accelerate" rel="noopener nofollow ugc">https://github.com/huggingface/accelerate</a></p>
<p>Nevertheless in inference I can’t see the use of my two gpus. I have done</p>
<pre><code class="lang-auto">   generator = Generator().to(device)
    # Load weights
    checkpoint = torch.load(weights_dir, map_location="cpu")
    generator.load_state_dict(checkpoint['gen_state_dict'])
    generator = accelerator.prepare(generator)
    generator.eval()


How would you do it for inference? 

Thanks !</code></pre>

eqy: <p>I’m not familiar with accelerator but why prevents the same approach from being used at inference time? For example, just using the same accelerator workflow but removing the gradient computation and setting the model to eval mode?</p>
Specify this sampler with DataLoader

Luca_Pamparana: <p>Looking at the pytorch data loader docs, one can specify a custom sampler. In my current code, I sampled the indexes as follows:</p>
<pre><code class="lang-auto">batch_index = np.random.choice(batch_order, batch_size, p=seq_sample_probs).tolist()
batch = torch.tensor(data.x_train[batch_index], dtype=torch_dtype, device=torch_device, requires_grad=False)
</code></pre>
<p>I am moving everything to more modern pytorch and I have a <code>Dataset</code> object now. I wonder how I can now connect such a sampler in the corresponding data loader?</p>

eqy: <p>You can subclass <code>torch.utils.data.Sampler</code> <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.10.0 documentation</a> and pass an instance of this when constructing the DataLoader:<br>
<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.10.0 documentation</a></p>
<p>Since it looks like you are sampling batches here, you might also want to take a look at using <code>batch_sampler</code> instead.</p>
How do I Build and Install a Nightly Build from a Specific Day?

Stanley_C: <p>How do I Build and Install a Nightly Build from a specific day? I need to see if a PR is causing issues with my code.</p>

ptrblck: <p>You can specify the date and CUDA runtime in the install command. E.g to install the nightly from Sept. 28th with CUDA11.3:</p>
<pre><code class="lang-python">pip install --pre torch==1.11.0.dev20210928+cu113 -f https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html
</code></pre>

Stanley_C: <p>I’m trying to install the pytorch version before this PR:(<a href="https://github.com/pytorch/pytorch/pull/63089" class="inline-onebox" rel="noopener nofollow ugc">Derivatives of relu (#63027) by MengeTM · Pull Request #63089 · pytorch/pytorch · GitHub</a>), but I can can’t seem to get it to work. Are all nightly build deleted with the release of each new version?</p>
<p>pip3 install --pre torch==1.9.1.dev20210825+cu113 -f <a href="https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html</a></p>
Validation accuracy doesn't increase

farahnaz: <p>my validation accuracy doesn’t increase even though i do augmentation,add drop out and try freezing i should say that my data is connectivity matrics for 3 class. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9d568b6da61e4c9385329433c6906f889a81361.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a9d568b6da61e4c9385329433c6906f889a81361" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9d568b6da61e4c9385329433c6906f889a81361_2_690x257.jpeg" alt="Capture" data-base62-sha1="oepUZEJQyXgNNiDodwFSp8Egn3b" width="690" height="257" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9d568b6da61e4c9385329433c6906f889a81361_2_690x257.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9d568b6da61e4c9385329433c6906f889a81361.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9d568b6da61e4c9385329433c6906f889a81361.jpeg 2x" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">789×295 30.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

farahnaz: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0abc1b6db6aeb9b570cbab4a466b2e9d935eb184.jpeg" alt="Capture11" data-base62-sha1="1wXN7YWVfYvAxJp28BVYbOgwFi4" width="529" height="174"></p>

farahnaz: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b65e45c2f1da92da41a5ad5d671ab34780935765.jpeg" alt="hbhb" data-base62-sha1="q1iSpeVSo5M2tB9hChxeFxxF4mp" width="649" height="353"></p>

ptrblck: <p>It seems you are trying to replace the <code>model.fc</code> layer with a new dropout and linear layer.<br>
Your current code snippet won’t quite work, as you are first replacing <code>model.fc</code> with <code>nn.Sequential(nn.Dropout())</code> and are afterwards replacing this new layer with <code>nn.Linear</code>.<br>
If you want to use both new layers, use:</p>
<pre><code class="lang-python">model.fc = nn.Sequential(
    nn.Dropout(p=0.5),
    nn.Linear(512, 3)
)
</code></pre>
<p>As a quick test to check if other parts of the code have bugs, try to overfit a small dataset (e.g. just take 10 samples) and make sure your model is able to overfit this small data chunk.<br>
Once this is done, you could try to scale up the use case again by using more data and adding more regularization as the current training seems to be overfitting to the training data.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

farahnaz: <p>I will do your recommendations and yes i am newcomer😅<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=10" title=":pray:" class="emoji" alt=":pray:"></p>
I got error while creating Custom Dataset? help!

Bibhabasu_Mohapatra: <pre><code class="lang-auto">import numpy as np 
import pandas as pd
import torch 
import torch.nn as nn
from skimage import io
# image = io.imread(self.image_path[item])
# mask and img learnt from Awsaf's Notebook
class CustomDataset(torch.utils.data.Dataset):
    def __init__(self,image_paths,mask_paths = None,augmentations,preprocessing_fn=None):
        self.image_paths = image_paths
        self.mask_paths = mask_paths
        self.augmentations = augmentations
    def __len__(self):
        return len(self.image_paths)
    
    def __getitem__(self,item):
        image = io.imread(self.image_path[item])
     
        if mask_paths is not None:
            mask = np.load(mask_paths[item])
        
            augmented = self.augmentations(image=image,mask=mask)
            image = augmented["image"]
            mask = augmented["mask"]
            image = np.transpose(image, (2, 0, 1)).astype(np.float32)
            mask = np.transpose(mask, (2, 0, 1)).astype(np.float32)
            
            if preprocessing_fn is not None:
                image = self.preprocessing_fn(image)
            return {
                "image": torch.tensor(image, dtype=torch.float),
                "mask": torch.tensor(mask, dtype=torch.float)
            }
        else:
            augmented = self.augmentations(image=image)
            image = augmented["image"]
            image = np.transpose(image,(2,0,1)).astype(np.float32)
            if preprocessing_fn is not None:
                image = self.preprocessing_fn(image)
            return {
                "image": torch.tensor(image, dtype=torch.float)
            }
</code></pre>
<p>File “/tmp/ipykernel_39/3960429903.py”, line 9<br>
def <strong>init</strong>(self,image_paths,mask_paths = None,augmentations,preprocessing_fn=None):<br>
^<br>
SyntaxError: non-default argument follows default argument</p>

Bibhabasu_Mohapatra: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/nishant_bhansali">@Nishant_Bhansali</a></p>

ptrblck: <p>As the error message explains, you need to provide default values for all arguments to a function call after the first default was set.</p>
<pre><code class="lang-python">def __init__(self,image_paths,mask_paths = None,augmentations,preprocessing_fn=None):
</code></pre>
<p>is thus wrong and you would have to use:</p>
<pre><code class="lang-python">def __init__(self,image_paths,mask_paths = None, augmentations=DEFAULT_MISSING_HERE, preprocessing_fn=None):
</code></pre>
First SGD manually and then Adam optimizer, gets a runtimeerror

zilong: <p>Given a model with weights w(t) during a trainstep t and the next step w(t+1), what is the most efficient way to return the model weights back to w(t) from w(t+1)?</p>
<p>After going back to w(t) another training step follows to a new and different w(t+1), where second order derivatives should be possible through the first w(t+1) and w(t).</p>
<p>I am trying to implement the following <a href="https://github.com/cbfinn/maml" rel="noopener nofollow ugc">code</a>,  but in PyTorch.</p>
<p>EDIT:<br>
Currently my solution is:</p>
<pre><code class="lang-auto">optim = torch.optim.Adam(self.model.parameters(), lr=ARGS.lr)
losses = []
for inner_x, outer_x, inner_y, outer_y in SomeDataLoader:
    gradss = []

        for j in range(iterations):
            self.model.zero_grad()
            inner_output = self.model(inner_x)
            inner_loss = self.criterion(inner_output, inner_y)
            grads = torch.autograd.grad(inner_loss, self.model.parameters(), create_graph=True)
            gradss.append(grads)
            with torch.no_grad():
                for i, param in enumerate(self.model.parameters()):
                    param -= self.inner_lr * grads[i]

        outer_output = self.model(outer_x)
        outer_loss = self.criterion(outer_output, outer_y)

        with torch.no_grad():
            for j in range(iterations):
                for i, param in enumerate(self.model.parameters()):
                   param.data.copy_(param.data.clone().detach() + self.inner_lr * gradss[j][i].detach())
losses.append(outer_loss)
loss = torch.stack(losses).mean()
loss.backward()
optim.step()
</code></pre>
<p>However, I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
</code></pre>
Can TrivialAugment safely be used for object detection?

FaithfulAndTrue: <p>As the title says, I would like to use <a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.TrivialAugmentWide" rel="noopener nofollow ugc">TrivialAugment</a> within my training setup. So far I have been using Albumentations which appears to ensure that my bounding boxes remain valid after augmentations are applied. I didn’t see anything similar for torchvision’s TrivialAugment. Are we able to safely use this for an object detection model training?</p>
<p>Thanks.</p>
JIT compile staticmethods to a Module class

Amit_Gupta: <p>I want to export my model to ONNX, as a first step I want to compile my Module using <code>@jit.script</code>. My module contains one helper staticmethod to be used in inference. However when I call helper function in jit compiled model, I get the following error:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “SimpleONNX.py”, line 28, in <br>
print(model.return_ata(mat, x))<br>
File “/Users/…/venv/lib/python3.7/site-packages/torch/jit/_script.py”, line 667, in <strong>getattr</strong><br>
return super(RecursiveScriptModule, self).<strong>getattr</strong>(attr)<br>
File “/Users/…/venv/lib/python3.7/site-packages/torch/jit/_script.py”, line 384, in <strong>getattr</strong><br>
return super(ScriptModule, self).<strong>getattr</strong>(attr)<br>
File “/Users/…/venv/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1131, in <strong>getattr</strong><br>
type(self).<strong>name</strong>, name))<br>
AttributeError: ‘RecursiveScriptModule’ object has no attribute ‘return_ata’</p>
</blockquote>
<p>Given below is my python program</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn

class SimpleLinear(nn.Module):
    def __init__(self):
        super().__init__()
        self.mat = torch.rand(3,3)

    def forward(self, x:torch.Tensor):
        return torch.matmul(self.mat, x)

    @staticmethod
    def return_ata(mat:torch.Tensor,x:torch.Tensor):
        matx = torch.matmul(mat,x)
        xt = torch.transpose(x,0,1)
        return torch.matmul(xt, matx)


x = torch.rand((3,1))
mat = torch.rand(3,3)

SL = SimpleLinear()

model = torch.jit.script(SL)
print(model(x))
print(model.return_ata(mat, x))
</code></pre>
<p>Is there anyway to keep <code>return_ata</code> as a member function? Also will it retain itself during ONNX runtime?</p>
<hr>
<p>I can get it worked as class member with <code>@torch.jit.export</code> ie:</p>
<pre><code class="lang-auto">
    @torch.jit.export
    def return_ata(self, mat:torch.Tensor,x:torch.Tensor):
        matx = torch.matmul(mat,x)
        xt = torch.transpose(x,0,1)
        return torch.matmul(xt, matx)
</code></pre>
Docker with no ipc=host but multiple workers

rasoolfa: <p>Hi,</p>
<p>I was wondering is there any way to have dataloader with multiple workers but not running the docker with --ipc=host (i.e.  .nvidia-docker run --rm -ti)? I wasn’t able to use num_workers &gt; 0 when not using --ipc=host.</p>
<p>Asking because, on a gpu cluster machines that lunch a job via docker, all processes are running inside the same container, and are being inside the same ipc namespace. Since Pytorch’s containers will be running on a host with other jobs too, let’s say with Tensforflow jobs, it might cause a problem [?]</p>

hanshans: <p>Hello is this issue resolved by now? I have the same problem. I have a machine learning project using pytorch that is trained remotely. However the docker container is not started using the flags (–ipc=host nor --shm-size). The pytorch documentation (<a href="https://github.com/pytorch/pytorch#docker-image" rel="nofollow noopener">https://github.com/pytorch/pytorch#docker-image</a>) says this is required to run multiple workers in a docker container. Setting the number of workers to 0 is not an adequate solution, because then training takes takes 10 times longer. Is there any way to get pytorch dataloaders working in a docker container? But: We are able to create the dockerfile by our selfs? Can we define it there? Can we apply a different method?</p>

ptrblck: <p>What issue are you seeing with <code>--ipc=host</code> or setting the shared memory size via e.g. <code>--shm-size 8g </code>?</p>

hanshans: <p>I do not see any issues at all, but I am not able to set the flags myself. It is up to the admin. And he does not like to change the current settings for compatibility reasons (other users other frameworks). The only thing I am able to provide is a Dockerfile and the python code itself.</p>

ptrblck: <p>I’m not a docker expert, but to my understanding without these flags docker would only use a tiny amount of shared memory and thus (some) multiprocessing applications wouldn’t work, as they are unable to share data.<br>
If your admin cannot use these flags, you would either have to use a single worker (or a lower number) or maybe swap docker for another container solution (I’m unfortunately not familiar with other approaches).</p>

DeepakSaini119: <p>Hi <a class="mention" href="/u/hanshans">@hanshans</a>, I am faced with the same issue where I am provided with a docker without <code>--ipc==host</code> and any pytorch dataloader with more than 0 threads is getting killed. Were you able to solve this issue? Thanks.</p>
Is my code correct?

Yasar_Niyazoglu: <p>I’m doing transfer learning with pytorch and I did it this way, is it correct?</p>
<pre><code class="lang-auto">start=time.time()
print("Transfer Learning Eğitimi Başlıyor")
print("----------------------------------------------")
import torchvision.models as models


model = models.densenet161(pretrained=True) # Önceden eğitilmiş modelleri kullanmak için pretrained=True 

for param in model.parameters(): # Tüm modeli eğitmek için True
    param.requires_grad = False # Parametreleri donduruyoruz

model.classifier=nn.Sequential(
    
nn.Linear(2208,1000),
nn.ReLU(inplace=True),
nn.Linear(1000,500),
nn.ReLU(inplace=True),
nn.Linear(500,200),
nn.ReLU(inplace=True),
nn.Linear(200,100),
nn.ReLU(inplace=True),
nn.Linear(100,50),
nn.ReLU(inplace=True),
nn.Linear(50,20),
nn.ReLU(inplace=True),
nn.Linear(20,10),
nn.ReLU(inplace=True),
nn.Linear(10,4)
)


error = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

epoch=1
count=0
iterasyon=[]
for i in range(epoch):

    for i,(images,label) in enumerate (train_loader):
        
        out = model(images)
        loss = error(out, label)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        count+=1
        
        if count % 20 == 0:
            iterasyon.append(count)
            print("İterasyon:{}--loss:{:.4f}".format(count,loss.item()))

print("Transfer Learning Eğitimi Bitti")
print("----------------------------------------------")            
end=time.time()
print("süre:",end-start)
</code></pre>

ptrblck: <p>Yes, your code looks good. Do you see any issues with it?</p>

Yasar_Niyazoglu: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> hanks for your help.</p>
How could I set different factor along one dimension using pow?

coincheung: <p>Hi,</p>
<p>I can use <code>torch.pow</code> like this:</p>
<pre><code class="lang-auto">factor = 2.3
a = torch.randn(2, 4, 2, 2)
b = a.pow(factor)
</code></pre>
<p>However, I need to use something like this:</p>
<pre><code class="lang-auto">factor = torch.tensor([2.1, 2.2, 2.3, 2.4])
a = torch.randn(2, 4, 2, 2)
b = a.pow(factor, dim=1)
</code></pre>
<p>It seems that pytorch does not support this. How could I do that along with its backward computation?</p>

my3bikaht: <p><img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=10" title=":grin:" class="emoji only-emoji" alt=":grin:"></p>
<p><code>b = a.pow(factor.broadcast_to((2,4)).unsqueeze(2).broadcast_to((2,4,2)).unsqueeze(3).broadcast_to((2,4,2,2)))</code></p>
Fasterrcnn_resnet50_fpn loss is converging but no detections during prediction

FalseFeint: <p>Hi, I’m a complete beginner with object detection, so apologies if my issue is blaringly obvious!</p>
<p>I’m working with fasterrcnn_resnet50_fpn which has been modified to take in an arbitrary number of input channels using the following code:</p>
<pre><code class="lang-auto">self.faster_rcnn = create_faster_rcnn_model(
            num_classes, image_mean=image_mean, image_std=image_std
        )
        
        if num_channels != 3:
            # Adjusting initial layer to handle arbitrary number of inputchannels
            self.faster_rcnn.backbone.body.conv1 = nn.Conv2d(
                num_channels,
                self.faster_rcnn.backbone.body.conv1.out_channels,
                kernel_size=7,
                stride=2,
                padding=3,
                bias=False,
            )
</code></pre>
<p>If I use a 3 channel input, then I’m able to get predictions during inference no problem.  However, if I use anything else I always get 0 predictions.</p>
<p>I’ve check my loss for a 2-channel input and it converges:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a77325df2a9c2b0399e3d2603b5446e0348a8ea2.png" data-download-href="https://discuss.pytorch.org/uploads/default/a77325df2a9c2b0399e3d2603b5446e0348a8ea2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a77325df2a9c2b0399e3d2603b5446e0348a8ea2_2_306x250.png" alt="image" data-base62-sha1="nTkrkKk8L0rO1SS5bINAdXAbQNI" width="306" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a77325df2a9c2b0399e3d2603b5446e0348a8ea2_2_306x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a77325df2a9c2b0399e3d2603b5446e0348a8ea2_2_459x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a77325df2a9c2b0399e3d2603b5446e0348a8ea2_2_612x500.png 2x" data-dominant-color="FCFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">732×597 14.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So I think that means my gradients are connected and training properly.  Has anyone run into a similar issue?  Please let me know if more information is needed!</p>
Does PyTorch discard nodes while exporting to ONNX format?

jr.r: <p>I created a simple model using Torch</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        
    def forward(self, x, y):
        if self.training:
            return torch.matmul(x, y)
        return torch.add(x, y)

tensor1 = torch.Tensor([4])
tensor2 = torch.Tensor([3])
model = MyModel()
torch.onnx.export(model, (tensor1,tensor2), 'train_model.onnx', input_names=["x","y"], output_names=["z"], training=2)
</code></pre>
<p>In this, the exported ONNX file has only one Add node and does not include the Matmul node. I require to add both the nodes in the ONNX file and be able to switch between them while training &amp; Inference.</p>
<p>Thanks.</p>

ptrblck: <p>Based on the <a href="https://pytorch.org/docs/stable/onnx.html">docs of <code>torch.onnx.export</code></a> it seems you have the following choices for the <code>training</code> argument:</p>
<blockquote>
<p><strong>training</strong> (<em>enum</em> <em>,</em> <em>default TrainingMode.EVAL</em>) –</p>
<ul>
<li><code>TrainingMode.EVAL</code>: export the model in inference mode.</li>
<li><code>TrainingMode.PRESERVE</code>: export the model in inference mode if model.training is False and in training mode if model.training is True.</li>
<li><code>TrainingMode.TRAINING</code>: export the model in training mode. Disables optimizations which might interfere with training.</li>
</ul>
</blockquote>
<p>Based on this I don’t think the exported model can switch between training and evaluation and you would need to pick one option.</p>

jr.r: <p>Thank you. Helped me alot.</p>
Combine autograd with finite difference

pipme: <p>Hi,</p>
<p>I am wondering if there are ways to combine autograd and numerical gradients with finite difference method.</p>
<p>Specifically, for <a href="https://discuss.pytorch.org/t/modified-bessel-function-of-order-0/18609">Modified Bessel Function</a>, I also want to compute the gradient with respect to $\nu$. It’s simple to compute its gradient with finite difference while hard with autograd. How to wrap finite difference as an autograd function and use it in <code>torch.nn.Module</code> subclasses?</p>
<p>Thanks!</p>

googlebot: <p>You’d calculate &amp; store dBessel(…)/dorder (generally, dF(x)/dx). This can be done in [autograd.Function’s] forward() or postponed to backward(). Irrespective of how you obtained a gradient, in backward you do multiplication: dLoss/dx = dLoss/dF(x) * dF(x)/dx = grad_out * local_grad  and return that for each function input.</p>
<p>nn.Module is orthogonal to this, it would only be an additional wrapper around a function (or rather, a call to Function.apply). That is mainly useful for adding local nn.Parameters or if you use nn.Sequential declarative pipelines.</p>

pipme: <p>Thank you Alex! Yes I’d like to register $\nu$ as a nn.Parameter and optimize it. Will take a try, thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Customized wieghts for linear layer prohibit their updated after optimizer.step()

yuri: <p>the issue should be clearly explained in the title and this piece of code<br>
the only if condition was never met</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import numpy as np
dtype = torch.float32
X = torch.tensor([[1, 2, 3, 4, 5, 6]], dtype=dtype)
Y = torch.tensor([[1, 4, 9, 16, 25, 36]], dtype=dtype)


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.l1 = torch.nn.Linear(6, 6)
        self.l2 = torch.nn.Linear(6, 6)
        self.l3 = torch.nn.Linear(6, 6)
        self.l4 = torch.nn.Linear(6, 6)

    def forward(self, x):
        x1 = self.l1(x)
        x1 = F.relu(x1)
        x1 = self.l2(x1)
        x2 = x1
        x2 = F.relu(x2)
        print('l2 inside forward before assignment', self.l3.weight[2, 3])
        self.l3.weight = nn.Parameter(x1.repeat(6, 1)) # this is the cause
        print('l2 inside forward after assignment', self.l3.weight[2, 3])
        x2 = self.l3(x2)
        x2 = F.relu(x2)
        x2 = self.l4(x2)
        return x2, x1


model = Model()
optimizer = optim.SGD(model.parameters(), lr=.05, momentum=0.9)  # model.parameters()
Loss = nn.MSELoss()
n_iters = 100
for epoch in range(n_iters):
    optimizer.zero_grad()
    y_pred, x1 = model(X)
    loss = Loss(y_pred, Y)
    print('l2 outside forward', model.l3.weight[2, 3])
    e = torch.tensor(np.array(model.l3.weight.data.detach()))
    loss.backward()
    optimizer.step()
    print('l2 after backward and step', model.l3.weight[2, 3])
    print('e l2 after backward and step', e[2, 3])
    if torch.all(torch.eq(e, model.l3.weight.data)) == False: # to check if any parameter has changed
        print('something has changed in l3 :)')
    print('')
</code></pre>

yuri: <p>the weights assignments must be excuted as follows:</p>
<blockquote>
<p>self.l3.weight.data = x1.repeat(6, 1)</p>
</blockquote>
Problem when I import my onnx model

xamapl99: <p>Hi, I’m trying to import my onnx model that I exported with Pytorch (with torch.onnx.export()), and with dnn.readNetFromONNX I get an error. Wheras if i import the model with an onnx session it’s all correct.</p>
<p>The error that I get is this:</p>
<p>[ERROR:0] global /tmp/pip-req-build-afu9cjzs/opencv/modules/dnn/src/onnx/onnx_importer.cpp (2125) handleNode DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [Unsqueeze]:(30)</p>
I am getting this error when I try to run the pretrained hindsight-experience-replay

Sahithi_lavu: <p>Traceback (most recent call last):<br>
File “demo.py”, line 21, in <br>
o_mean, o_std, g_mean, g_std, model = torch.load(model_path, map_location=lambda storage, loc: storage)<br>
File “C:\Users\Student\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py”, line 365, in load<br>
f = open(f, ‘rb’)<br>
FileNotFoundError: [Errno 2] No such file or directory: “saved_models/‘FetchReach-v1’/model.pt”</p>
<p>The link of source is <a href="https://github.com/TianhongDai/hindsight-experience-replay" class="inline-onebox" rel="noopener nofollow ugc">GitHub - TianhongDai/hindsight-experience-replay: This is the pytorch implementation of Hindsight Experience Replay (HER) - Experiment on all fetch robotic environments.</a></p>

ptrblck: <p>As the error message explains, it seems the specified file is not found and I guess you might have skipped <a href="https://github.com/TianhongDai/hindsight-experience-replay#download-the-pre-trained-model">this step</a>, which asks you to download the files.</p>

Sahithi_lavu: <p>I have downloaded and saved it as mentioned but got the error.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3ab1cf43a48368626a0cb2ec33d280508cb1d4c6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3ab1cf43a48368626a0cb2ec33d280508cb1d4c6" title="Screenshot (38)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3ab1cf43a48368626a0cb2ec33d280508cb1d4c6_2_690x388.jpeg" alt="Screenshot (38)" data-base62-sha1="8neEdq4qlZGv6ci08ferjjV0ODs" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3ab1cf43a48368626a0cb2ec33d280508cb1d4c6_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3ab1cf43a48368626a0cb2ec33d280508cb1d4c6_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3ab1cf43a48368626a0cb2ec33d280508cb1d4c6_2_1380x776.jpeg 2x" data-dominant-color="98C4E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (38)</span><span class="informations">1920×1080 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>In that case the additional characters around <code>FetchReach-v1</code> might cause the issue or using a relative path from a wrong working directory.</p>
Should I use pytorch or albumentations totensor functions in dataset?

Oliver: <p>Hi</p>
<p>I’ve noticed there is a shape mismatch when I use albumentations to tensor (channel gets appended in the mask).<br>
I.e. I get batch_size, height, width, channels<br>
When I use pytorch transforms to tensor I get the correct <code>batch_size, channels, height, width</code> for both images and masks.<br>
Now I’m transposing, but I’m wondering if this is the wrong approach?</p>
Model performance in papers

Honnang: <p>Hello. I have been researching deep learning super-resolution networks.<br>
The problem is that I never get the same results shown on published papers when I create the exact same network with the exact same settings using PyTorch.<br>
I get 0.3dB~0.5dB lower performance which is a big deal in super-resolution.<br>
Is there something these guys do to the model to improve results?<br>
Do they use a lower-level language?<br>
How do they tweak their performance using the presented datasets?</p>

ptrblck: <p>Reproducing results from a paper is often not trivial especially if the authors were not adding a lot of training details. In the best case the script with the used library versions would be published. If that’s not the case you would most likely need to play around with the hyperparameters or contact the authors for more information.</p>
Package not found error: while installing pytorch

bing: <p>Hi,<br>
I am trying to install pytorch version 1.6 using the below command<br>
<code>conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch</code><br>
but I am getting the following error</p>
<pre><code class="lang-auto">PackagesNotFoundError: The following packages are not available from current channels:

  - torchvision==0.7.0
  - pytorch==1.6.0

Current channels:

  - https://conda.anaconda.org/pytorch/linux-ppc64le
  - https://conda.anaconda.org/pytorch/noarch
  - https://ftp.osuosl.org/pub/open-ce/current/linux-ppc64le
  - https://ftp.osuosl.org/pub/open-ce/current/noarch
  - https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/linux-ppc64le
  - https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda-early-access/noarch
  - https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/linux-ppc64le
  - https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/noarch
  - https://repo.anaconda.com/pkgs/main/linux-ppc64le
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/linux-ppc64le
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/linux-ppc64le
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

</code></pre>
<p>I don’t understand where I am doing wrong.</p>

kBodolai: <p>Hi,</p>
<p>I had the same issue recently, when trying to install with python 3.10 , downgrading python to 3.9 did the trick for me.</p>
<p>What OS &amp; python version are you using?</p>
<p>Cheers,<br>
K.</p>
How to calculate gradient of a component of the loss

Dongwei_Ye: <p>Hi,</p>
<p>I have a loss function with two parts: <code>loss = loss1+loss2</code> and backpropagate.</p>
<p>I am looking for a way to calculate the gradient of each part with respect to the model parameter (weight &amp;bias). How should I handle this?</p>
<p>Currently, I can only get the gradient from the summation <code>loss</code>:</p>
<pre><code class="lang-auto">loss.backward()
print(self.Network[numlayer].weight.grad)
</code></pre>
<p>How do print out the for example gradient of loss1 wrt weight?</p>
<p>Thank you!</p>
<p>Bests,</p>

dazzle-me: <p>I think the naive solution will be the following :<br>
First, you don’t sum the losses, but instead do this</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss1.backward()
### do something with these gradients, like
## print(self.Network[numlayer].weight.grad)
optimizer.zero_grad()
loss2.backward()
## do something with these gradients
## ...
</code></pre>
<p>Since we do backpropagation 2 times in the same step, it can slow down the step, but I’m not sure about that since we compute gradients separately, like, in out case d(loss)/dW = d(loss_1 + loss_2)/dW = d(loss_1)/dW + d(loss_2)/dW =&gt; autograd engine will compute these gradients separately too and the only overhead we’ll get is optimizer.zero_grad()?</p>
<p>Though, I’m not sure how to handle the case when you need to have access to both gradients at once<br>
: thinking :</p>

Dongwei_Ye: <p>Thank you for your advice! But I am wondering if it is correct to have <code>optimizer.zero_grad()</code> in between two backward(). Would zero_grad() clear out all the grad of loss1 and then optimizer.step() only update the gradient from loss2?</p>

dazzle-me: <p>Yes, sure if you want to update weights using gradients from the backward pass of loss1 you should put optimizer.step() after loss1.backward()</p>
<p>But again, I’m not sure about influence on speed of this approach, you can try to measure it by comparing these snippets</p>
<pre><code class="lang-auto">import time
start_time = time.time()
loss = loss1 + loss2

optimizer.zero_grad()
loss.backward() 
optimizer.step()

end_time = time.time() - start_time
print("%s seconds" % (end_time))
</code></pre>
<p>and this one</p>
<pre><code class="lang-auto">import time
start_time = time.time()

optimizer.zero_grad()
loss1.backward() 
optimizer.step()

optimizer.zero_grad()
loss2.backward()
optimizer.step()

end_time = time.time() - start_time
print("%s seconds" % (end_time))
</code></pre>
Training error not decreasing

Istvan_Ferenc_Toth: <p>Hello, everyone!</p>
<p>I am trying to train a Densenet model, modified for personal purposes. The input is an image and the output is an array. The problem is, that after the first few epochs I have a training and validation error almost constant.</p>
<p>I tried to train it with a single image and the error decreases, then I used only 10 images for training, and again for a smaller Densenet model I got a decreasing training error. Then I moved to a larger training set of 3500 images and I got the flat error.</p>
<p>My question is, how can I find out a connection between the training set size and the number of network parameters? I am inclined to think that this might be the problem, since for example when I used only 10 samples the network had to be reduced in size for the error to decrease. I am using L1Loss since I have to calculate the error between complex quantities.</p>
Strange results on QAT YOLOv5

MrOCW: <p>Hi, I quantized YOLOv5 by adding QuantStub and DeQuantStubs to all the ConvBnReLU2d in the network, exported to ONNX and folded the QDQ nodes into QLinearConv.</p><aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/11-7Mnss4g6_RJ2CizITIWFo9bGPDNkO_/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/11-7Mnss4g6_RJ2CizITIWFo9bGPDNkO_/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/11-7Mnss4g6_RJ2CizITIWFo9bGPDNkO_/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/11-7Mnss4g6_RJ2CizITIWFo9bGPDNkO_/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">QAT.onnx</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>However, during inference, this is the result for the whole image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46af8c8917d0b3044cecce9f14edbf6083e81f03.png" alt="image" data-base62-sha1="a5jx9ZIHDjQzqhrRRtPLohyfVcL" width="217" height="177"></p>