akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to unpack data_loader in small size

Ajinkya.Bankar: <p>Hello,<br>
I have a PyTorch data_loader with three fields and iterates as follows:<br>
for i, (input, target_class, name) in enumerate(data_loader):</p>
<p>But I want it to enumerate over ‘input’ and ‘target_class’. How can I unpack data_loader in two values given that it has originally three values? Please help. Thanks.</p>

ariG23498: <p>I think either you can create a <code>DataSet</code> that only returns the <code>input</code> and <code>target_class</code>, or you could do something like this:</p>
<pre><code class="lang-python">for i, element in enumerate(data_loader):
    input, target_class, _ = element
</code></pre>
<p>Here you are indeed iterating over the three fields but are not using the name field.</p>

ejguan: <p>Do you want something like this?</p>
<pre><code class="lang-auto">for i, (input, target, _) in enumerate(dataloader):
    ...
</code></pre>
<p>Or, you can change your Dataset to yield (_<em>iter</em>_) or return (_<em>getitem</em>_) only input and target without name.</p>
AI for medical diagnoses

Tornike: <p>Hello, i want to try some medical problems with PyTorch. Would be very helpful if you shared your experience with me. How should i start? What articles or video tutorials would you recommend? What are the steps you go through while working on it ?</p>
<p>Thanks in Advance</p>

aksg87: <p>There are lots of places to start <a class="mention" href="/u/tornike">@Tornike</a></p>
<p>If you are interested in radiology-focused problems.</p>
<p>You could start by looking at some of the Kaggle competitions which have been on Pneumonia, Pneumothorax, Intracranial hemorrhage. Those discussion forums are full of more resources that you will be linked to!</p>
Proper way of freezing BatchNorm running statistics

Evangelos_Kazakos: <p>Hi everybody,</p>
<p>What I want to do is to use a pretrained network that contains batch normalization layers and perform finetuning. So I want to freeze the weights of the network. Apart from freezing the weight and bias of batch norm, I would like also to freeze the running_mean and running_std and use the values from the pretrained network. I 've seen many posts that address this issue, but there is not clear answer. The general answer is to put the batchnorm layers in eval mode. But people report that if you first put your whole model in train mode and after that only the batchnorm layers in eval mode, training is not converging. Another post suggests to override the train() function by putting the batchnorm layers in eval mode inside train(). Is this the way to go? Does anyone have to suggest a better way? Thank you.</p>

it_happens: <p>Hey, I’m reporting exactly what you said there almost 3 years ago. My network won’t converge after I call module.eval() for all my batchnorm2d modules. But I still do need to find a way to freeze the running stats of my batchnorm2d layers. I wonder if you have found a way?</p>

overfitted: <p>Hi,</p>
<p>You can freeze BN trainable parameters with something like:</p>
<pre><code class="lang-auto">for m in model.modules():
    if isinstance(m, nn.BatchNorm2d):
        m.weight.requires_grad_(False)
        m.bias.requires_grad_(False)
</code></pre>
<p>This should be done once before training starts. Also don’t forget to pass to the optimiser ONLY the trainable params.</p>
<p>And then you can freeze BN statistics at the beginning of each epoch (after you call model.train()) with:</p>
<pre><code class="lang-auto">for m in model.modules():
    if isinstance(m, nn.BatchNorm2d):
        m.eval()
</code></pre>
<p>Hope this helps!</p>

it_happens: <p>First of all, thanks for your immediate reply.<br>
Regarding calling model.eval(), I am doing exactly what you suggest here.<br>
Regarding configuring the trainable parameters so that they won’t require gradients, I found that this was actually the exact cause for gradient explosion in my case (I’m training with half precision a Resnet-based architecture for 3D human pose estimation). After I switch the code back so that the gradients for bn.weight and bn.bias are updated as usual (but the bn layers still function under eval mode at training time), the gradient explosion stopped. And for my own experiments this is the ideal behaviour (at training time the trainable params of bn layers are updated as usual but the runnning stats are fixed)</p>
Programmatically check if PyTorch is using a GPU?

crossjbeer: <p>Hi All,</p>
<p>Apologies if this is a repeat question. I have searched around for some time and I am not able to find any resources.</p>
<p>My Situation:<br>
I am attempting to create a CRON job which will automatically engage a Neural Network building pipeline. I have a work station with 2 GPU installed. I would like a python script to detect when either GPU is being used, and begin building the model on one that is not being used.</p>
<p>What I have found so far:<br>
I have done a bit of searching, and keep coming back to people’s mention of the ‘torch.cuda.is_available()’ function, as mentioned in this link [<a href="https://stackoverflow.com/questions/48152674/how-to-check-if-pytorch-is-using-the-gpu" class="inline-onebox" rel="noopener nofollow ugc">python - How to check if pytorch is using the GPU? - Stack Overflow</a>]. However, as is also mentioned in that link, this function doesn’t tell us whether or not the GPU is currently building a model. Rather, it simply tells us whether a GPU exists at all.</p>
<p>My Question:<br>
Is there a way to programmatically check whether or not a PyTorch model is building on a GPU (or anything analogous) from within a Python script?</p>
<p>Thanks a lot and have a nice day!</p>

eqy: <p>Simply checking whether a GPU is “used” might be dangerous as it might be a race with something else that is contending for a GPU. However, if you are confident about the scheduling of jobs, you can try something like <code>nvidia-smi --query-compute-apps=pid,process_name,used_memory,gpu_bus_id --format=csv</code>.</p>

crossjbeer: <p>Thank you for your response Eqy. I appreciate you taking the time out.</p>
<p>I think this command should be by far the easiest output of nvidia-smi to parse in a python script. Would you mind elaborating on what you mean when you say “it might be a race with something else that is contending for a GPU”?</p>
<p>Have a nice day</p>

eqy: <p>For example, if the intent is to avoid stomping over a user’s job that is running on the same system, this method is dangerous because a user can start a job <em>just</em> after <code>nvidia-smi</code> polls the GPUs. So a GPU might “look” idle when it has in fact just been used.</p>

crossjbeer: <p>This is a great point. Luckily, I am the only one who uses the system. I believe, unless I do anything manually, the scheduling script will be the only thing making use of the GPU at any given time.</p>
<p>Your command is parsing well, and it seems you have solved my problem! Thanks a lot.</p>
How to get nn.Parameter listed when net is printed?

Tony_Gracious: <pre><code class="lang-auto">import torch
import torch.nn as nn

class MyNet(torch.nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.layer = nn.Linear(10, 10)
        self.parameter = torch.nn.Parameter(torch.zeros(10,10, requires_grad=True))

net = MyNet()
print(net)

</code></pre>
<p>Ouput is</p>
<pre><code class="lang-auto">MyNet(
  (layer): Linear(in_features=10, out_features=10, bias=True)
)
</code></pre>

ptrblck: <p>You could print all parameters of the model via:</p>
<pre><code class="lang-python">print(list(net.parameters()))
# or
print(dict(net.named_parameters()))
</code></pre>

Tony_Gracious: <p>I want parameters to come in this command print(net)<br>
This is more interpretable that others</p>

ptrblck: <p>In that case you could override the <a href="https://github.com/pytorch/pytorch/blob/bb640efa404fb2b4ac0f972dbce4ad87c21f6e3e/torch/nn/modules/module.py#L1711-L1718"><code>extra_repr</code></a> method for the module.</p>

Tony_Gracious: <pre><code class="lang-auto">class MyNet(torch.nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.layer = nn.Linear(10, 10)
        self.parameter1 = torch.nn.Parameter(torch.zeros(10,10, requires_grad=True))
        self.parameter2 = torch.nn.Parameter(torch.zeros(10,10, requires_grad=True))
        
    def extra_repr(self) -&gt; str:
        
        named_modules = set()
        for p in net.named_modules():
            named_modules.update([p[0]]  )    
        named_modules = list(named_modules)

        string_repr = ''
        for p in net.named_parameters():
            name = p[0].split('.')[0]
            if name not in name_modules:
                string_repr = string_repr + '('+ name +'): ' \
                    +'tensor(' + str(tuple(p[1].shape))+ ', requires_grad='+ str(p[1].requires_grad) +')\n' 
        
        return string_repr

net = MyNet()
print(net)
</code></pre>
<p>Output is</p>
<pre><code class="lang-auto">MyNet(
  (parameter1): tensor((10, 10), requires_grad=True)
  (parameter2): tensor((10, 10), requires_grad=True)
  
  (layer): Linear(in_features=10, out_features=10, bias=True)
)

</code></pre>
<p>Is there a better automated version that this?</p>
Custom transform output does not match the original

m3tobom_M: <p>I am trying to implement the below transforms myself but custom outputs and torch outputs are different. A portion of the output is below (some pixel values from different channels). What am I doing wrong?</p>
<p>Note: I do not need tensor as output.</p>
<p>Custom transforms</p>
<pre><code class="lang-auto">@njit() 
def normalize_cv2(image, mean, std):
    for d in range(3):
        image[d, :, :] = np.divide(image[d, :, :], 255)
        image[d, :, :] = np.divide(np.subtract(image[d, :, :], mean[d]), std[d])
    return image

cvresized = cv2.resize(cvimage, (250, 250))
cvresized = cv2.cvtColor(cvresized, cv2.COLOR_BGR2RGB)
cvresized = np.array(cvresized, dtype = np.float32) 
cvresized = np.transpose(cvresized, [2, 0, 1])
normalized_cvimage = normalize_cv2(cvresized, mean, std)
</code></pre>
<p>Torch transforms</p>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.Resize((250, 250)),
    transforms.ToTensor(),
    transforms.Normalize(mean, std)  
])
</code></pre>
<p>Outputs</p>
<pre><code class="lang-auto">1 - Torch output:  tensor(-0.3176)
1 - Custom output:  -0.30196077
2 - Torch output:  tensor(-0.2627)
2 - Custom output:  -0.24705881
3 - Torch output:  tensor(-0.1922)
3 - Custom output:  -0.19215685
4 - Torch output:  tensor(-0.0980)
4 - Custom output:  -0.11372548
</code></pre>

ptrblck: <p>I don’t know how you are comparing these two outputs, but make sure to create a clone of the input numpy array, since you are manipulating it inplace.<br>
Also, note that <code>ToTensor()</code> will normalize numpy arrays, if they are passed in the <code>uint8</code> dtype, not if they are using the <code>float32</code> type.<br>
Additionally, I’ve removed the resizing, as I don’t know what kind of interpolation is used by OpenCV by default.<br>
Given that, your method works fine:</p>
<pre><code class="lang-python">def normalize_cv2(image, mean, std):
    for d in range(3):
        image[d, :, :] = np.divide(image[d, :, :], 255)
        image[d, :, :] = np.divide(np.subtract(image[d, :, :], mean[d]), std[d])
    return image


mean = np.array([0.5, 0.5, 0.5])
std = np.array([0.5, 0.5, 0.5])
cvresized = np.random.randint(0, 256, (3, 250, 250)).astype(np.float32)
x = copy.deepcopy(cvresized)
normalized_cvimage = normalize_cv2(cvresized, mean, std)

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean, std)  
])


out = transform(x.transpose(1, 2, 0).astype(np.uint8))
print(np.max(np.abs((out.numpy() - normalized_cvimage))))
&gt; 0.0
</code></pre>

m3tobom_M: <p>I guess differences occur due to resizing operations of PIL and OpenCV. Even I use BILINEAR in PIL and INTER_LINEAR in OpenCV there is a difference in outputs.</p>
Train ResNet50 on pytorch 1.8

jS5t3r: <p>Hi,</p>
<p>I want to train this repository: <a href="https://github.com/weiaicunzai/pytorch-cifar100/blob/master/train.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch-cifar100/train.py at master · weiaicunzai/pytorch-cifar100 · GitHub</a> which is some years old.</p>
<p>I am getting this message during training:</p>
<pre><code class="lang-auto">python train.py -net resnet50 -b 64 &gt; runs/performance/a100_128_b64_resnet50.log
/home/user/.conda/envs/a100/lib/python3.8/site-packages/torch/jit/_trace.py:152: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  if a.grad is not None:
</code></pre>
<pre><code class="lang-auto">/home/user/.conda/envs/a100/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:154: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
  warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
</code></pre>
<p>Do I need to care about these messages? If yes, how to fix them?</p>

ptrblck: <p>Both warnings mention a proper fix, so did you try to apply it and was it not working?<br>
E.g. the first one mentions that the <code>.grad</code> attribute of a non-leaf tensor can be accessed after calling <code>.retain_grad()</code> on this tensor.</p>

jS5t3r: <p>I tried to apply and it seems to work… but I am not sure, if this is good letting as it is…</p>
Model parameters as sum of variable and constant weight

paulhofman: <p>Hello everyone,<br>
I am trying to train a model that uses parameters that are a sum of two sets weights and I want to to train only one set of weights. That is, for every weight w = w_1 + w_2, where w_2 is a constant and w_1 is to be optimized using an optimizer. Thus, in the forward step w is used to compute the loss, but in the backward step only w_1 is adjusted. Is this possible?</p>

pascal_notsawo: <p>It is possible.</p>
<p>Just make <code>w1.requires_grad = True ~~ w1.requires_grad_(True)</code> and <code>w2.requires_grad_(False)</code>.</p>
<p>Look at the example below.</p>
<pre><code class="lang-auto">import torch
class Model(torch.nn.Module) :
    def __init__(self) :
        super().__init__()
        self.w1 = torch.nn.parameter.Parameter(data = torch.tensor(1.), requires_grad = True)
        self.w2 = torch.nn.parameter.Parameter(data = torch.tensor(2.), requires_grad = False)

    def forward(self, x) :
        w = self.w1 + self.w2
        return w * x  

def mae(y, y_pred) :
    """mean absolute error"""
    return (y - y_pred).abs().mean()


# data 
x = torch.tensor([1., 3, 3])
y = 2 * x
# model
model = Model()


# model.w1.grad and model.w2.grad are None 
y_pred = model(x) # tensor([3., 9., 9.], grad_fn=&lt;MulBackward0&gt;)
loss = mae(y_pred, y) # tensor(2.3333, grad_fn=&lt;MeanBackward0&gt;

loss.backward() 

# model.w1.grad is equal to tensor(2.3333) and model.w2.grad is always None (or 0 if you zero the gradients)
</code></pre>
<p>In this case, <code>optimizer.step()</code> will update the parameters. For example : <code>w1 = w1 - learning_rate*w1.grad != w1</code> and  <code>w2 = w2 - learning_rate*w2.grad = w2 - 0 = w2</code> (In fact, optimizing it won’t even do that for w2, since w2 will be seen in the computation graph as a constant)</p>

paulhofman: <p>I understand, thank you for the example!</p>
Issue tracing FasterRcnn model

Miafr: <p>Hi all,<br>
I’m trying to trace fasterrcnn_mobilenet_v3_large_fpn model (fine tuned with custom classes) to be used in an android app<br>
when running the following code:</p>
<pre><code class="lang-auto">model = fasterrcnn_mobilenet_v3_large_fpn(pretrained=True)

model.eval()
input_tensor = torch.rand(1,3,224,224)
print(input_tensor)
script_model = torch.jit.trace_module(model,input_tensor)
script_model.save("basic_fasterrcnn_mobilenet3.pt")
</code></pre>
<p>I get this problem:</p>
<blockquote>
<p>/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)<br>
910<br>
911     if not isinstance(inputs, dict):<br>
 → 912         raise AttributeError(“expected a dictionary of (method_name, input) pairs”)<br>
913<br>
914 old_module_map = torch.jit._trace._trace_module_map<br>
<strong>AttributeError: expected a dictionary of (method_name, input) pairs</strong></p>
</blockquote>
<p>I guesss I’m missing something and not converting the model correctly? could you refer me to how it should be done?</p>
How to Train two sub-networks

davidlee: <p>Hi all,</p>
<p>I am trying to train the autoencoder model that consists of two sub-encoders and one generic decoder (not concatenation!)<br>
please see the attached figure.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/897d3a3ce70de790c3e4c7b137af4f762fd01413.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/897d3a3ce70de790c3e4c7b137af4f762fd01413" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/897d3a3ce70de790c3e4c7b137af4f762fd01413_2_690x420.jpeg" alt="image" data-base62-sha1="jChI9n1eO9H3etTKKrWcV6LKMhR" width="690" height="420" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/897d3a3ce70de790c3e4c7b137af4f762fd01413_2_690x420.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/897d3a3ce70de790c3e4c7b137af4f762fd01413_2_1035x630.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/9/897d3a3ce70de790c3e4c7b137af4f762fd01413.jpeg 2x" data-dominant-color="EFF1EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1115×680 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Would the code below work? or please advise me how to train this!.</p>
<pre><code class="lang-auto"># Define individual modules
encoder1 = torch.nn.Linear(10, 4).to(device)
encoder2 = torch.nn.Linear(10, 4).to(device)
decoder = torch.nn.Linear(4, 10).to(device)

# Define each model
model1 = nn.Sequential(encoder1, decoder)
model2 = nn.Sequential(encoder2, decoder)

# Optimizer
optim_1 = torch.optim.Adam(model1 .parameters(), lr=learning_rate)
optim_2 = torch.optim.Adam(model2.parameters(), lr=learning_rate)

# Train

for epoch in range(epochs):
model1.train()
model2.train()

     for i, data in enumerate(train_loader):
           input1, output1 = data[:5]
           input2, output2 = data[5:]

           model1.zero_grad()
           pred = model1(input1)
           loss_1 = criterion(pred , output1)
           loss_1 .backward()

           model2.zero_grad()
           pred = model2(input2)
           loss_2 = criterion(pred , output2)
           loss_2.backward()</code></pre>

ariG23498: <p>Hey <a class="mention" href="/u/davidlee">@davidlee</a><br>
The code that you have provided creates two encoders and one decoder. In the training loop you split the data into two and send each half to each encoder. There is no updation of the decoder whatsoever. Is that what you want?</p>
<p>Another obeservtion is that in the picutre it says that each encoder gets to have the data in alternate iterations. Does this align with the splitting of the data?</p>
Define custom dataloader

rui_zhou: <p>Is there any way to use my own dataset (image and groundtruth in png format) instead of coco dataset with .json file in pytorch ?<br>
or how can I change my image and label to coco dataset form?</p>

ariG23498: <p>Hey <a class="mention" href="/u/rui_zhou">@rui_zhou</a><br>
I think <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">custom dataset tutorial</a> could be a good place for you to start.</p>
<p>This will help you deal with the what and how of a custom dataset and dataloader.</p>
<p>Hope this helps you <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=9" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"></p>
Gradient computation error in inplace operation

shinshiner: <p>I have an encoder-decode model and a function that calculating something iteratively:</p>
<pre><code class="lang-auto">f(a,b,c):
   # a: [batchsize, n, channel]
   res_a = torch.empty(a.shape)
   res_b = torch.empty(b.shape)
   for i in range(a.shape[1]):
        res_a[:,i] = balabala
        res_b[:,i] = balabala
   return torch.cat([res_a,res_b],-1)
</code></pre>
<p>This is the main procedure of forward function:</p>
<pre><code class="lang-auto">def forward(x):
    a, b, c = x[0:i], x[i:j], x[j:k]
    q = f(a,b,c)
    x' = decoder(encoder(x))
    a, b, c = x'[0:i], x'[i:j], x'[j:k]
    q' = f(a,b,c)
    return q, q'
</code></pre>
<p>The loss is the difference of q and q1</p>
<p>Then an error occurs when backward for the computation of <code>q'</code>:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 3]], which is output 0 of SelectBackward, is at version 21; expected version 20 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>I really have no idea where I modify the variables for gradient computation, all I do in function <code>f</code> is just indexing some data from input and conduct some computations (add, multiply), then store the result to a new tensor.</p>
Distributeddataparallel across two machine with different version of Pytorch and CUDA

kolohe113: <p>Hello, I am trying to set up distributed learning across two laptops using distributeddataparallel. One computer has CUDA 10 and the other one has CUDA 11. Will this create problem during training?</p>

ptrblck: <p>I haven’t tried mixing different CUDA and NCCL versions in DDP runs, but guess it might yield unexpected behavior or other issues. Especially, if you are also mixing different PyTorch releases.</p>
Does my loss function need a tensor with a gradient?

hyt: <p>Hi,</p>
<p>so my Neural Network’s forward() returns a tensor of shape: z.size() torch.Size([64, 1]) whereas</p>
<p>z=<br>
tensor([[0.000],<br>
[0.264],<br>
…<br>
[0.000]], grad_fn=)</p>
<p>Now my original y is of shape y.size()= torch.Size([64]) looking like</p>
<p>y = tensor([0., 0., 0. … 0. ])</p>
<p>Now my y_pred = model(X) has shape torch.Size([64, 1]).</p>
<p>Now I want to do loss = loss_fn(y_pred, y) but I need the same dimensions. I know I could use z.view(-1) to get the correct reshaping but then I loose grad_fn= no? Isn’t that problematic for the backpropagation?</p>
<p>Sorry for the missing formatting but I don’t get that forum here. Does it support markdown? Latex?</p>

KFrank: <p>Hi Hyt!</p>
<aside class="quote no-group" data-username="hyt" data-post="1" data-topic="120753" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/f08c70/48.png" class="avatar"> hyt:</div>
<blockquote>
<p>I know I could use z.view(-1) to get the correct reshaping but then I loose grad_fn= no?</p>
</blockquote>
</aside>
<p>No, you won’t lose your <code>grad_fn</code> when using <code>.view()</code>.  Here is a<br>
short example:</p>
<pre><code class="lang-plaintext">'1.9.0.dev20210504'
&gt;&gt;&gt; t = torch.randn (4, 1, 2, requires_grad = True)
&gt;&gt;&gt; y_pred = t.sum (dim = 2)
&gt;&gt;&gt; y_pred
tensor([[ 0.6060],
        [-0.4268],
        [-0.6957],
        [ 1.3155]], grad_fn=&lt;SumBackward1&gt;)
&gt;&gt;&gt; y_pred.view (-1)
tensor([ 0.6060, -0.4268, -0.6957,  1.3155], grad_fn=&lt;ViewBackward&gt;)
</code></pre>
<p>It is true that the “root node” of the computation graph now has<br>
<code>grad_fn=&lt;ViewBackward&gt;</code>, but the <code>grad_fn=&lt;SumBackward1&gt;</code><br>
(in my example) is still there – it’s just one node up from the root in<br>
the computation graph and not printed out as part of the root-node<br>
tensor.</p>
<p>To answer the question in your title:  Yes, to backpropagate usefully<br>
you do need to feed a tensor with a gradient (<code>requires_grad = True</code>)<br>
to your loss function.  But, as noted above, <code>.view()</code> doesn’t spoil this.</p>
<p>Best.</p>
<p>K. Frank</p>
How does a transposed convolutional layer apply its kernels to the inputs?

Jide_Olabisi: <p>Hi, I know how a convolutional layer apply its kernels to an input but I don’t understand how transposed convolutional layer apply its kernels to an input. My main question is below.</p>
<p>Question:<br>
If a convolutional layer takes in a three channel input and outputs a two channel output and the shape of each kernels are 2x2, I know surely that it will have 6 kernels divided into two groups because it is to give out a 2 channel output so therefore two groups has shown below.</p>
<blockquote>
<blockquote>
<blockquote>
<p>from torch import nn<br>
conv1 = nn.Conv2d(3,2,2,1,0)<br>
conv1.weight.data.numpy()</p>
</blockquote>
</blockquote>
</blockquote>
<pre><code>array([[[[-0.09294177,  0.19253497],
     [-0.27820718, -0.07189114]],

    [[-0.27682984, -0.05606458],           &lt;---- one group
     [ 0.01134909, -0.21749675]],

    [[-0.2351923 ,  0.2374857 ],
     [-0.0346033 , -0.26447   ]]],


   [[[ 0.2467276 ,  0.06628369],
     [ 0.26501465,  0.11644475]],
                                           &lt;----- one group making two 
                                                  groups of three
    [[-0.09835644, -0.06396657],
     [-0.05590855,  0.06890304]],

    [[ 0.22788118,  0.22287966],
     [-0.20899878, -0.03188486]]]], dtype=float32)
</code></pre>
<p>You can see that it has six kernels divided into two groups of 3 which makes sense but if a transposed convolutional layer were to take in a three channel input and outputs a two channel output and the shape of each kernels being 2x2, why is it that instead of it having 6 kernels divided into two groups of three, it has 6 kernels divided into 3 groups of two as shown below?</p>
<blockquote>
<blockquote>
<blockquote>
<p>conv2 = nn.ConvTranspose2d(3,2,2,1,0)<br>
conv2.weight.data.numpy()</p>
</blockquote>
</blockquote>
</blockquote>
<pre><code>array([[[[ 0.28577724, -0.29587495],
     [-0.24003945, -0.3524448 ]],
                                    &lt;---- one group
    [[-0.15984103,  0.22188954],
     [-0.10990701, -0.20565327]]],


   [[[ 0.17101079, -0.17623127],
     [-0.12097928, -0.0211492 ]],
                                     &lt;----- one group
    [[-0.21161021, -0.33530322],
     [-0.16497111,  0.19984488]]],


   [[[-0.05084743, -0.2563213 ],
     [-0.28287342, -0.30839682]],
                                     &lt;----- one group making three groups 
                                            two
    [[-0.330719  ,  0.07809895],
     [-0.16823643, -0.34404978]]]], dtype=float32)
</code></pre>
<p>You can see above that it has 6 kernels divide into 3 groups of 2. Isn’t it meant to be divided into 2 groups of three because it is to give out two channel thats why it should be divided into two groups like the convolutional layer? Why is the arrangement this way and how does 6 kernels divided into 3 groups of 2 turn a 3 channel input into two?</p>
An issue with trained model mobility

Aiman_Mutasem-bellh: <p>Dear <span class="mention">@all</span></p>
<p>I have an issue with the trained model it works well each time that I loaded on the same machine (server). The problem is that the models don’t work on any other machine (server) I’m not able to deal with like the standard models such as BERT.</p>
<p>Any explanation for this issue and how to fix it?</p>
<p>Kind regards,<br>
Aiman Solyman</p>

Aiman_Mutasem-bellh: <p>Any suggestions to fix this issue? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to get duration of wav in torchaudio

wxystudio: <p>Before I always use python wave package to calculate the wav duration, and use torchaudio for processing.<br>
So how can I calculate the wav duration in torchaudio? Sorry I can’t find the API in pytorch documentation</p>
<p>thx</p>

JuanFMontesinos: <p>You can calculate the duration dividing the amount of elements of the audio by the samplerate.</p>
'is crowd' parameter in FasterRcnn torchvision

Arun_Mohan: <p>Can someone explain about ‘is crowd’ parameter in FasterRcnn torchvision.what I understood is that those annotations with is_crowd=True is ignored during evaluation.Please someone explain it more.</p>

ptrblck: <p>AS far as I know, the <code>iscrowd</code> label is used in the <code>cocoapi</code> to determine the target type.<br>
I.e. <code>iscrowd = 0</code> uses polygons for the object instances, while <code>iscrowd = 1</code> uses run-length encoding (RLE).</p>

Arun_Mohan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for correcting me.Now i got it</p>
Reshape dataset outputs

Chamroukhi: <p>i have ResNet 3D with 4 input (batch, number_video_frame, height=112, width=112)<br>
and my dataset(BATCH x FRAMES x CHANNELS x HEIGHT x WIDTH)<br>
how to reshape dataset outputs to make it work with ResNet 3D ?</p>

AKASH_KUMAR: <p>ResNet 3D network has input of 5 dimensions includin batch size. 3D means 3 number of channels in the input. Are you sure you are using ResNet3D?</p>

Chamroukhi: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/131a05a038151311f80fcb0897f503f35c0adb84.png" data-download-href="https://discuss.pytorch.org/uploads/default/131a05a038151311f80fcb0897f503f35c0adb84" title="Capture d’écran 2021-05-09 à 12.12.50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/131a05a038151311f80fcb0897f503f35c0adb84_2_690x207.png" alt="Capture d’écran 2021-05-09 à 12.12.50" data-base62-sha1="2IYP4ygcP8d85eCQxtjMHRSc0uw" width="690" height="207" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/131a05a038151311f80fcb0897f503f35c0adb84_2_690x207.png, https://discuss.pytorch.org/uploads/default/original/3X/1/3/131a05a038151311f80fcb0897f503f35c0adb84.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/3/131a05a038151311f80fcb0897f503f35c0adb84.png 2x" data-dominant-color="ECEDEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2021-05-09 à 12.12.50</span><span class="informations">730×219 34.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AKASH_KUMAR: <p>Yeah, so it means mini-batches of 3-channel RGB clips. The batch size is the 5th dimension. 3 is the number of channels which is the 4th dimension. T denotes the number of frames → 3rd dimension and spatial size of the frame are the last two dimensions.<br>
For e.g. 16, 3, 16, 112, 112 → Here 16 is batch size. This should be the input to your 3D model.</p>

Chamroukhi: <p>thanks for help but i have torch.Size([3, 15, 112, 112]) how add batch size  to be like this ([32,3, 15, 112, 112]) ?</p>

aza: <p><a href="https://pytorch.org/docs/stable/generated/torch.stack.html" class="inline-onebox" rel="noopener nofollow ugc">torch.stack — PyTorch 1.8.1 documentation</a> should do the trick – just take your 32 tensors and stack them into the desired batch</p>
Reshape pytorch tensor

Chamroukhi: <p>i have Video Tensor Size: torch.Size([15, 3, 112, 112])<br>
i want to reshape all tensor to [3,15,112,112] it is possible ?</p>

ariG23498: <p>If the variable is x then <code>x.permute(1,0,2,3)</code> might work in your case.</p>
Distributed Dataparallel check if working

whoab: <p>Hi,<br>
I’m not sure how to check if distributed data parallel is working. I saw some sources that said you have to manually split the input data, but I didn’t do it and according to nvidia-smi both of my GPUs are being used.</p>
<p>How can I make sure distributed data parallell is collecting gradients to one model and not training two separate models?</p>
Question about dataset for GRU network, future sequance prediction

Harardin: <p>I want to build model that predicts for 12 timestamps in the future movement of the graph.</p>
<p>But I am new in GRU network using.<br>
Can someone who already used GRU give me an advise. What data is better to use with features or just single one input static data.</p>
<p>Excample Dataset with features:</p>
<pre><code class="lang-auto">feature1, feature2, feature3,
77.01, 77.68, 78.91
etc.
</code></pre>
<p>Excample Dataset without features:</p>
<pre><code class="lang-auto">data
77.91
76.12
etc.
</code></pre>
Loss is nan, stopping training in MultiheadAttention

xincz: <p>I encountered ‘Loss is nan, stopping training’ when training my model with an additional multiheadAttention module. I have checked that when I’m not using the <code>if</code> block, the training is passing without error. Can anyone spot what’s causing the <code>nan</code> in this part of the code?</p>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.forward_features(x)  # [32, 198, 384]
        if self.add_attn:  # This if block is causing the `nan`
            B, N, C = x.shape  # batch_size, sequence_length, embed_dim
            x_qkv = self.add_forward(x)  # This is nn.Linear(self.embed_dim, 3*self.embed_dim)
            x_qkv = x_qkv.reshape(B, N, 3, C).permute(2, 1, 0, 3)
            x, _ = self.add_attn(x_qkv[0], x_qkv[1], x_qkv[2])  # This is nn.MultiheadAttention(embed_dim=embed_dim, num_heads=12)
            x = x.permute(1, 0, 2)
        x = self.GeM(x, axis=1)
        x = self.head(x)
        return x
</code></pre>

ariG23498: <p>Hey <a class="mention" href="/u/xincz">@xincz</a><br>
Could you please share a colab notebook of sorts where we could run the models and try debugging them?</p>
<p>With a code snippet, it is not very easy to debug or point to the issue for that matters.</p>
How to combine bidirectional LSTM sequentially

morgankohler: <p>The LSTM class includes a num_layers argument which stacks sequential LSTMs. I am wondering how to do this external of this argument. As in, I initialize several bidirectional LSTMs with num_layers=1 and want to put data through them sequentially.</p>
<p>Say the input to the LSTM is of shape [10, 16, 64] where 10 is the sequence length, 16 is the batch size, and 64 is the dimension of the input. When this is passed into a bidirectional LSTM, the output becomes [10, 16, 128] which can also be viewed as [10, 16, 2, 64] where the 2 comes from each bidirectional pass of the input.</p>
<p>Since the input to the LSTM is the same whether it is bidirectional or not, (seq_len, batch, input_size), how should I be passing the bidirectional output from the first LSTM to the second? As of now I am just passing the entire (10, 16, 128) output to the next LSTM and letting each directional workflow access the entire 128 vector from both directions of the previous LSTM. How does this work within the LSTM class itself?</p>

ariG23498: <p>Hey <a class="mention" href="/u/morgankohler">@morgankohler</a><br>
As far as I know, bi-directional LSTMs have different modes of concatenation for the hidden states. If you achieve <code>2*hidden_size</code> that means you are stacking the vectors one on top of the other. It is how the other bi/uni directional LSTM stacked above will receive their inputs.</p>
<p>Having said that, I feel the way you have organised the code is good to go.</p>
SLURM job - Problem with plot-function / save plot als png

MaxM: <p>Hello guys,</p>
<p>I am running a python script on a HPC via SLURM.<br>
I want to save a plot without displaying it.<br>
That seems to be a huge problem.<br>
My script looks like this:</p>
<p>import matplotlib.pyplot as plt<br>
import numpy as np</p>
<p>a = np.array([1,2,2,2,2,3,3,3,4,4,5])</p>
<p>plt.ioff()<br>
plt.hist(a)<br>
plt.savefig(‘testimg.png’)<br>
plt.close()</p>
<p>I thought that plt.ioff() would prevent the plt.hist(a) from creating a figure that wants to be displayed.<br>
Unfortunately I get the following error.</p>
<p>Traceback (most recent call last):<br>
File “PlotTest.py”, line 7, in <br>
plt.hist(a)<br>
File “/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/matplotlib/pyplot.py”, line 2631, in hist<br>
return gca().hist(<br>
File “/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/matplotlib/pyplot.py”, line 920, in gca<br>
return gcf().gca(**kwargs)<br>
File “/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/matplotlib/pyplot.py”, line 597, in gcf<br>
return figure()<br>
File “/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/matplotlib/pyplot.py”, line 539, in figure<br>
**kwargs)<br>
File “/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/matplotlib/backend_bases.py”, line 3252, in new_figure_manager<br>
return cls.new_figure_manager_given_figure(num, fig)<br>
File “/fibus/fs1/16/cql7772/.local/lib/python3.7/site-packages/matplotlib/backends/_backend_tk.py”, line 946, in new_figure_manager_given_figure<br>
window = tk.Tk(className=“matplotlib”)<br>
File “/fibus/fs1/16/cql7772/.conda/envs/ganomaly/lib/python3.7/tkinter/<strong>init</strong>.py”, line 2023, in <strong>init</strong><br>
self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)<br>
_tkinter.TclError: couldn’t connect to display “localhost:11.0”</p>
<p>Does anyone knows how to prevent this error? I just want to get the plot as a .png file</p>
<p>Many thanks in advance!</p>

ptrblck: <p>This issue doesn’t seem to be PyTorch-related, so you might get a better and faster answer in their support board or e.g. StackOverflow.<br>
With that being said, you could try to switch to the <code>Agg</code> backend in <code>matplotlib</code> at the beginning of the script so avoid such issues.</p>

MaxM: <p>Many thanks! Switching to the Agg backend solved my issue!</p>
Deep CNN model training

krishna511: <p>Can anyone comment on this model learning,<br>
Is it overfitting, I am already using BN, Dropout.<br>
Or should I simply reduce my epochs to approx 50, is it fine.<br>
Actually, I want to know what else I can do to improve the performance.</p>
<p>Regards<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a78326dd46357b5c9302816bf1f5dc2e20504b73.png" alt="image" data-base62-sha1="nTSJcokJQV6eaGgQgKZzEP608fN" width="386" height="262"></p>

ptrblck: <p>Yes, a perfect training accuracy and a gap to the validation accuracy would indicate overfitting.<br>
Generally you could increase the regularization through dropout, a smaller model, more aggressive data augmentation etc., but as always it depends on your use case what would work best.</p>

krishna511: <p>What  is meant by regressive augmentation, is it allowed in research paper writing? Do people mention it either. Actually during my studies I found results mismatched with mine from so many paper , are they using augmentation?<br>
I am using this for audio classification.<br>
Can I train the model after 40 epochs coz after that nothing is changed much sir?</p>
Does pytorch register modules that are assigned to object properties?

Alexander_Soare: <p>Say I have a module like:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn = CNN(params)
</code></pre>
<p>And then I do:</p>
<pre><code class="lang-auto">module = MyModule()
module.cnn = CNN(some_other_params)
</code></pre>
<p>Is the replacement registered? Will there be any nasty side effects down the line?</p>

ptrblck: <p>Yes, the assignment of the new module will properly register the new layer with all parameters and is the recommended way to e.g. replace the last linear layer in a fine-tuning use case (assuming the number of classes changed and you would like to retrain this layer). The parameters are also shown in the <code>state_dict</code> and in the <code>parameters()</code> generator:</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn = nn.Conv2d(3, 1, 3, 1, 1)

model = MyModule()
print(model.state_dict())
print(dict(model.named_parameters()))

model.cnn = nn.Linear(3, 3)
print(model.state_dict())
print(dict(model.named_parameters()))
</code></pre>
<p>You could run into “side effects”, if you’ve already created an optimizer and don’t add the newly registered parameters to it, since they wouldn’t be trained.</p>
Operands could not be broadcast together with shapes

Biswajit_Roy_ISGPP: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/752012ae345992a95060bb7c77559f12685757e0.png" data-download-href="https://discuss.pytorch.org/uploads/default/752012ae345992a95060bb7c77559f12685757e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/752012ae345992a95060bb7c77559f12685757e0_2_690x389.png" alt="image" data-base62-sha1="gI8zXLMU1ms6anNkv0A2vSKdq00" width="690" height="389" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/752012ae345992a95060bb7c77559f12685757e0_2_690x389.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/752012ae345992a95060bb7c77559f12685757e0_2_1035x583.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/5/752012ae345992a95060bb7c77559f12685757e0.png 2x" data-dominant-color="4F4F4E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1360×768 85.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
some of the images are having different  size of channels how to resize the channel and have a common channel</p>

ptrblck: <p>You could either write a custom transformation or use <code>transforms.Lambda</code> to apply your operation, which would yield the desired number of channels. Here is a simple example, which slices the channels:</p>
<pre><code class="lang-python">transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Lambda(lambda x: x[:3] if x.size(0)&gt;3 else x),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])
])


img = transforms.ToPILImage()(torch.randn(3, 24, 24))
out = transform(img)
print(out.shape)
&gt; torch.Size([3, 24, 24])

img = transforms.ToPILImage()(torch.randn(4, 24, 24))
out = transform(img)
print(out.shape)
&gt; torch.Size([3, 24, 24])
</code></pre>
Does my hook is correct?

Aman_Gupta: <p>I am using forward and backward hook in my pytorch densenet121 model.<br>
I set <code>requires_grad</code> to False at the time of training.</p>
<pre><code class="lang-auto">for param in model.features.parameters():
    param.requires_grad = False
</code></pre>
<p>and I want to get gradient of last conv layer in Neural Network , for this i define hook like this:</p>
<pre><code class="lang-auto">model.features.denseblock4.denselayer16.conv2.register_backward_hook(h1)
</code></pre>
<p>but this is giving me empty list. But is i set <code>requires_grad = True</code> for the last layer then i get the gradient.</p>
<pre><code class="lang-auto">for param in model.features.denseblock4.denselayer16.conv2.parameters():
    param.requires_grad = True
</code></pre>
<p>I use gradient for <strong>gradCAM</strong> .</p>
<p><strong>is this write way to doing or is there  any other way to get gradient?</strong></p>

ptrblck: <p>It’s expected that no gradients will be available in the hook, if you disable the gradient calculation.<br>
<code>register_backward_hook</code> is also deprecated in favor of <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_full_backward_hook"><code>register_full_backward_hook</code></a>, so you might want to change the usage.</p>
How to empty_cache by passing gpu-tensor as a parameter to a function?

oasjd7: <p>How to empty_cache by passing gpu-tensor as a parameter to a function?</p>
<pre><code class="lang-auto">In this case, It works!
x = torch.tensor([1., 2.]).cuda() 
print(torch.cuda.memory_allocated()) # 512
del x 
print(torch.cuda.memory_allocated()) # 0
</code></pre>
<pre><code class="lang-auto">x = torch.tensor([1., 2.]).cuda() 
print(torch.cuda.memory_allocated()) # 512
cleaner(x)
print(torch.cuda.memory_allocated()) # still 512 ...

def cleaner(x):
   del x


</code></pre>

ptrblck: <p>The <code>cleaner</code> method will delete the <code>x</code> variable from the local scope, not the global one.<br>
To remove the global <code>x</code>, you could use the <code>global</code> keyword:</p>
<pre><code class="lang-python">x = torch.tensor([1., 2.]).cuda() 
print(torch.cuda.memory_allocated()) # 512
del x 
print(torch.cuda.memory_allocated()) # 0

def cleaner():
    global x
    del x

x = torch.tensor([1., 2.]).cuda() 
print(torch.cuda.memory_allocated()) # 512
cleaner()
print(torch.cuda.memory_allocated()) # 0
</code></pre>
Video Feature extraction using ResNet3D pre-trained model

Chamroukhi: <p>Dear all,<br>
i’m new in Pytorch and i need to use ResNet 3D pre-trained model for video classification,<br>
in Tensorflow it’s just remove classify layer and create new head with custom classes and train the model.<br>
someone have an idea or tutoriels how to do this with Pytorch?<br>
thanks for advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

a_d: <p>Hello,<br>
you could do it as follows -</p>
<pre><code class="lang-auto">pretrained_model = torchvision.models.video.r3d_18(pretrained=True)
modules = list(pretrained_model.modules())

pretrained_sequential = nn.Sequential()

for _, module in enumerate(modules):
    pretrained_sequential.add_module('module' + str(_), module)
</code></pre>
<p>You can decide up to what layer you want to add into to your sequential block. For that you might want to see how ResNet3D is implemented in torchvision</p>

Chamroukhi: <p>i think it is necessary that remove last layer(classify) before add sequential (new head)?</p>

a_d: <p>When you use the <code>.modueles()</code> method, you get a list of all the modules present in the network, it is then up to you which ones you want to keep and which ones you don’t. You can check the implementation of the model or simply print the list to see what all is present.</p>
<p>Yes the last layer is a classification one and if you want to add another convolution block, you might have to remove it.</p>
<p>Otherwise it is not compulsory to remove it, for example, you could add another linear layer on top of the classification layer.</p>