akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Dose pytorch support join operator in SQL？

renbo_zhu: <p>Dose pytorch support <code>join</code> operator in SQL？</p>
<p>I have a tensor <code>a</code> and a tensor <code>b</code>, I want to <code>join</code> the first row of <code>a</code> and the second row of <code>b</code> to get <code>c</code>, based on the second row of <code>a</code> and the first row of <code>b</code>.</p>
<p>Dose any function support it? Or how can I achieve it with short time and small memory?</p>
<pre><code class="lang-auto">a = torch.tensor([[11, 1],
                  [12, 1],
                  [12, 2]])

b = torch.tensor([[1, 21],
                  [1, 22],
                  [2, 23]])

c = torch.tensor([[11, 21],
                  [11, 22],
                  [12, 21],
                  [12, 22],
                  [12, 23]])

c = f(a,b)
</code></pre>
Complexity of attention in dynamic convolution

SAI_VARSHITTHA: <p>I’m reading this paper , <a href="https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_Dynamic_Convolution_Attention_Over_Convolution_Kernels_CVPR_2020_paper.pdf" rel="noopener nofollow ugc">Dynamic convolution-Attention over Convolution kernels</a>.</p>
<pre><code>    I couldn't understand the complexity of attention i.e., 
</code></pre>
<p><a href="https://i.stack.imgur.com/zbZEZ.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/zbZEZ.png" alt="Pic from above paper" width="462" height="158"></a></p>
<p>How to calculate O(π(x)) ? Explain.</p>
Batched index_add_

ratishsp: <p>Hi,<br>
I wanted to have index_add_ operation applied on a batch.<br>
For instance:<br>
A = torch.FloatTensor([[[1, 2, 3], [1, 2, 3]], [[4, 5, 6], [4, 5, 6]], [[7, 8, 9], [7,8, 9]]])<br>
ind = torch.LongTensor([[0, 1], [1, 2], [0, 2]])<br>
to_add = torch.LongTensor([[[0,1],[0,1]],[[2,3],[2,3]],[[4,5],[4,5]]])</p>
<p>I would have liked a batched index_add_ operation<br>
A.index_add_(2, ind, to_add)<br>
and the expected output as:<br>
output = torch.FloatTensor([[[1, 3, 3], [1, 3, 3]], [[6, 8, 6], [6, 8, 6]], [[11, 13, 9], [11, 13, 9]]])</p>
<p>Currently I am achieving the same using a loop through each tensor in batch and using index_add_ on each tensor. I would like to avoid this loop. Please let me know if there is a more efficient way of doing the same.</p>
<p>Thanks.</p>

Poeroz: <p>Hi,</p>
<p>I just met the same problem as you, and I found <code>torch.scatter_add_()</code> is helpful! You can have a try.</p>
<p>Btw, I found the methods named <code>gather</code> and <code>scatter</code> in PyTorch can solve this kind of problems well, which can be seemed as a “universal” version of methods named <code>index*</code>.</p>
Mean calculation of angles

Angelina_Robert: <p>Hi,<br>
In a batch of 128 I have extracted the features and then found the mean, and angles between mean and corresponding class features, The dimensions of calculated angles are:<br>
[8,2,128], …[5,2,128]  where the first index shows the total number of a certain class, say, 8 classes.<br>
Can anyone help to find the mean of the angles of these dimensions?</p>

Kushaj: <p>Can you clarify more, what you want?</p>

Angelina_Robert: <p>Thanks Kushaj for ur reply,<br>
I want to find the mean of angles of different classes. To make it simple, assume that we have angles of different classes stored in in 3D tensors I.e. (10, 2,128), (13,2, 128) …etc. where 1st dimension represents the number of certain classes. 2, 128 is the dimension of angles. Now I want to iterate over all these angles to find the mean of angles.<br>
Cheers,<br>
Angelina</p>

Kushaj: <p>For the tensor (10,2,128), <code>torch.mean(tensor, dim=(1,2))</code> would give you the mean along the dim 0. Is this what you are referring to?</p>
Custom pre-trained model remove layer

edshkim98: <p>Hello everyone,<br>
I am trying to use my own pre-trained model to train another network.<br>
My model looks as below:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1a04fb318e4644e47d7fb4fb9b63223fd02f60c.png" alt="image" data-base62-sha1="rCTCgD8t2MwfcJAR6YdZUestykk" width="585" height="438"></p>
<p>I am trying to change the conv11 to Conv1d(128,15)… but how can I access to this conv11 layer?</p>
<p>self.model = nn.Sequential(*list(self.model.children())[:-1]), this will result deleting the whole model and return an empty list <span class="chcklst-box fa fa-square-o fa-fw"></span>.</p>

ptrblck: <p>I guess you could access it directly via <code>model.conv11</code>.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

edshkim98: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for the help!<br>
Your solution worked out nicely</p>
Concat tensor but mixing order

111422: <pre><code class="lang-auto">import torch

import numpy as np


x = torch.rand(4,4)

y = torch.rand(4,4)

print(x)

print(y)

print(torch.cat((x,y),0))
</code></pre>
<p>If I execute this code it will be like</p>
<pre><code class="lang-auto">tensor([[0.1211, 0.7577, 0.8376, 0.4488],
        [0.6628, 0.8632, 0.9736, 0.8368],
        [0.5848, 0.0872, 0.0469, 0.2834],
        [0.8561, 0.6229, 0.3667, 0.1358]])
tensor([[0.1820, 0.8955, 0.3811, 0.5496],
        [0.5745, 0.3199, 0.5228, 0.8269],
        [0.8408, 0.6984, 0.7248, 0.1973],
        [0.3045, 0.9651, 0.8701, 0.3180]])
tensor([[0.1211, 0.7577, 0.8376, 0.4488],
        [0.6628, 0.8632, 0.9736, 0.8368],
        [0.5848, 0.0872, 0.0469, 0.2834],
        [0.8561, 0.6229, 0.3667, 0.1358],
        [0.1820, 0.8955, 0.3811, 0.5496],
        [0.5745, 0.3199, 0.5228, 0.8269],
        [0.8408, 0.6984, 0.7248, 0.1973],
        [0.3045, 0.9651, 0.8701, 0.3180]])
</code></pre>
<p>is there a way that I coult concat like</p>
<p>tensor([[0.1211, 0.7577, 0.8376, 0.4488],<br>
[[0.1820, 0.8955, 0.3811, 0.5496],<br>
[0.6628, 0.8632, 0.9736, 0.8368],<br>
…<br>
which is x row 1 comes first<br>
and y row 1 and x row 2 and so on</p>
<p>Thank you for reading</p>

111422: <p>I think I solved it</p>
<pre><code class="lang-auto">for  k in range(x.shape[0]):
  if test is None:
    test = torch.cat((z[k:k+1],z[k + x.shape[0]:k + x.shape[0]+1]),0)
    print(test)
  else:
    test = torch.cat((test,z[k:k+1],z[k + x.shape[0]:k + x.shape[0]+1]),0)
</code></pre>
<p>I think this does the job</p>
<p>thank you</p>

ptrblck: <p>Alternatively to the loop, you could also interleave the tensors directly as seen here:</p>
<pre><code class="lang-python">x = torch.rand(4,4)
y = torch.rand(4,4)
print(x)
print(y)

z = torch.cat((x,y),0)
test = None
for  k in range(x.shape[0]):
  if test is None:
    test = torch.cat((z[k:k+1],z[k + x.shape[0]:k + x.shape[0]+1]),0)
    print(test)
  else:
    test = torch.cat((test,z[k:k+1],z[k + x.shape[0]:k + x.shape[0]+1]),0)
print(test)

res = torch.cat((x.unsqueeze(1), y.unsqueeze(1)), 1).view(8, 4)
print((test == res).all())
&gt; tensor(True)
</code></pre>
Loss is not decreasing for 3D CNN Encoder-Decoder

Preetham_R_Patlolla: <p>Hello,</p>
<p>I’ve been trying to work on encoding the Spatio-temporal features using 3D CNN’s. I’m passing the sequence of images of shape (16,22,3,28,28) ----&gt; (batch, sequence_length, channels, height, width) to the encoder-decoder model and trying to generate the compressed vector space that represents the sequence of images. But the network isn’t learning/converging. As in, the loss isn’t decreasing. Sometimes, I also get nan for loss.</p>
<p>About the data I’m using:<br>
I’m using synthetic data and I have generated 2000 image sequences (2000,22,3,28,28). I have augmented it by 7 times using the techniques like injecting random Gaussian noise, cropping, changing the contrast and color space, etc making it (14000,22,3,28,28).</p>
<p>About the model:<br>
It is an encoder-decoder architecture with 3D convolutional layers (5 in encoder and 5 in decoder).</p>
<p>Loss: MSE loss<br>
Optimizer: AdamW</p>
<p>I have tried playing with the learning rate, using dropout layers, normalization layers, changing the dimension of the compressed vector space of the encoder. But nothing helped so far. Please find my model and code below. Your help and suggestions will be appreciated.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201120-085602-RC00_343491535" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/1CFxjBR9zUs0Rtr7gUgrlIAfcWeNmI7YT?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1CFxjBR9zUs0Rtr7gUgrlIAfcWeNmI7YT?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Data Parallel isn't working on submodules

Tejas_P: <p>I get a device mismatch error when using Data-Parallel for training with multiple GPUs.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7ac00e3412f7cbd47be4e0072acf28ff86c4f47.png" data-download-href="https://discuss.pytorch.org/uploads/default/a7ac00e3412f7cbd47be4e0072acf28ff86c4f47" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7ac00e3412f7cbd47be4e0072acf28ff86c4f47_2_690x27.png" alt="image" data-base62-sha1="nVifHFfh6U1K27RyRt6HoRiAMjd" width="690" height="27" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7ac00e3412f7cbd47be4e0072acf28ff86c4f47_2_690x27.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7ac00e3412f7cbd47be4e0072acf28ff86c4f47_2_1035x40.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7ac00e3412f7cbd47be4e0072acf28ff86c4f47_2_1380x54.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1758×70 50.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>After debugging I got to know that the data-parallel module doesn’t work with submodules.The model essentially is inceptionnet_v1/googlenet.  My model design template is as below:</p>
<p>Class Submodule(nn.Module):</p>
<pre><code>def __init__():

      super .......

      self.a = nn.Seq(.......)

      self.b = nn.Seq(.......)

   ..........

 def forward(self,x):

     return torch.cat([self.a(x)......])
</code></pre>
<p>Class Model(nn.Module):</p>
<pre><code> def __init__():

      super .......

      self.conv = conv

      self.sublayer1 = Submodule(....)

      self.sublayer2 = Submodule(....)

      .......

def forward(self,x):

     x= self.conv(x)

     x = self.sublayer1(x)

     x = self.sublayer2 (x)

     ........

return x
</code></pre>
<p>model = Model()</p>
<p>model = nn.DataParallel(model).cuda()</p>
<p>The DataParallel Works with Conv layers and not sublayers. It would be great if anyone can give me a pointer to debug this. I couldn’t find support from previous Pytorch Forums.</p>

JuanFMontesinos: <p>It does work with submodules.<br>
The problem is you probably have some hardcoded device in the submodules.<br>
This is calling <code>.cuda()</code> instead of coding it as <code>.to(device)</code> being device based on forward’s input.</p>

Tejas_P: <p>I tried changing it to .to(device) and have the same error. The problem here is I’m trying to work with DataParallel method and I think this method is not used by the submodule.</p>

JuanFMontesinos: <p>Can you post the model code?<br>
The submodule has to be a nn.Module, a nn.modulelist or torch dict.<br>
Any other object is not properly traced by the system.</p>
Tutor/consultant

AlexSwiderski: <p>Hi everyone,</p>
<p>I am a graduate student working with a functioning model, which utilizes word embeddings as inputs and sequences of letters as output.  I am looking to work with someone to essentially fact check that my model is doing what I intended it to do, and then also help me develop a skill set to analyze model weights analyze the output of the network.</p>
<p>I am happy to reimburse you for your time to a reasonable extent as well.  Please email me at aswiderski@pitt.edu should you be interested.</p>
How to check for residual connection or any custom code in unknown module?

marvosyntactical: <p>Im quantizing nn.Modules (with my own API, not torch.quantization). I want to train black box user modules and convert them by simply replacing some of the modules afterwards. So far so good, but I don’t know how to properly access the control flow, which I need to to know for architectures that are not only sequential: For example, if there are residual adding connections in the forward pass, for example like so:</p>
<pre><code class="lang-auto">def forward(self, x):
    x = F.relu(self.fc1(x))
    x = self.sequentialstack(x) + x
    x = self.softmax(x)
    return x
</code></pre>
<p>which Id like to replace by something like</p>
<pre><code class="lang-auto">def forward(self, x):
    x = F.relu(self.fc1_quant(x))
    x_residual_cache = x
    x = self.sequentialstack_quant(x)
    x = self.quantized_adder(x, x_residual_cache)
    x = self.softmax_quant(x)
    return x
</code></pre>
<p>What is the recommended way to do this? I have read the <a href="https://pytorch.org/docs/stable/jit.html" rel="noopener nofollow ugc">Intro to Torchscript</a> but dont see immediately what the recommended way to find out about the internal control flow is; so before I throw regexes at scripted_model.forward.code, I thought I’d ask what the right way to do this is.</p>

marvosyntactical: <p>I found <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html" rel="noopener nofollow ugc">this quantization tutorial</a>; the InvertedResidual class is basically what I want. So I will just change my model definition instead of modifying model code.</p>
How to save trained images without a burden on network

hbolandi: <p>I am training a convolutional neural network in pytorch and want to save trained images. I append each trained image in a data loader loop to save all the trained images into numpy file (train_pred in code below) and works properly. But it is a huge burden on the network and increase running time. Is there any other way to do this?</p>
<pre><code class="lang-auto">for epoch in range(epochs):
    mse_train_losses= []
    N_train = []
    
    train_pred=[]     
    model.train()
    for data in train_loader:
     
        x_train_batch, y_train_batch = data[0].to(device, 
            dtype=torch.float), data[1].to(device, dtype=torch.float)  

        y_train_pred = model(x_train_batch)          
        mse_train_loss = criterion(y_train_batch, y_train_pred, x_train_batch, mse)  
        
        optimizer.zero_grad()                   
        mse_train_loss.backward()                         
        optimizer.step()                       
        
        mse_train_losses.append(mse_train_loss.item())
       
        N_train.append(len(x_train_batch))
        
        train_pred.append(y_train_pred)
        train_pred_de=torch.stack(train_pred).cpu().detach().numpy()
</code></pre>

ptrblck: <p>Since you are appending the model output (<code>y_train_pred</code>) inside the training loop directly, the computation graph would also be stored, so you might want to <code>detach()</code> it before appending it to the list.</p>
<p>Also, the <code>cpu()</code> operation is synchronizing your code which might result in a slowdown.<br>
There is unfortunately no workaround, since the values have to be already calculated if you want to push them to the CPU and convert them to a numpy array.</p>
RuntimeError: Function 'CudnnBatchNormBackward' returned nan values in its 0th output in 5-fold cross-validation

maferoaloaiza: <p>Hi, I am currently working on a CT scan classification dataset with 5-fold cross-validation. The images go through an online preprocessing stage before entering the backbone to reduce their size, so the network fits in the GPU.  When I train the network from scratch, there are no issues. However, when I use pre-trained weights to make finetuning in the backbone, I have an error in the second fold. I am currently using PyTorch 1.6 with amp. The first fold trains without any problem, but the second one cannot. I have tried changing the fold that trains first, but the second one, no matter what fold it is, shows this error:</p>
<p>Traceback (most recent call last):<br>
File “main.py”, line 399, in <br>
main()<br>
File “main.py”, line 190, in main<br>
bce, scaler)<br>
File “main.py”, line 301, in train<br>
scaler.scale(loss).backward()<br>
File “/home/mfroa/anaconda3/envs/lung/lib/python3.7/site-packages/torch/tensor.py”, line 184, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/mfroa/anaconda3/envs/lung/lib/python3.7/site-packages/torch/autograd/init.py”, line 115, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: Function ‘CudnnBatchNormBackward’ returned nan values in its 0th output.</p>

ptrblck: <p>Are you using anomaly detection in your code?<br>
If so, could you disable it if you are using <code>amp</code>, since NaN gradients are expected to occur sometimes (especially during the first iterations before the <code>GradScaler</code> stabilizes the scaling factor)?</p>

maferoaloaiza: <p>Hi, I was using anomaly detection. I deactivated it, and it seems to work just fine. Thank you for your help. I was wondering, when is it recommended to use anomaly detection?</p>

ptrblck: <p>You can use it in case you are seeing invalid outputs or gradients, so could this be an issue for your use case?<br>
I mentioned mixed-precision training (<code>torch.cuda.amp</code>), since gradients might overflow and the <code>GradScaler</code> would skip these updates. These iterations might however trigger the anomaly detection, so you should activate anomaly detection once the loss scaling factor is stable.</p>
<p>If you are not using <code>torch.cuda.amp</code>, then something might still be broken in your code and I would like to debug it in this case.</p>

maferoaloaiza: <p>Thank you for your answer. I am using the torch.cuda.amp, so I guess the error is related to the first iterations, as you mention. I haven’t seen invalid outputs or gradients, so I think this isn’t an issue neither.</p>
Sharing variable with computation graph between processes

tetratrio: <p>Hey</p>
<p>Is it possible to use a buffer for sending variables (with computation graphs) over a multiprocessing queue? I am currently gathering log probability variables for policy gradient with multiple child processes and the bottleneck at the moment is transferring the variables over to the parent process.</p>
<p>How can I make this transfer faster? If it was normal tensors I could just use a buffer as I would know the size but with variables that contain computation graphs I dont know how to do it.</p>

richard: <p>Are you planning to backprop through those Variables?</p>

tetratrio: <p>Yes, so I need the computation graph. Just getting the data would not be enough!</p>

leonardo0lyj: <p>Any luck <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">? I am also having the same issue. Really appreciated!</p>

tetratrio: <p>This was so long ago I dont remember if I was able to solve it. Last time I used policy gradients I went with the hogwild style of training, where instead I share the weights over multiple processes and each process can update the weights asynchronously.</p>
<p>Another idea would be to use rpc. The only experience I have is with gRPC where each servicer can run in its own process, maybe that can be used in a similar way? <a href="https://pytorch.org/docs/stable/rpc.html" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/rpc.html</a></p>
Integration issue with opencv

chinmay5: <p>I have a transforms class which only does:</p>
<pre><code class="lang-auto">        if transform is None:
            transform = transforms.Compose([
                transforms.Resize((256, 256)),
                transforms.ToTensor()
            ])
        root = os.path.join(PROJECT_ROOT_DIR, "data")
        super(AttributesDataset, self).__init__()
        self.data = torchvision.datasets.CelebA(
            root=root,
            split=split,
            target_type='attr',
            download=True,
            transform=transform
        )
</code></pre>
<p>I want to visualize some of the outputs coming from the model. As such, I created a simple method which does:-</p>
<pre><code class="lang-auto">    for img, label in dataloader:
        img.squeeze_(0)
        # permute the channels. cv2 expects image in format (h, w, c)
        unscaled_img = img.permute(1, 2, 0)
        # move images to cpu and convert to numpy as required by cv2 library
        unscaled_img = torch.round(unscaled_img * 255)
        unscaled_img = unscaled_img.to(torch.uint8)
        # unscaled_img = np.rint(unscaled_img * 255).astype(np.uint8) 
        unscaled_img = cv2.cvtColor(unscaled_img, cv2.COLOR_RGB2BGR)
        cv2.imshow(unscaled_img.numpy())
</code></pre>
<p>However, all the images that are created have an unusually blue shade. For instance,</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f7ee0bd91eb0410d737f3c5db3aae4c04e321a9a.jpeg" alt="image" data-base62-sha1="znhWIinlsB51yvqvbVTkP8840em" width="278" height="283"></p>
<p>Can someone please tell me what exactly am I doing wrong here? Your help would be highly appreciated</p>

chinmay5: <p>The culprit was the line</p>
<pre><code class="lang-auto">unscaled_img = cv2.cvtColor(unscaled_img, cv2.COLOR_RGB2BGR)
</code></pre>
<p>Removing it solved the issue</p>
How to access source code of specific function?

fermoren: <p>Hello, I’m using the function torch.conj on my code and I need to run it on a server which uses Pytorch 1.3.0, which does not implement this function (it does not depend on me to update the pytorch version used on the server).</p>
<p>The first option that comes to my mind in order to be able to use torch.conj on my code is to copy/paste the source code on my own model but I have not been capable to find it on the Github repository, could anyone help me to be able to use this function on pytorch 1.3.0?</p>
<p>Thanks!</p>
What type hints do I use for a __torch_function__ function?

ProGamerGov: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/notes/extending.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/notes/extending.html" target="_blank" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">    @classmethod
    def __torch_function__(cls, func, types, args=(), kwargs=None):
        logging.info(f"func: {func.__name__}, args: {args!r}, kwargs: {kwargs!r}")
        if kwargs is None:
            kwargs = {}
        return super().__torch_function__(func, types, args, kwargs)
</code></pre>
<p>What type hints do I use for the arguments, and what type hint do I use for the return type: <code>-&gt; ReturnType</code>?</p>

ProGamerGov: <p>After some testing, I think this is how you properly type hint <code>__torch_function__</code>:</p>
<pre><code class="lang-auto">from typing import Any, Callable, Dict, Optional, Tuple

class LoggingTensor(torch.Tensor):

    @classmethod
    def __torch_function__(cls, func: Callable, types: Tuple, args: Tuple = (), kwargs: Optional[Dict] = None) -&gt; Any:
        if kwargs is None:
            kwargs = {}
        return super().__torch_function__(func, types, args, kwargs)
</code></pre>
Backpropagation through two concatenated networks

Prakhar_Pradhan: <p>Hello Everyone !<br>
I am doing one problem which aims in improving the semantic segmentation pixelwise results. The approach requires two networks (1. CNN 2. MLP), which are concatenated. The output of net1 i.e. CNN is 3 numbers , which are then serves as input to net2 i.e. MLP. The output of net2 (MLP) is a single number (not an array or higher dimensional tensor), at the output of net2, MSE Loss is calculated.<br>
Now I have to backpropagate and update the gradients of both networks using the calculated MSE loss. The backpropagation of net2 is straight forward, i.e. loss.backward(), but how to do backpropagation of net1 using the gradients of net2 ?</p>

Megh_Bhalerao: <p>When you do<code> loss.backward()</code>, the gradients are calculated for both the networks and stored along with the parameters, assuming you are doing something like <code>net2(net1(input))</code>. Then according to how your optimizer is defined, when you do <code>optimizer.step()</code> the weights will get updated.</p>

Prakhar_Pradhan: <p><a class="mention" href="/u/megh_bhalerao">@Megh_Bhalerao</a><br>
Thank you for answering. Please see the overview of the process<br>
x=net1(input)<br>
x1 = f(x) {some processing, means doing some summation operation, to calculate the input of net2}<br>
y=net2(x1)<br>
loss = |y-gt| {gt is ground truth}<br>
loss.backward()<br>
so, does this loss.backward(), can take care of calculation of gradients of both the networks. In optimizer, the parameters of both the networks will be passed.</p>

Megh_Bhalerao: <p>Yes, to the best of my knowledge, because there is no reason the computational graph should break by doing the <code>f(x)</code>. Yes, if you pass both the model’s parameters to the <code>optimizer</code>, the the <code>loss.backward()</code> will calculate the gradients of both the networks.</p>
Batch size > 1 for adversarial example

chinmay5: <p>The tutorial notebook on <a href="https://github.com/pytorch/tutorials/blob/master/beginner_source/fgsm_tutorial.py" rel="noopener nofollow ugc">Adverserial Training</a> uses a batch size of 1. What code changes are needed if we want to train on a batch size of say 16. My understanding is, we only need to change the logic of<br>
<code>final_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability</code><br>
<code> # Now we have batch size &gt; 1</code><br>
<code>final_pred.squeeze_()</code><br>
<code>indexes = final_pred == target</code><br>
<code>correct += torch.sum(indexes).item()</code></p>
<p>Is there something else needed. With this code change, I get values that are very similar to the case of batch_size=1 although not the same values. Any help would be appreciated.</p>
How the dim parameter of the Softmax() is reflected in CrossEntropyLoss()

Lincoln10153: <p>Hello,</p>
<p>I am new to PyTorch, and I encountered a quesiton about Softmax() and CrossEntropyLoss().</p>
<p>In a multi-classification task, I set dim=1 in Softmax(). <strong>I wanna know if I need to set the similar parameter in CrossEntropyLoss().</strong> However, I did not find the similar parameter as dim in softmax(). <strong>And how the dim parameter of the Softmax() is reflected in CrossEntropyLoss()?</strong></p>
<p>Thanks!</p>

Lincoln10153: <p>One of my guesses is that “dim” is set to a default value. For example, the default value is 1.</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects the class dimension in <code>dim1</code> as explained <a href="https://discuss.pytorch.org/t/multi-class-cross-entropy-loss-and-softmax-in-pytorch/24920/9">here</a>.</p>
register_forward_hook

Nazaninm: <p>I have a question about “register_forward_hook”. Part of my code is as follow,</p>
<p>def hook(module, input, output):<br>
pass</p>
<p>with torch.no_grad():<br>
model.layer3[0].conv2.register_forward_hook(hook)<br>
embed=model(torch.unsqueeze(image_, 0))<br>
It does not give me the intermediate feature. I am new to pytorch and I am not sure what is the problem.</p>

ptrblck: <p>Your current <code>hook</code> function doesn’t do anything with the activation, so you could add e.g. a <code>print</code> statement:</p>
<pre><code class="lang-python">def hook(module, input, output):
    print(output.shape)

model = nn.Conv2d(1, 1, 3)

with torch.no_grad():
    model.register_forward_hook(hook)
    embed = model(torch.randn(1, 1, 4, 4))
</code></pre>
How to know the size of the data after loading into data loader?

Muhammad4hmed: <p>I loaded the dataset using data loader as follows:</p>
<pre><code class="lang-auto">data_loader = torchvision.datasets.ImageFolder('/content/drive/My Drive/Dataset/malimg_paper_dataset_imgs',
                                                 transform = torchvision.transforms.Compose([
                                                              torchvision.transforms.Resize((224, 224)),
                                                              torchvision.transforms.ToTensor(),
                                                              torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                                                    std=[0.229, 0.224, 0.225])]))
</code></pre>
<p>The original dataset’s size is 1.1GB but in the data loader, I have applied resizing and normalization, now I want to know what will be the size of data that is loaded into the data loader. I can’t find anything related in the documentations. Thanks</p>

ptrblck: <p>You are not creating a <code>DataLoader</code>, but an <code>ImageFolder</code> which is a <code>Dataset</code>.<br>
The <code>ImageFolder</code> dataset will lazily load and process each sample. If you wrap it into a <code>DataLoader</code> via:</p>
<pre><code class="lang-python">dataset = ImageFolder(...)
loader = DataLoader(dataset, batch_size=..., num_workers=...)
</code></pre>
<p>each worker of the <code>loader</code> will load a complete batch and add it to a queue.<br>
By default the <code>prefetch_factor</code> in the <code>DataLoader</code> is set to <code>2</code>, which will load <code>2*num_workers</code> batches.</p>

Muhammad4hmed: <p>I did that, after that how can I know the size of loader where complete data is stored in the form of batches?</p>

ptrblck: <p>You can get the number of samples via <code>len(dataset)</code> and the number of batches via <code>len(loader)</code>.</p>

Muhammad4hmed: <p>yes but I don’t want to know the length of samples or batches. I want to know the memory size those samples are holding.</p>

ptrblck: <p>If you are lazily loading the data, only <code>2*num_workers*batch_size</code> will be loaded into the RAM. The rest will stay on the drive. You can check the shape of a batch size and calculate the needed RAM manually using the data type of your tensors.</p>
Keras to Pytorch

algol1: <p>I’m trying to build a model to classify MNIST.<br>
I saw this website: <a href="https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist" rel="noopener nofollow ugc">https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist</a><br>
And tried to copy what they did, besides the fact I only want 1 CNN and not 15.</p>
<p>But the results I get have very different accuracy, so I got to be missing something.<br>
Here is my code:</p>
<pre><code class="lang-auto">class Net(nn.Module):
     def __init__(self):
         super(Net, self).__init__()
         self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
         self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
         self.conv2_drop = nn.Dropout2d()
         self.fc1 = nn.Linear(320, 50)
         self.fc2 = nn.Linear(50, 10)

     def forward(self, x):
         x = F.relu(F.max_pool2d(self.conv1(x), 2))
         x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
         x = x.view(-1, 320)
         x = F.relu(self.fc1(x))
         x = F.dropout(x, training=self.training)
         x = self.fc2(x)
         return F.log_softmax(x)
</code></pre>
<p>Can you please help me convert?</p>

ptrblck: <p>Based on the notebook it seems the author create CNNs containing 7 conv-bn blocks and a final linear layer, while your current model seems to be quite different.<br>
Am I missing the “right” model implementation or is the posted code not supposed to replicate the Keras model?</p>
Is there a framework for managing, running, and recording experiments with different architectures and hyperparameters?

agt: <p>Is there a recommended framework for managing, running, and recording experiments with different architectures and hyperparameters, or should I roll one from scratch for whatever I’m doing?</p>
<p>I mean a free and open source framework like pytorch.</p>

ptrblck: <p><a href="https://optuna.org/">Optuna</a> or <a href="https://discuss.pytorch.org/t/hyperparameter-optimization-for-pytorch-models/85128">Weights &amp; Biases</a> would be two frameworks which come to my mind.</p>
When broadcasting tensors:RuntimeError: The size of tensor a (12) must match the size of tensor b (3) at non-singleton dimension 2

enterthevoidf22: <p>i want to standardize images.<br>
let’s say i have a tensor of shape [20000,3,12,12].</p>
<pre><code class="lang-auto">#images is my tensor of said shape
images = (images - images.mean(dim = [1,2,3])) / images.std(dim=[1,2,3])
RuntimeError: The size of tensor a (12) must match the size of tensor b (20000) at non-singleton dimension 2
</code></pre>
<p>follow up question:<br>
if i want to standardize each color matrix separately, meaning for each image i extract mean and std of each color and subtract by that separately - how do i do that?</p>
<p>i tried using the built-in normalization transform but got another error i couldn’t pass -</p>
<pre><code class="lang-auto">img = img.to(dtype=torch.float64)
img_mean = img.mean(dim=[1,2])
img_std = img.std(dim=[1,2])
img = TF.normalize(img,mean=[img_mean[0],img_mean[1],img_mean[2]],std=[img_std[0],img_std[1],img_std[2]])
        return {'image':img,'target':target}
</code></pre>
<p>ValueError: std evaluated to zero after conversion to torch.float64, leading to division by zero.</p>
<p>most searches advise changing the dtype if the image but as can be seen i did that and still the error occurs</p>

ptrblck: <p>You can pass the <code>keepdims</code> argument to use broadcasting:</p>
<pre data-code-wrap="python"><code class="lang-python">images = (images - images.mean(dim = [1,2,3], keepdims=True)) / images.std(dim=[1,2,3], keepdims=True)
</code></pre>
<aside class="quote no-group" data-username="enterthevoidf22" data-post="1" data-topic="103628">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/enterthevoidf22/48/29338_2.png" class="avatar"> enterthevoidf22:</div>
<blockquote>
<p>if i want to standardize each color matrix separately, meaning for each image i extract mean and std of each color and subtract by that separately - how do i do that?</p>
</blockquote>
</aside>
<p>Wouldn’t that be the standard normalization applied in batchnorm layers?<br>
If so, you could use your code and use <code>dim=[0, 2, 3]</code> instead.</p>
How to accelerate resize for video data processing

KaiHoo: <p>Hello, I am doing a project related to video. I found video data processing may be the main run time bottleneck.<br>
If I use a batch size of 128, each one is a sequence  of 32 images. Then I have to resize 4096 images. If I use PIL.Image or opencv to resize, they do not support batch resize (so they are very slow). If I use torch.nn.functional.interpolate, I need to convert images to float32. As I understand it, we do not need such a high precision for resize, uint8 or float16 may be enough.</p>
<p>If I use more num_workers, the shared memory is not enough (DDP mode, 8x10=80workers). My machine has 100G shm and the training can use 70-90G shm.</p>
<p>Any suggestions? Thanks!</p>

ptrblck: <p>Since your server seems quite big and you are also using DDP, you could have a look at <a href="https://developer.nvidia.com/DALI">DALI</a>, which provides video operators.</p>
How to edit in U-2-Net to load in my own images and labels?

rui_zhou: <p>I got  train images:  0 train labels:  0<br>
I set my folders like this:<br>
––DUTS-TR<br>
——im_aug<br>
———0.jpg<br>
———1.png<br>
——gt_aug<br>
———0.png<br>
———1.png<br>
is there something wrong?</p>
<p><code>ValueError: num_samples should be a positive integer value, but got num_samples=0</code></p>
<p>what should I edit, any guidance will be helpful</p>
<h2>`data_dir = os.path.join(os.getcwd(), ‘train_data’ + os.sep)</h2>
<p>| tra_image_dir = os.path.join(‘DUTS’, ‘DUTS-TR’, ‘DUTS-TR’, ‘im_aug’ + os.sep)<br>
| tra_label_dir = os.path.join(‘DUTS’, ‘DUTS-TR’, ‘DUTS-TR’, ‘gt_aug’ + os.sep)`</p>

ptrblck: <p>I assume you are using <code>ImageFolder</code>, which won’t work in your use case, as it’ll create the targets for a multi-class classification, while you seem to be dealing with a segmentation use case.<br>
If that’s the case, I would recommend to write a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a>.<br>
In the <code>__init__</code> you could define the image paths to the data and target images and load the corresponding pair in the <code>__getitem__</code> method.</p>