akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Masking Rows of Tensor

Jack_Rolph: <p>I have a tensor, <code>A</code> of shape <code>[X,Y,Z]</code> and an indexing tensor, <code>B</code>, of <code>[X,Y,1]</code>. I would like to index <code>A</code> with <code>B</code>, and replace those elements of <code>A</code> with 0.</p>
<p><code>B</code> is the output of a neural network, with the desired result that it is either true or false for each <code>Y</code>.</p>
<pre><code class="lang-auto">r1 = -10
r2 = 10  #some ranges

X = 20
Y=100
Z=5

activation = nn.Tanh() #an activation function
 
#A is an arbitrary random tensor [0,1]
A = torch.FloatTensor(X,Y,Z).uniform_(0, 1)

#B is an arbitrary random tensor [-1,1]
B = activation(torch.FloatTensor(X,Y,1).uniform_(r1, r2))

#idx are where B is less than 0
idx = (B&lt;0)

#Some function to mask A with B such that where B is True, A=0
A = foo(A,B)
</code></pre>
<p>What is the correct function, <code>foo</code>, required such that <code>A</code> is both masked with zero and differentiable?</p>
Conv2D or ConvTranspose2D

Marcel_Iten: <p>I have a CNN for image reconstruction. My Input is a low resolution image resized to the same dimension as the high resolution ouput. Should i use Conv2D or ConvTranspose2D?</p>

matthewleigh: <p>Deconvolution (ConvTranspose2D) can result in some unwanted artefacts in the final images.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://distill.pub/2016/deconv-checkerboard/">
  <header class="source">
      <img class="site-icon">

      <a href="https://distill.pub/2016/deconv-checkerboard/" target="_blank" rel="noopener nofollow ugc">Distill</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:560/295;"><img src="http://distill.pub/2016/deconv-checkerboard/thumbnail.jpg" class="thumbnail" width="690" height="363"></div>

<h3><a href="https://distill.pub/2016/deconv-checkerboard/" target="_blank" rel="noopener nofollow ugc">Deconvolution and Checkerboard Artifacts</a></h3>

  <p>When we look very closely at images generated by neural networks, we often see a strange checkerboard pattern of artifacts.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Most state of the art image generation networks (see stylegan or progan) have dedicated upscaling followed by non-resizing convolution (stride-1) layers.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b94b26812909f14048753ed15af0ddc5add2bd1c.png" data-download-href="https://discuss.pytorch.org/uploads/default/b94b26812909f14048753ed15af0ddc5add2bd1c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b94b26812909f14048753ed15af0ddc5add2bd1c_2_517x295.png" alt="image" data-base62-sha1="qrbl8XsTaKG3rn4c7uoN7WLd2LG" width="517" height="295" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b94b26812909f14048753ed15af0ddc5add2bd1c_2_517x295.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b94b26812909f14048753ed15af0ddc5add2bd1c_2_775x442.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b94b26812909f14048753ed15af0ddc5add2bd1c.png 2x" data-dominant-color="EFEFEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">875×500 58 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<a href="https://towardsdatascience.com/progan-how-nvidia-generated-images-of-unprecedented-quality-51c98ec2cbd2" class="onebox" target="_blank" rel="noopener nofollow ugc">https://towardsdatascience.com/progan-how-nvidia-generated-images-of-unprecedented-quality-51c98ec2cbd2</a></p>
<p>I would suggest something like this (say we want to double the image resolution):</p>
<pre><code class="lang-auto">nn.Upsample(scale_factor=2, mode="nearest" )
nn.Conv2d( in_channels=C_in, out_channels=C_out, kernel_size=3, stride=1, padding=1 )
</code></pre>
Why doesn't torch.utils.checkpoint.checkpoint work with torch.autograd.grad? Is there a work-around?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I just had a quick question to ask about <code>torch.utils.checkpoint.checkpoint</code> and how it relates to <code>torch.autograd.grad</code>. Within the <a href="https://pytorch.org/docs/stable/checkpoint.html?highlight=checkpoint#torch.utils.checkpoint.checkpoint" rel="noopener nofollow ugc">docs</a> it states that <code>torch.utils.checkpoint.checkpoint</code> doesn’t work with <code>torch.autograd.grad</code> and only works with <code>torch.autograd.backward()</code>.</p>
<p>Could I ask why that’s the case and if there’s any work-around? In my current use case I have a loss function which has a component that uses <code>torch.autograd.grad</code> but I do <code>detach()</code> those values and therefore don’t need to backprop through those values. However, when I use checkpointing I get the error stated within the docs.</p>
<pre><code class="lang-auto">RuntimeError: Checkpointing is not compatible with .grad(), please use .backward() if possible
</code></pre>
<p>Might it be possible to add the checkpointing after using <code>torch.autograd.grad</code>? So, in my case, I have a loss function that contains <code>torch.autograd.grad</code> terms but I do a trick that allows me to detach those values and backdrop through a different graph that gives me the same derivatives as if I were to backprop through the original <code>torch.autograd.grad</code> graph. Might it be possible to add and remove checkpointing modules like module hooks? Or is checkpointing with <code>torch.autograd.grad</code> completely out of the question even with the ‘trick’ I mentioned above?</p>
<p>Any help is appreciated! Thank you in advance!</p>
Pytorch LBFGS with implicit direct gradient calculation

zezo: <p>I am using pytorch to optimize a certain function <code>f(x)</code> . SO, I created a function <code>f</code> , That gives me the gradient as following:</p>
<pre><code class="lang-auto">f  = lambda x: gradient(x)
"gradient (x) is a function to calculate the gradient"
</code></pre>
<p>Then, I try to use <code>LBFGS</code> to optimize this function as follwoing:</p>
<pre><code class="lang-auto">opt = optim.LBFGS([X],lr=.1)
for _ in range(1000):
    def closure():
        opt.zero_grad()
        X.grad = -f(X)
        return X
    opt.step(closure)
</code></pre>
<p>But, I got this error:</p>
<pre><code class="lang-auto">RuntimeError: assigned grad has data of a different size
</code></pre>
<p>How can I solve this issue, with the same setup.</p>

albanD: <p>Hi,</p>
<p>From the error message, it looks like the gradient computed by your <code>f(X)</code> is not of the same size as <code>X</code>. Could you verify that?</p>

zezo: <p>Hi, no it’s same size. I used other optiimizers and all works, except LBFGS, since it needs a sort of closure.<br>
X   = m0.clone()<br>
gr  = lambda x: f.gradient(x)<br>
opt = optim.Adam([X], lr=1e-2)<br>
for _ in range(5000):<br>
opt.zero_grad()<br>
X.grad = -gr(X)<br>
opt.step()</p>
<p>This Adam, and all other optimizers work</p>

albanD: <p>Can you share the stack trace? The error  is raised at line <code>X.grad = -f(X)</code> right?<br>
If so, then I would add prints here.<br>
Keep in mind that LBFGS (contrary to all other optimizers) actually evaluate the gradients multiple times after updating the value of your X.</p>

zezo: <p>yes, That’s why i need to find a solution to bypass this issue.</p>

albanD: <p>So it is not returning gradients with the same size in subsequent calls?</p>
<p>Maybe if you could share a code sample that reproduces the issue that would help debugging.</p>

zezo: <p>Thanks <a class="mention" href="/u/alband">@AlbanD</a>. I found a nice solution <a href="https://gist.github.com/tuelwer/0b52817e9b6251d940fd8e2921ec5e20" rel="noopener nofollow ugc">pytorch L-BFGS</a>. I believe we need to call the function and use backward().<br>
Thanks for your help</p>
Guided Gradcam on 3D images

Roshni_Pathak: <p>I am using pytorch lightning and I am want to visualize my validation data using GradCAM.I am using a pretrained Resnet 3d model and this is the code I wrote:<br>
my_model.eval()<br>
guided_gc = GuidedGradCam(my_model, my_model.model.fc)<br>
input_image = next(iter(mri_dataset.val_dataloader()))<br>
attribution = guided_gc.attribute(input_image, 1)<br>
Error:<br>
assert isinstance(<br>
AssertionError: <code>inputs</code> must have type torch.Tensor but &lt;class ‘dict’&gt; found:<br>
my val_dataloader is a list of dictionary and I am unable to pass it properly. Kindly suggest how should I go about it and how can I also view the output.</p>

vivekm: <p>Hi <a class="mention" href="/u/roshni_pathak">@Roshni_Pathak</a> , inputs must be provided as a tensor or tuple of tensors to appropriately compute attributions. If the tensors are contained as values within the dictionary, then you can define a wrapper function which takes the tensors as input and constructs an appropriate dictionary and calls my_model with this dictionary. An example of this approach using a wrapper method with GuidedGradCam can be found in the discussion <a href="https://github.com/pytorch/captum/issues/444" rel="noopener nofollow ugc">here</a>. Hope this helps!</p>
Occlusion Sensitivity-Sliding window shape

Roshni_Pathak: <p>I am working with a tensor shape ([1,9]) and passing it as a tuple of bath size:2.<br>
I am unable to pass the right sliding window shape for Occlusion Model.<br>
Can you help me to know what would be the ideal input for sliding window shape and stride?<br>
A snippet of my Occlusion model is here:</p>
<p>clinical=torch.stack(tuple(clinical), dim=1) <span class="hashtag">#torch</span>.Size([2, 9])<br>
print(clinical.shape)<br>
clinical = clinical[:, :].detach().cpu().numpy()<br>
ablator = Occlusion(model)<br>
attr = ablator.attribute(tuple(clinical), target=ds_item[‘Class’], sliding_window_shapes=(2,9))<br>
occ_sens = OcclusionSensitivity(nn_module=my_model, n_batch=1, stride=(2,2))<br>
occ_map, most_probable_class = occ_sens(clinical,<br>
b_box=[-1, -1, 2, 3, -1, -1, -1, -1])<br>
print(occ_map)</p>
<p>Error message:<br>
in _format_and_verify_sliding_window_shapes<br>
assert len(sliding_window_shapes) == len(<br>
AssertionError: Must provide sliding window dimensions for each input tensor.</p>

vivekm: <p>Hi <a class="mention" href="/u/roshni_pathak">@Roshni_Pathak</a> , it looks like the input being passed to Occlusion here - <code>tuple(clinical)</code> - is a tuple of 2 numpy arrays. It is expected that the input should be a torch tensor.</p>
<p>For the sliding window shape, if your model’s input will still be 2 separate tensors, then the sliding window shape of each should be a separate tuple with one element e.g. ((9,), (9,)). This corresponds to one sliding window of length 9, which is equivalent to occluding each of the full input tensors together.<br>
Alternatively, if the input to the model (or wrapper forward function) would be a tensor of shape 2x9 such as the initial clinical tensor, then the sliding window shape can just be provided as (9,). The stride for Occlusion is not relevant if the sliding window covers the entire tensor, since it will not be shifted. If you would like to occlude smaller segments of the tensor, a smaller sliding window can be selected, with stride defining the shift of the window at each step. For more info regarding the Occlusion parameters, you can refer to the documentation <a href="https://captum.ai/api/occlusion.html" rel="noopener nofollow ugc">here</a>.</p>
Concatenation tensors of different dimensions - Alexnet and Renet

nik1: <p>Hello Everyone,</p>
<p>I have a question regarding concatenating two tensors. So basically, I need to extract first layer of ALexnet from my trained model, that output tensor of dimension ([8,64,34,34]) and first layer of Renet model that is already trained, which output tensor of dimension ([8,320,16,16])(Batch size, channel, H, W).<br>
If I concatenate these two tensors, getting the below error.<br>
RuntimeError: Sizes of tensors must match except in dimension 0. Got 64 and 320 in dimension 1 (The offending index is 1)<br>
I am aware that the dimension should be the same but not sure how to fix the issue.<br>
Could anyone help me with this issue?<br>
Thanks in advance.</p>

Eta_C: <p>Some options:</p>
<ul>
<li>down sample [8, 64, 34, 34] → [8, 320, 16, 16]</li>
<li>up sample [8, 320, 16, 16] → [8, 320, 32, 32]</li>
<li>repeat [8, 320, 16, 16] → [8, 320, 32, 32], then crop [8, 64, 34, 34] → [8, 64, 32, 32]</li>
</ul>
<p>and so on…<br>
It depends your task…</p>
GRU model overfitting

szahan: <p>I am trying to train my GRU network but it is overfitting. After around 20 epoch train accuracy and test accuracy differs a lot, while train reaches ~90%, test is only 67% at 91 epoch.<br>
I tried to reduce the overfit by adding a dropout laye but it doesn’t work. And I have 40091 train data and 16487 test data.</p>
<pre><code class="lang-auto">class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, bi_flag):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.gru = nn.GRU(input_size, hidden_size, num_layers=num_layers, bidirectional=bi_flag, batch_first=True)
        self.num_layers = num_layers

    def forward(self, input_tensor, seq_len):
        
        encoder_hidden = torch.Tensor().to(device)
        
        for it in range(max(seq_len)):
          if it == 0:           
            enout_tmp, hidden_tmp = self.gru(input_tensor[:, it:it+1, :])
          else:
            enout_tmp, hidden_tmp = self.gru(input_tensor[:, it:it+1, :], hidden_tmp)
          encoder_hidden = torch.cat((encoder_hidden, enout_tmp),1)

        hidden = torch.empty((1, len(seq_len), encoder_hidden.shape[-1])).to(device)
        count = 0       
        for ith_len in seq_len:       
            hidden[0, count, :] = encoder_hidden[count, ith_len - 1, :]
            count += 1
        
        return hidden
class seq2seq(nn.Module):
    def __init__(self, en_input_size, en_hidden_size, output_size, batch_size,
                 en_num_layers=3, de_num_layers=1, 
                 fix_state=False, fix_weight=False, teacher_force=False, bi_flag  = False, negative_r = None, reverse_flag = False):
        super(seq2seq, self).__init__()
        num_class = 60
        if bi_flag:
            decoder_num = 2
        else:
            decoder_num = 1

        self.batch_size = batch_size
        self.en_num_layers = en_num_layers
        self.encoder = EncoderRNN(en_input_size, en_hidden_size, en_num_layers, bi_flag).to(device)
        
        self.en_input_size = en_input_size
        self.teacher_force = teacher_force
        self.dropout = nn.Dropout(0.5)       
        self.linear = nn.Sequential(       
          nn.Linear(1024, 256), 
          nn.ReLU(inplace=True)
          )
        self.fcn = nn.Linear(256, num_class)

    def forward(self, input_tensor, seq_len, index = None, cluster_result=None):
        self.batch_size = len(seq_len)        
        encoder_hidden = self.encoder(input_tensor, seq_len)      
        encoder_hidden = self.dropout(encoder_hidden)
        middle = self.linear(encoder_hidden)       
        x = self.fcn(middle)
        x = x.view(x.size(1), -1)
        return x</code></pre>
Modify trainable affinity matrix (parameter) to calculate Laplacian Matrix

GiorgioMorales: <p>Hi there!</p>
<p>I have a problem in which part of the loss function depends on a trainable affinity matrix <code>W</code> and pairwise differences between elements of the output, which looks like this: <code>\sum_{ij} w_ij * (y_i - y_j)</code>, where <code>w_ij</code> represents the affinity between position <code>i</code> and <code>j</code>. The previous equation can be written in terms of the Laplacian matrix: <code>y*L*y^T</code> where <code>L =  D - W</code> and <code>D</code>, the degree matrix, is a diagonal matrix so that <code>D_ii = \sum_j w_ij</code> (each element of the diagonal is the sum of the elements of the corresponding row in the affinity matrix).</p>
<p>The thing is that I want to learn <code>W</code> but it is easier to use <code>L</code> instead, so I created a module to calculate my loss that would look like this:</p>
<pre><code class="lang-auto">class CRFlayer(nn.Module, ABC):
    def __init__(self, nodes=25):
        super(CRFlayer, self).__init__()
        self.nodes = nodes
        self.W = nn.Parameter(zeros(nodes, nodes))
        self.W.requires_grad = True

    def forward(self, Y, H):
        # Calculate Degree matrix using W
        D = "???"
        # Calculate Laplacian matrix using W
        L = D - self.W
        # Calculate loss based on L, the output Y, and other inputs
        loss = ...
        return loss
</code></pre>
<p>My question is that if it’s possible to do that. In particular, I don’t know how to code the degree matrix <code>D</code> and I don’t know if the gradients will be correctly calculated over <code>W</code> after that. Do you have any recommendations?</p>
<p>Thanks!</p>
Should I evaluate data per each or at once?

morphism: <p>Hello, everyone<br>
I have one question which I cannot solve</p>
<p>In PyTorch<br>
when evaluating (not training),<br>
codes are just like as below</p>
<p>output = np.full(len(testdata), -0.5)</p>
<p>model.eval()<br>
with torch.no_grad():<br>
for ii in range(len(testdata)):<br>
data = torch.Tensor( testdata.iloc[…<br>
output[ii] = model(data)</p>
<p>The above source code evaluates data per each sample<br>
which is very very slow</p>
<p>My question is:<br>
May I evaluate the testdata wholely at once?<br>
That is,</p>
<p>model.eval()<br>
with torch.no_grad():<br>
data = … whole test data at once<br>
output = copy.deepcopy(model(data)) //something like this</p>
<p>Can I do this? (which looks very fast)</p>
<p>Thank you in advance</p>

janhenr: <p>Hi, if your data fits into RAM to test it all at once then you can certainly do this.<br>
The reason why we use batches normally is exactly because we cannot load all the data into RAM.</p>
<p>Hope this helps!</p>

morphism: <p>Hello, janhenr.<br>
Thank you for your answer.<br>
So is there no difference between the two results (at once and per sample)??</p>
<p>Have a nice day</p>
Train using many input files without overloading memory

matthewleigh: <p>I am trying to setup up my dataloader to sample from multiple csv files within a directory, each of which contains a variable number of samples.</p>
<ul>
<li>Each sample is a row in one of the csv files</li>
<li>Each file is too big to load in as a single batch (~50k)</li>
<li>Each file contains a different number of samples (between 30k-60k)</li>
<li>There are several thousand csv files in the folder</li>
<li>The entire training set is to large too hold in memory (around 200M samples)</li>
</ul>
<p>I have looked at some of the examples on this forum and they include using torchvision.datafolder, however I think that assumes that each csv file contains a single sample, which does not apply to my situation.</p>
DataLoader randomly crashes after few epochs

sauhaardac: <p>I’m training with a DataLoader and it randomly crashes with this error after three epochs:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 46, in &lt;module&gt;
    for batch_idx, (song, label) in enumerate(train_loader):
  File "/home/sauhaarda/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 280, in __next__
    idx, batch = self._get_batch()
  File "/home/sauhaarda/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 259, in _get_batch
    return self.data_queue.get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 343, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
  File "/home/sauhaarda/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 178, in
 handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 9777) exited unexpectedly with exit code 1.
</code></pre>
<p>My code is available here:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/sauhaardac/raga_training/blob/nobptt/train.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/sauhaardac/raga_training/blob/nobptt/train.py" target="_blank" rel="nofollow noopener">sauhaardac/raga_training/blob/nobptt/train.py</a></h4>
<pre><code class="lang-py">from data import *
from models import RagaDetector
from utils import Averager
from torch.utils.data import DataLoader
import multiprocessing
import progressbar
from tensorboardX import SummaryWriter

writer = SummaryWriter()
bptt = 1000

def repackage_hidden(h):
    """Wraps hidden states in new Tensors, to detach them from their history."""
    if isinstance(h, torch.Tensor):
        return h.detach()
    else:
        return tuple(repackage_hidden(v) for v in h)


def time_split(source, i):
</code></pre>

  This file has been truncated. <a href="https://github.com/sauhaardac/raga_training/blob/nobptt/train.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

ptrblck: <p>Could you set <code>num_workers=0</code>, run it again and see, if the worker still crashes.<br>
If not, set it to <code>1</code> and try it again. It would be interesting to see, if your data is somehow corrupt or not.</p>

Muna: <p>Hi, I have the same problem. It works with num_worker=0 but it is so slow.<br>
I am running my code in cluster with 4 GPU, 48 CPU, and 360 GB memory.</p>
Why if("bn" not in name): param.requires_grad_(False)

Mona_Jalal: <p>Do you know what is the reason for the lines</p>
<pre><code class="lang-auto"> if("bn" not in name):
                param.requires_grad_(False)
</code></pre>
<p>in the following code snippet?</p>
<pre><code class="lang-auto">        mnet = models.mobilenet_v2(pretrained=True)
        for name, param in mnet.named_parameters():
            if("bn" not in name):
                param.requires_grad_(False)
</code></pre>

Unity05: <p>Hi,</p>
<p>BatchNormalization has two trainable (weight and bias / γ and β) and two mini - batch dependent (mean and variance) parameters. Therefore your code snippet freezes all layers except for the BatchNormalization - Layers what can be useful for finetuning (setting the BatchNormalization - Layers requires_grad flag to false would therefore freeze the two trainable parameters).</p>
<p>Regards,<br>
Unity05</p>
Undefined symbol in Xcode when using libtorch

111410: <p>I’ve compiled <em>libtorch</em> for iOS simulator as shown <a href="https://pytorch.org/mobile/ios/#build-pytorch-ios-libraries-from-source" rel="noopener nofollow ugc">here</a> and linked it to my Xcode project.</p>
<p>Includes work fine, but actual <em>libtorch</em> code cases this error when building:</p>
<p><strong>Undefined symbols for architecture arm64:<br>
“c10::UndefinedTensorImpl::_singleton”, referenced from:<br>
c10::UndefinedTensorImpl::singleton() in test.o<br>
ld: symbol(s) not found for architecture arm64<br>
clang: error: linker command failed with exit code 1 (use -v to see invocation)</strong></p>
<p>Code as <em>test.cpp</em>:</p>
<pre><code class="lang-auto">#include "test.hpp"
#include "ATen/ATen.h"
#include "caffe2/core/timer.h"
#include "caffe2/utils/string_utils.h"
#include "torch/csrc/autograd/grad_mode.h"
#include "torch/csrc/jit/serialization/import.h"
#include "torch/script.h"

at::Tensor v;
</code></pre>
<p>Please, tell what I am doing wrong and how to fix it.</p>
The purpose of aten::size → prim::ListConstruct → aten::view triplet

hidefromkgb: <p>While exploring a <code>torch.jit.trace()</code> of the standard ResNet50 from <code>torchvision.models</code> I stumbled upon a peculiar structure at the very end of the network:</p>
<pre><code class="lang-auto">       &lt;...&gt;
         |
 ________v_________        ________             _________
|                  |      |        |           |         |
| aten::avg_pool2d |      | Int(0) |           | Int(-1) |
|__________________|      |________|           |_________|
         |                    |                     |
         |               _____v______     __________v__________
         |              |            |   |                     |
         |--------------&gt; aten::size |---&gt; prim::ListConstruct |
         |              |____________|   |_____________________|
         |                                          |
         |               ____________               |
         |              |            |              |
         `--------------&gt; aten::view &lt;--------------'
                        |____________|
                              |
                              v             _________
                       #==============# &lt;--|_weights_|
                       # aten::linear #     _________
                       #==============# &lt;--|_biases__|

</code></pre>
<p>What exactly is it needed for?<br>
Why cannot <code>aten::avg_pool2d</code> be plugged directly into <code>aten::linear</code>?</p>
<p>At the moment I`m not too well-versed in PyTorch — but in those ML frameworks I`m familiar with nothing prevents such direct connections between 2D Average Pooling and Inner Product.</p>

tom: <p>I think with more recent PyTorch/TorchVision you’d get <code>aten::flatten</code> instead – this a traced version of <code>torch.view(x, [x.size(0), -1])</code> to move from <code>n,c,h,w</code> to <code>n,features</code>.<br>
Keras does this flattening implicitly when you use “GlobalAveragePooling” because it actually removes the spatial dimensions which PyTorch’s pooling does not.</p>

hidefromkgb: <blockquote>
<p>torch.view(x, [x.size(0), -1])</p>
</blockquote>
<p>Do I get it right that this just tells <code>torch.view</code> to keep the batch count and merge the remaining dimensions of <code>x</code> together?<br>
Damn, this explains so much. Thank you!</p>

tom: <aside class="quote no-group" data-username="hidefromkgb" data-post="3" data-topic="113985">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hidefromkgb/48/30660_2.png" class="avatar"> hidefromkgb:</div>
<blockquote>
<p>Do I get it right that this just tells <code>torch.view</code> to keep the batch count and merge the remaining dimensions of <code>x</code> together?</p>
</blockquote>
</aside>
<p>Yes, this is exactly what it does.</p>
ONNX to TorchScript conversion for quantized networks

hidefromkgb: <p>There is a quantized network in ONNX format that I`d like to convert to TorchScript.<br>
I know that such conversions are usually done backwards, from TS to ONNX, but I want to test my TS integration on some quantized models and I was told the easiest way is to use ONNX.</p>
<ol>
<li>Does ONNX use the same execution pipeline as TorchScript which I can interface via <code>RegisterPass</code> and <code>RegisterOperators</code>?</li>
<li>If not, then how do I perform a ONNX → TorchScript conversion?</li>
<li>If this is impossible, how do I acquire and trace a quantized, preferably asymmetric, ResNet50 model?</li>
</ol>
Applying a Conv2d filter to an image tensor

erik_li: <p>Hey guys,</p>
<p>I’m new to Python, and trying to do some manipulations with filters in Pytorch.</p>
<p>I’m struggling re how to apply a Conv2d. I’ve got the following code which creates a 3x3 moving avg filter</p>
<pre><code class="lang-auto">resized_image4D = np.reshape(image_noisy, (1, 1, image_noisy.shape[0], image_noisy.shape[1]))
t = torch.from_numpy(resized_image4D)

conv = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, padding=1, bias=False)
conv.weight = torch.nn.Parameter(torch.ones((1,1,3, 3))/9.0)
</code></pre>
<p>Normally in np I’d just call <code>convolve2x(image,kernel)</code>, but I’ve not been able to figure out what the PyTorch equivalent is after days of searching.</p>
<p>Apologies if a dumb question. Any pointers in the right direction would be greatly appreciated.</p>
<p>Thank you very much<br>
Erik</p>

Dwight_Foster: <p>You could use torch.nn.functional.conv2d then you can pass the input image right into the conv2d layer. <a href="https://pytorch.org/docs/stable/nn.functional.html#conv2d" rel="noopener nofollow ugc">Here</a> is the documentation.</p>

erik_li: <p>Got it, managed to get it to work. Thank you very much!!!</p>
Sharing list in DistributedDataParallel

Seo: <p>Hi everyone,<br>
I am in trouble since I don’t know how to manage evaluation phase with DistributedDataParallel model. In my evaluation loop I accumulate the correct predictions in order to compute the final accuracy per epoch. These predictions are stored inside a list of dictionaries. My model is wrapped in a Distributed DataParallel and so, each process will compute predictions on a separate portion of the dataset.<br>
Unfortunately, predictions are not tensors and so I cannot use the utilities provided in <code>torch.distributed</code>. I tried to save all the lists on disk and concatenate the results in the main process (<code>rank == 0</code>) but this method will not work in distributed scenario where I have multiple nodes.</p>
<p>Do you know how to gather the list from all the different processes in order to compute the final accuracy per epoch?</p>

pritamdamania87: <p><a class="mention" href="/u/seo">@Seo</a> You can probably use <a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.gather_object" rel="noopener nofollow ugc">gather_object</a> to gather objects on a single rank which are not tensors.</p>

Seo: <p>Hi <a class="mention" href="/u/pritamdamania87">@pritamdamania87</a>, thank you for your answer. I didn’t notice the <code>gather_object</code> because it is a new feature included in the current 1.8.0 torch nightly. Unfortunately, the <code>gather_object</code> doesn’t work with NCCL backend, so I have used the <code>all_gather_object</code> to broadcast the results to all the processes (and then use it only in the main one). I hope <code>gather_object</code> will be available for NCCL backend in the next release.<br>
Thank you!</p>

pritamdamania87: <p>Looks like gather is not supported in NCCL as a native API. Although I think we can support gather in PyTorch for the NCCL backend by using ncclSend/ncclRecv under the hood. cc <a class="mention" href="/u/rvarm1">@rvarm1</a> <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a></p>

ibro45: <p>That would be great, for the moment (1.8.0) <code>gather_object</code> still doesn’t work with NCCL backend.</p>
Conv2d layer dimension order using shape

Kamil_Adamczewski: <p>It seems that the input and output channels are reversed in the definition of the function and when using tensor shape. That is, we have</p>
<p><code>torch.nn.</code> <code>Conv2d</code> (<em>in_channels</em>, <em>out_channels</em>, <em>kernel_size</em>, <em>stride=1</em>, <em>padding=0</em>, <em>dilation=1</em>, <em>groups=1</em>, <em>bias=True</em>, <em>padding_mode=‘zeros’</em> )</p>
<p>if when in_channels=16, and out_channels = 32 (and kernel_size =3) and we call</p>
<p>conv.weight.shape</p>
<p>we get (32, 16, 3, 3).</p>
<p>Is that correct?</p>

ksz: <p>A simple experiment shows that the output is (32,16,3,3) - and this is correct:</p>
<pre><code class="lang-auto">import torch
conv = torch.nn.Conv2d(16, 32, 3)
print(conv.weight.shape)

&gt;&gt;&gt; torch.Size([32, 16, 3, 3])
</code></pre>
<p>Please, think about it. We have a convolution window (kernel) with shape 3x3 - part 3,3. Now, we need as many windows as there are input channels - so in this case we need 16 kernels - part 16,3,3. And all these numbers 16,3,3 are needed to create only one new channel output. 1,16,3,3. But you want to have 32 outputs, so you need 32,16,3,3 number of weights to be able to calculate this.</p>
<p>See example: <a href="https://www.researchgate.net/profile/Santhiya-Rajan/post/How-will-channels-RGB-effect-convolutional-neural-network/attachment/5c67b72d3843b0544e664e12/AS%3A726829115666434%401550300973344/download/cnn_1.gif" rel="noopener nofollow ugc">gif</a><br>
To create one new output channel based on three input channels(rgb) you need to store (1, 3, 3, 3) weights. To get two outputs channels you need to double number of weights: (2, 3, 3, 3)</p>
Stacking images forming one image

Zidane: <p>I have gabor filtered images and I am trying to stack them to get the original image back. I tried the concatente functions in tensorflow and numpy but they put the images next to each other vertically or horizontally and that is not what I need. I need to have something like the picture attached</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/781e04bb18630ac43d412b91c609a10036e89705.png" alt="image" data-base62-sha1="h8BBRIUiIhnCOUSgVLshPbijHsF" width="220" height="147"></p>
<p>In the picture, on the top left, this is the original image, on the right are the gabor filtered images and on the bottom left is the image after stacking the gabor filtered images. Any suggestions?</p>
Steps are not sequential strange behaviour

Pranavk: <p>I am running PPO , with CUDA on a GPU I saw that the steps are not in the sequential order<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/7835dfd05a5ad8140b43a83191b1187b4ccc386b.png" data-download-href="https://discuss.pytorch.org/uploads/default/7835dfd05a5ad8140b43a83191b1187b4ccc386b" title="RF"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/7835dfd05a5ad8140b43a83191b1187b4ccc386b_2_690x342.png" alt="RF" data-base62-sha1="h9qIJDTzJlgo7Ig50p4HZUDc7iz" width="690" height="342" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/7835dfd05a5ad8140b43a83191b1187b4ccc386b_2_690x342.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/7835dfd05a5ad8140b43a83191b1187b4ccc386b_2_1035x513.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/7835dfd05a5ad8140b43a83191b1187b4ccc386b_2_1380x684.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">RF</span><span class="informations">1392×690 35.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this because of parallel processing or something is wrong.<br>
I have seen that many steps are repeating in different order ? can anyone help me .Thank you</p>

ptrblck: <p>Could you post (pseudo) code which shows how these operations are applied?</p>

Pranavk: <p>Hi sorry I forgot to close this issue it was an error with notebook I was using it was printing everything twice.Thank you for your response.</p>
Data Augmentation transormed vs original

serdarrader: <p>When we are creating transformations and augment images, those are generally probabilistic. So model is generally not seeing original images, it always sees transformed images, isn’t it a problem. Wouldn’t an approach that both feed transformed and original image be better ?</p>

Nikronic: <p>Hi,</p>
<p>Actually, that is not the case. Some transforms like cropping or resizing image are made to all images for instance for computational reasons.<br>
But other transforms like, affine, colorjitter, etc can be done randomly which in this case, you provide <code>p=your_desired_prob</code> to ensure transformation is only happening with probability of <code>p</code>. So, there <code>p-1</code> of augmented images of that transform are original images.</p>
<p>For instance, you can see the docs of <a>RandomApply</a>.</p>
<p>Bests</p>

serdarrader: <p>Yes but we generally stack bunch of transformations, so usually at least one of them is ending up happening, so usually model is not seeing original images, but transformed ones?</p>
Custom length of x and y at each iteration of DataLoader

teotfw: <p>I’m dealing with evaluation step for image-text matching problem. I want to compute the similarity between all images and a number of captions (batch_size). For example, with batch_size=32, at each iteration, DataLoader will return all images and 32 captions.</p>
<p>Is there anyway to do that?!</p>

janhenr: <p>Hi there,<br>
I think for your usecase you need all images at every training_step, right?<br>
In that case, I would make two different dataloaders, <code>img_loader</code> and <code>text_loader</code>.<br>
Get a batch that consists of all images before training starts and save it in a<code>img</code> variable.<br>
Then use your <code>text_loader</code> like a normal dataloader, getting a text batch at every step.</p>
<p>Hope this helps!</p>
Training GAN on MNIST gives very strange results

Darek554: <p>Hi.</p>
<p>Results of training are quite strange, because Discriminator loss and Generator loss are very small:</p>
<p>Loss in epoch  10<br>
Discriminator:  0.00015746265571486972<br>
Generator:  1.1572355696669042e-08</p>
<p>Loss in epoch  20<br>
Discriminator:  0.000207384460320012<br>
Generator:  3.4869694783008414e-10</p>
<p>Loss in epoch  30<br>
Discriminator:  4.014897100638508e-05<br>
Generator:  0.0</p>
<p>Loss in epoch  40<br>
Discriminator:  0.00012505375613592535<br>
Generator:  0.0</p>
<p>Loss in epoch  50<br>
Discriminator:  0.00015698512803940945<br>
Generator:  0.0</p>
<p>but data generated by Generator is some random nonsense.<br>
Generated data sample, almost everything looks like this:<br>
                        <a href="https://imgur.com/T3Ow5zS" target="_blank" rel="noopener nofollow ugc" class="onebox">
              <img src="https://i.imgur.com/T3Ow5zS.png" title="Imgur: The magic of the Internet" alt="Imgur" width="251" height="248">
            </a>

</p>
<p>I’ve checked loss functions, tried a lot of different values of hyperparameters, but nothing works.</p>
<p>If someone could help i’ll appreciate <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Training loop code:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://pastebin.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://pastebin.com/NacDh4Nd" target="_blank" rel="noopener nofollow ugc">Pastebin</a>
  </header>
  <article class="onebox-body">
    <img src="https://pastebin.com/i/facebook.png" class="thumbnail" width="" height="">

<h3><a href="https://pastebin.com/NacDh4Nd" target="_blank" rel="noopener nofollow ugc">mnist_gan = GAN(data_dimension=784, latent_dimension=64,...</a></h3>

<p>Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Model implementation:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://pastebin.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://pastebin.com/qeAYWaFp" target="_blank" rel="noopener nofollow ugc">Pastebin</a>
  </header>
  <article class="onebox-body">
    <img src="https://pastebin.com/i/facebook.png" class="thumbnail" width="" height="">

<h3><a href="https://pastebin.com/qeAYWaFp" target="_blank" rel="noopener nofollow ugc">import torchimport torch.nn as nnimport torch.nn.functional as Fimport num -...</a></h3>

<p>Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Derivative = 0 with higher learning rates

knoahcud: <p>Hello guys,</p>
<p>I’m quite new to pytorch. Thus I’ve created a dummy dataset with the function y = x_1 + 2<em>x_2 + 3</em>x_3 + 4*x_4 to get used to the autograd function. The whole code isn’t working for higher learning rates e.g. 0.1. Does anybody know why?<br>
Thanks for your help!</p>
<blockquote>
<p>import torch.nn as nn<br>
import torch.nn.functional as f</p>
<p>class deriv_net(nn.Module):</p>
<p> def <strong>init</strong>(self):<br>
          super(deriv_net, self).<strong>init</strong>()<br>
         self.fc1 = nn.Linear(4, 20)<br>
          self.fc2 = nn.Linear(20, 40)<br>
         self.last = nn.Linear(40, 1)</p>
<p>    def forward(self, x):<br>
         out = f.relu(self.fc1(x))<br>
         out = f.relu(self.fc2(out))<br>
         out = self.last(out)<br>
         return out</p>
<p>from autograd_dataset import autograd_dataset<br>
from deriv_nn import deriv_net<br>
import torch<br>
import torch.nn.functional as f<br>
import torch.optim as optim<br>
from torch.utils.data import DataLoader</p>
<p>def train(deriv_model, epoch):<br>
     optimizer = optim.SGD(deriv_model.parameters(), lr=0.1, momentum=0.8)<br>
    for batch_id, (data, target) in enumerate(train_data):<br>
         data.requires_grad = True<br>
         out = deriv_model(data.float())<br>
        deriv = torch.autograd.grad(outputs=[out], inputs=[data],<br>
              grad_outputs=torch.ones_like(out),<br>
             allow_unused=True, retain_graph=True, create_graph=True)[0]<br>
      print(deriv)<br>
        optimizer.zero_grad()<br>
      loss = f.mse_loss(out, target)<br>
        loss.backward()<br>
      optimizer.step()<br>
       print(‘Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.06f}’.format(<br>
                epoch, batch_id * len(data), len(train_data.dataset), 100. *<br>
    batch_id / len(train_data), loss.data.item()))</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
    <span class="hashtag-raw">#define</span> train_data<br>
     dataset = autograd_dataset(‘test_autograd’)<br>
    train_data = DataLoader(dataset=dataset, batch_size=1, shuffle=True)</p>
<p>     deriv_model = deriv_net()<br>
    for Epoch in range(1, 100):<br>
      train(deriv_model, Epoch)</p>
</blockquote>
Nn.utils.rnn.pad_sequence with values from sequence?

ZimoNitrome: <p>I have a list of irregular tensors like:</p>
<pre><code class="lang-python">tensor_list = [
    torch.tensor([    # shape [4, 3]
        [8, 8, 3],
        [7, 2, 9],
        [2, 4, 3],
        [3, 3, 3]
    ]),
    torch.tensor([    # shape [2, 3]
        [1, 3, 4],
        [5, 5, 5]
    ]), 
    torch.tensor([    # shape [3, 3]
        [3, 3, 3],
        [7, 7, 9],
        [2, 1, 3]
    ])
]
</code></pre>
<p>To pad these tensors to a regular shape, the following code works:</p>
<pre><code class="lang-python">regular_tensor = nn.utils.rnn.pad_sequence(tensor_list, batch_first=True, padding_value=0)
</code></pre>
<p>And the regular tensor will be of shape [3, 4, 3]:</p>
<pre><code class="lang-python">torch.tensor([
    [
        [8, 8, 3],
        [7, 2, 9],
        [2, 4, 3],
        [3, 3, 3]
    ],
    [
        [1, 3, 4],
        [5, 5, 5],
        [0, 0, 0],    # &lt;-- padded value
        [0, 0, 0]     # &lt;-- padded value
    ],
    [
        [3, 3, 3],
        [7, 7, 9],
        [2, 1, 3],
        [0, 0, 0]    # &lt;-- padded value
    ])
])
</code></pre>
<p>But I would like the padded values to take triplet values from the already existing tensors 
(first/last/random doesn’t matter)
 <strong>preferably through random choice</strong>. So that the regular tensor will be something like:</p>
<pre><code class="lang-python">torch.tensor([
    [
        [8, 8, 3],
        [7, 2, 9],
        [2, 4, 3],
        [3, 3, 3]
    ],
    [
        [1, 3, 4],
        [5, 5, 5],
        [5, 5, 5],    # &lt;-- randomly selected value from 1 row above
        [1, 3, 4]     # &lt;-- randomly selected value from 3 rows above
    ],
    [
        [3, 3, 3],
        [7, 7, 9],
        [2, 1, 3],
        [7, 7, 9]     # &lt;-- randomly selected value from 2 rows above
    ])
])
</code></pre>
<p>How is this best achieved?</p>

ZimoNitrome: <p>I made a function for this. It’s pretty specific and probably not optimal but works:</p>
<pre><code class="lang-python">def pad_colors_random_choice(sequences):
    longest_colors = max([len(s) for s in sequences])
    
    out_tensor = torch.empty([len(sequences), longest_colors, 3], device=sequences[0].device)
    for i, colors in enumerate(sequences):
        n_colors = len(colors)
        diff = longest_colors - n_colors
        pad = colors[torch.randint(n_colors, [diff])]  
        out_tensor[i] = torch.cat([colors, pad])
        
    return out_tensor
</code></pre>
Setattr() Moving Parameter Tensors to CPU

J_Johnson: <p>If I try to use setattr() between training runs to make some changes to a layer, the device seems to change, too, moving the parameters to the CPU. Is there a correct way to use setattr to keep the parameters on the same device?</p>
<p>For example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.fc1 = nn.Linear(3,3)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.fc1(self.relu(x))
        return x

model=Model()
model.to('cuda')

for params in model.parameters():
    print(params.get_device())

for layer, p in zip(model.children(), model.named_children()):
    if type(layer) == nn.Linear:
        setattr(model, p[0], type(layer)(2, 2))

for params in model.parameters():
    print(params.get_device())
</code></pre>
<p>This yields:<br>
0<br>
0<br>
-1<br>
-1<br>
(Where 0 is cuda and -1 is cpu.)</p>

J_Johnson: <p>For now, I’m using <code>model.cuda(device)</code>. But it would seem preferable to have a solution which doesn’t have to move tensors.</p>

albanD: <p>Hi,</p>
<p>The problem is that the brand new layer you create is created like all new layers on the cpu by default. So you need to move it to the right device before setting it.</p>

J_Johnson: <p>The following <code>cuda</code> call works perfectly in creating each layer on the device:</p>
<pre><code class="lang-auto">for layer, p in zip(model.children(), model.named_children()):
    if type(layer) == nn.Linear:
        setattr(model, p[0], type(layer)(2, 2).cuda(device))
</code></pre>
How do I know the current version of pytorch?

earnp: <p>How do I know the current version of pytorch?</p>

smth: <pre><code class="lang-python">import torch
print(torch.__version__)
</code></pre>

isalirezag: <p>Just for the record:<br>
to check it from terminal in linux:</p>
<p><code>python -c "import torch; print(torch.__version__)"</code></p>

gireesh4manu: <p>So quick question here. When I check from Jupyter, I’m able to see the version printed but when I do the same from terminal, I get import error: no module named torch. Can you please help me out here?</p>
<p>Thanks.</p>

KevNull: <p>I’m guessing jupyter is running in a different python environment than your default one.</p>
<p>The first thing to try would be to see what happens if you replace ‘python’ with ‘python3’ at the start of that command.</p>

gireesh4manu: <p>Thank you. That was it.</p>

cgironda: <p>Thank you. It was very helpful!</p>

Bavalpreet_Singh: <p>with torch.<strong>version</strong> it gives<br>
module ‘torch’ has no attribute ‘<strong>version</strong>’</p>

ptrblck: <p>Did you use the two underscores in the command as it’s working fine:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0a0+df837d0'
</code></pre>