akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
[tutorial] Adding New Compiler Backends to PyTorch JIT

username: bwasti
cooked: <p>Hey all,</p>
<p>I wrote up a tutorial on how to integrate a new backend into PyTorch JIT. The process is relatively simple and only requires around 2-3 PyTorch specific API calls. It can be found <a href="https://jott.live/markdown/Writing%20a%20Toy%20Backend%20Compiler%20for%20PyTorch" rel="nofollow noopener">here</a>.</p>

username: ptrblck
cooked: <p>That looks like an awesome tutorial, Bram!<br>
Thanks for sharing. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: leaf_yyl
cooked: <p>It is very useful for me! Thanks for sharing!<br>
I want  to implement a third-party hardware backend compiler, is there any more examples or documents?<br>
Thanks in advance!</p>

username: SplitInfinity
cooked: <p>Take a look at the <code>torch/csrc/jit/backends/backend.h</code> header. You can see an example of <a href="https://github.com/pytorch/glow/blob/master/torch_glow/src/TorchGlowBackend.h" rel="noopener nofollow ugc">a backend registered using this API in the pytorch/glow repository</a>.</p>
How to effectively release a Tensor in Pytorch?

username: DanielWang
cooked: <p><strong>Hi, All</strong></p>
<p>How could we release a Pytorch Tensor when we decide not to use such a tensor anymore? In both python API and C++ API of Pytorch.</p>
<p><strong>Thanks!</strong></p>

username: ptrblck
cooked: <p>In Python <code>del tensor</code> will work, in <code>libtorch</code> you could use e.g. <a href="https://discuss.pytorch.org/t/how-to-manually-delete-free-a-tensor-in-aten/64153/4">this approach</a>.</p>

username: copythatpasta
cooked: <p>Please note in libtorch for tensors on the GPU you may have to call c10::cuda::CUDACachingAllocator::empty_cache() once the tensor goes out of scope if you want pytorch to release this memory. Otherwise, it still frees the memory once tensor goes out of scope but still keeps that memory available in libtorch’s cache. If you want to free that memory for other processes then you must call empty_cache()</p>

username: DanielWang
cooked: <p>Hi,</p>
<p>I have one more question is that. Assuming I create a customized Pytorch API that will create a tensor inside the C++ function during the execution. For example.</p>
<pre><code class="lang-auto">A = create_a_CUDA_tensor_via_customized_CPP_function();
</code></pre>
<p>inside the <code> create_a_CUDA_tensor_via_customized_CPP_function();</code> I create and return a tensor like <code>torch::ones().cuda()</code>.</p>
<p>if later on at some point, I want to assign <code>A</code> with another tensor.</p>
<pre><code class="lang-auto">A = Another_create_a_CUDA_tensor_via_customized_CPP_function();
</code></pre>
<p>Do I need to <code>del A</code> before I am calling the another function to avoid the GPU memory leak?</p>
<p>Thanks!</p>

username: copythatpasta
cooked: <p>Hey Daniel, just seeing this.</p>
<p>If you are allocating memory on the GPU in a custom function, you will want to call empty_cache() right after the function to clean up any memory from the orignal tensor constructed within the function. As long as you clean up after each function call, you do not need to worry about deleting any memory. By calling empty_cache() after the reassignment of A, you clean up any unused <em>memory in cache</em>.  Note that if you do not do this, it is not a memory leak. Emptying the cache only frees the memory pytorch is using. In reality pytorch is freeing the memory without you having to call empty_cache(), it just hold on to it in cache to be able to perform subsequent operations on the GPU easily. You only want to call empty_cache if you want to free the GPU memory for other processes to use (other models, programs, etc)</p>
Build Pytorch from source using conda-build

username: Haydnspass
cooked: <p>Hey,</p>
<p>We observe quite significant speedups when we compile pytorch from source. To make life for the people in the group easier I would like to build pytorch on our cluster using conda-build and then distribute it this way internally.</p>
<p>I tried to reproduce the build using the pytorch/builder repo via docker containers but failed. I have come across this discussion <a href="https://discuss.pytorch.org/t/building-from-source-with-conda-build/70297/3" class="inline-onebox">Building from source with conda build - #3 by seliad</a> basically stating that the builder is only for internal purposes. Is that still correct?<a class="mention" href="/u/alband">@albanD</a>, <a class="mention" href="/u/seliad">@seliad</a> how did it turn out for you in the end?</p>
<p>A local build using <code>python setup.py install</code> works nicely so I guess I am not too far away.<br>
What would be a good way to achieve what I am aiming for?</p>
<p>Does it make sense to use one of the docker images on the pytorch docker hub? How should the meta.yaml then look like?</p>
<p>Thanks in advance!</p>
<p>Cheers,<br>
Lucas</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think what the “internal only” message means is that we change it quite regularly to adapt to our CI/release process without special notice. So you should rely on it.</p>
<p>If all these installs will be on the same machines (with same local shared libs, cuda versions, GPUs, etc), then you should be able to <code>python setup.py bdist_wheel</code> to create such a wheel that you can then share (and pip install in each env).</p>
Add residual block between convolutional layer and FC layer

username: kl_divergence
cooked: <p>So my model looks like this:</p>
<pre><code class="lang-auto">class ResNet50(nn.Module):
    def __init__(self, **kwargs):
        super(ResNet50,self).__init__()
        resnet50 = torchvision.models.resnet50(pretrained=True)
        self.base= nn.Sequential(*list(resnet50.children())[:-2])
        setattr(self,"fc0", nn.Linear(2048,751)) 
        setattr(self,"fc1", nn.Linear(2048,1360))         

    def forward(self,x):
        x = self.base(x)
        x = F.avg_pool2d(x,x.size()[2:])
        f = x.view(x.size(0),-1)
        clf_outputs = {}
        num_fcs = 2
        
        for i in range(num_fcs):
            clf_outputs["fc%d" %i] = getattr(self, "fc%d" %i)(f)

        return clf_outputs,f 
</code></pre>
<p>I want to add residual block after last convolution layer and FC1,  how can i do that ?</p>

username: Mohammed_Awney
cooked: <p>I have the same question, Do you solve it?</p>
Data Preprocessing Issue

username: Chin_GM
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6055825432c61248ef729caabb590f4cd3e55eb.jpeg" alt="sample_data" data-base62-sha1="pYelGwkQzFJus7XjeB0F8Xv9ckr" width="398" height="406"><br>
Dear all,</p>
<p>Good day. I am new to machine learning.<br>
May I get some advice on data preprocessing?<br>
The attached image is sample data, which the first column is time step, second column is item_ID, and the last column is speed.</p>
<p>My question is, how can I create a sequences of time series based on each item_ID?<br>
I don’t want the the sequences is mix of both item.<br>
I am aware of creating cyclic sequences. But the length of time step for each item is different.</p>
<p>May I get some advice from you all?</p>
<p>Thank you very much.</p>
<p>Regards,<br>
GMC</p>

username: JuanFMontesinos
cooked: <p>Hmmm it’s a long story.<br>
When data is different length you can pad the data.<br>
Nowadays, there are architectures which deals with different-length samples (transformers).</p>
<p>The way to go is measuring or estimating the maximum length possible and then padding your samples according to that.</p>
<p>Another way is doing chunks of a representative length and working with a reduced duration.</p>
Combining ResNet50 and a MLP in a siamese structure

username: jorisvane
cooked: <p>Hi guys,<br>
I started combining pre-trained ResNet50 and an MLP in a Siamese Structure and am running into a problem with dimensions. The idea is that two images with each a price go through the network and it returns a probability of chosing the first image. My images are now of size [224,224,3] as they should be for ResNet and I removed the last layer which should give me a vector of 2048 features. When I try to run the following code to test is the model has an output</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    for i, (image1, image2, y_label, price1, price2) in enumerate(train_loader):
        
        # Get data to cuda

        image1 = image1.to(device=device)
        image2 = image2.to(device=device)
        y_label = y_label.to(device=device)
        price1 = price1.to(device=device)
        price2 = price2.to(device=device)
        
        #print(image1.shape)
        
        # Forward

        prob = model(image1, price1, image2, price2)
        print(prob)
</code></pre>
<p>I get the following error: RuntimeError: mat1 dim 1 must match mat2 dim 0<br>
this is my model</p>
<pre><code class="lang-auto">model = models.resnet50(pretrained=True)

newmodel = torch.nn.Sequential(*(list(model.children())[:-1]))

pretrained = newmodel

class NN(nn.Module):

    def __init__(self, my_pretrained_model):
        super(NN, self).__init__()
        self.pretrained = my_pretrained_model
        
        self.MLP = nn.Sequential(
            nn.Linear(2048, 124),
            nn.ReLU(),                 
            nn.Linear(124, 1)
        )
        self.last_node = nn.Linear(2,1)
    
    
    def forward_once(self, x, y):
        x = self.pretrained(x)
        x = self.MLP(x)
        torch.cat((x,y))
        x = self.last_node(x,y)
        return x
    
    def forward(self, image1, price1, image2, price2):
        output1 = self.forward_once(image1, price1)
        output2 = self.forward_once(image2, price2)
        prob = 1/(1 + math.exp(output2-output1))    # probability of choosing first image
        return prob
</code></pre>
<p>Any help is appreciated!<br>
Best regards</p>

username: jorisvane
cooked: <p>The problem was that the second last layer from ResNet50 gave a 4d vector in the shape of [batchsize, 2048,1,1] so after squeezing with torch.squeeze(x) it had the right dimensions to go into the MLP.</p>
Speed up loading traced network

username: Shakiba_Kh
cooked: <p>Hello All,</p>
<p>Is there a way to speed up loading the traced network in C++ (Windows 10)?</p>
<p>In python, I first need to run a network and based on its result decide whether to do the second one or not. So my tracing code looks like this:</p>
<pre><code>Class MySampleNetwork(nn.Module):
    def __init__(self, params):
        super(MySampleNetwork, self).__init__()
        # trace network1
        sample_data1 = torch.rand(size)
        sample_data1 = sample_data1.to(ddata.gpu, non_blocking=True)
        net1 = load_net(net1).eval().to(ddata.gpu)
        self.net1 = torch.jit.trace(net1, sample_data1)
        # trace network2
        sample_data2 = torch.rand(size)
        sample_data2 = sample_data2.to(ddata.gpu, non_blocking=True)
        net2 = load_net(net2).eval().to(ddata.gpu)
        self.net1 = torch.jit.trace(net2, sample_data2)

    def forward(self, net1_input, net2_input):
        net1_out = self.net1(net1_input) 
        net2_out = torch.zeros(expected_out_size)

        if condistion based on net1_out:
            net2_out = self.net2(net2_input)
        
        # combine the results
        result = torch.zeros(net1_out_size+net2_out_size)
        result[:net1_out_size] = net1_out[0]
        result[net1_out_size:] = net2_out[0]

        return result

traced_net = torch.jit.script(MySampleNetwork(params))
</code></pre>
<p>The saved network is about 26M.</p>
<p>In C++, I load the network</p>
<pre><code>torch::jit::script::Module module = torch::jit::load(net_path())
</code></pre>
<p>It takes more than 30 seconds to load the network. Is there anything I can do to reduce the time?</p>
<p>Thank you.</p>

username: dambo
cooked: <p>Try this:</p>
<pre><code class="lang-auto">torch::Device device(torch::kCUDA);
const std::string modelName = "candy_cpp.pt";
auto module = torch::jit::load(modelName, device);
</code></pre>
<p>How much time does this take to load?</p>

username: Shakiba_Kh
cooked: <p><a class="mention" href="/u/dambo">@dambo</a> Thank you for your response. Adding device does not make it faster.<br>
The problem is in if condition. Once I remove it, it loads in about 2-4 seconds. For now, I removed the if, so I’m computing the whole thing even though I sometimes do not need it. But would need to solve this issue if anyone can help.</p>

username: copythatpasta
cooked: <p>Have you added print statements to see where exactly its happening? I would time how long it takes to load the net(assuming you are loading both at the same time). My suspicion is that since you are trying to trace both models in the constructor, this slows things down.</p>
<p>There really isnt a need to trace/script the network a total of 3 times. I would remove the tracing in the constructor, and just script the whole graph at once. You only want to script the network when it will be used outside of a Python environment and scripting the network at the end like you have is all you need to achieve this.</p>
How Are Sizes Tracked When Tracing or Scripting?

username: dtoniolo
cooked: <p>When compiling to TorchScript either with tracing or scripting, I often have problems with operations that depend explicitly on tensor sizes. Sometimes sizes get hardcoded as constants, breaking compatibility with variable batch sizes. Other times PyTorch complains that I’m converting to torch integers to Python <code>int</code>s, which will be traced as constants.</p>
<p>I think these problem are based on my poor understanding of how <code>torch.Size</code> objects work, and how they are traced.  <a href="https://github.com/pytorch/pytorch/pull/6554" rel="noopener nofollow ugc">This pull request</a> mentions the issue and says that <code>torch.Size</code> objects basically are tuples of torch ints. The use of PyTorch integers is what allows for correct tracking. But when I check, the data type of each entry of a tensor size is an int.</p>
<p>How do <code>torch.Size</code> objects work internally? Of should I manipulate them in order to get correct TorchScript compiling?</p>
<p>Additional note: I know that tracing isn’t guaranteed to compile data-dependent operations correctly, but:</p>
<ol>
<li>often tracing is more convenient than scripting.</li>
<li>Manipulations of tensor sizes should be recorded correctly by tracing</li>
</ol>
<p>I’ve added a MWE to showcase the problem I’m facing. Define the following toy class:</p>
<pre><code class="lang-python">class Foo(nn.Module):
    """Toy class that plays with tensor shape to showcase tracing issue.
    """
    def __init__(self):
        nn.Module.__init__(self)

    def forward(self, x):
        new_shape = (x.shape[0], 2*x.shape[1])  # incriminated instruction
        x2 = torch.empty(size=new_shape)
        x2[:, ::2] = x
        x2[:, 1::2] = x + 1
        return x2
</code></pre>
<p>and run the test code:</p>
<pre><code class="lang-python">x = torch.randn((3, 5))  # create example input

foo = Foo()
traced_foo = torch.jit.trace(foo, x)  # trace
print(traced_foo(x).shape)  # obviously this works
print(traced_foo(x[:, :4]).shape)  # but fails with a different shape!
</code></pre>
<p>So, here the problem is that the tensor sized are hardcoded as constants, instead of being traced. How can I overcome this issue?</p>

username: dtoniolo
cooked: <p>I got an answer on <a href="https://stackoverflow.com/questions/67413808/tracing-tensor-sizes-in-torchscript" rel="noopener nofollow ugc">StackOverflow</a>.</p>
Should I use Embedding layer if I am vectorizing input text myself?

username: p-skaisgiris
cooked: <p>I want to run experiments by training the same model with different text representations: averaging word embeddings (e.g. Word2Vec, FastText), using sentence encoders (e.g. InferSent, USE), and lastly using my own representations acquired by training an AutoEncoder. I am planning to make a function that given an input text converts it into a vector using one of the aforementioned techniques. Thus, if I am vectorizing the input text myself, is there any reason for me to have an Embedding layer? Note, I am not planning to train my vectors further.</p>
<p>Is the Embedding layer just a convenient way to vectorize text and potentially train the word vectors on downstream tasks? Is there any reason to use the Embedding layer if I am using sentence encoders like InferSent? I would imagine I could just take a sentence, pass it through a sentence encoder thus getting a vector, and then pass it into the model.</p>
Distributed data parallel intentionally didn't work on requires_grad=False. why?

username: MakeDirtyCode
cooked: <p>When use DDP on requires=False module,</p>
<pre><code class="lang-auto">AssertionError: DistributedDataParallel is not needed when a module doesn't have any parameter that requires a gradient.
</code></pre>
<p>However, I think it is often good to use DDP even if there is no grad.</p>
<p>Why did block this?</p>

username: huahuanZ
cooked: <p>I think it’s more reasonable to use <code>with torch.no_grad():</code> to wrap up your codes that doesn’t require gradient.</p>
L1 loss between a single image and batch of images

username: sagy101
cooked: <p>Hi,<br>
I’m trying to assess which loss to use in some project and I need to calculate loss between a single image and batch of images.</p>
<p>So far this is what I have is this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3047024042113dd8d59ee7956b4a57cdfb59330.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a3047024042113dd8d59ee7956b4a57cdfb59330" title="Photo from Sagy"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3047024042113dd8d59ee7956b4a57cdfb59330_2_690x163.jpeg" alt="Photo from Sagy" data-base62-sha1="ng7kQcsWZUD9iDjG670KKAAf1L2" width="690" height="163" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3047024042113dd8d59ee7956b4a57cdfb59330_2_690x163.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3047024042113dd8d59ee7956b4a57cdfb59330.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3047024042113dd8d59ee7956b4a57cdfb59330.jpeg 2x" data-dominant-color="313133"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Photo from Sagy</span><span class="informations">805×191 34.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Which seems unnecessary since I don’t really need the same images to appear batch_num times.</p>
<p>Any other way? if i give only one image instead of copying it to be the same size? will by broadcasting it will achieve the same result?</p>

username: JuanFMontesinos
cooked: <p>You can use expand method to the single image. It points to the same mem adress than the original image.<br>
<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.expand" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/tensors.html#torch.Tensor.expand</a></p>

username: sagy101
cooked: <p>Thank didn’t know that existed, good idea</p>
How to avoid inline errors when doing autodiff with loops?

username: strawberrymichi
cooked: <p>Consider some algorithm that looks like this:</p>
<p>phi1 = torch.tensor(torch.empty(size=(p, p, m, m)) , dtype = torch.float, requires_grad = True)<br>
phi2 = torch.tensor(torch.empty(size=(p, p, m, m)) , dtype = torch.float, requires_grad = True)</p>
<p>for s in range§ :<br>
for k in range(s):<br>
phi1[s, k] = phi1[s-1, k] - phi1[s, s] @ phi2[s-1, s-k]<br>
phi2[s, k] = phi2[s-1, k] - phi2[s, s] @ phi1[s-1, s-k]</p>
<p>… [bunch of other functions]</p>
<p>lposterior = function of phi1 and phi2 which yields a constant<br>
lposterior.backward()</p>
<p>When I ran this i got a bunch of errors e.g. in-line errors etc. (presumably because I am slicing phi1 and phi2 and this is not allowed). This algo works fine without the ‘lposterior.backward()’ line. What can I do to make autodiff work here?</p>

username: ariG23498
cooked: <p>I think (just hypothesising) this is due to the <code>torch.empty</code> tensors. You need to initialise the tensors in order for you to have grads for the <code>.backward()</code> method.</p>
<p>You could initialise the <code>phi</code>s with random values and then check to see if the algorithm works and indeed is capable of autograd.</p>
Custom Loss in torch.lib c++

username: f1nsker
cooked: <p>Hi everyone,<br>
i am quite new to the torch.lib and c++ and i’m just wondering, if there any example how should i write and use my own loss function? I’m struggling the whole day through the internet and cannot find any documentation or example for that. I’m looking for DiceLoss or TverskyLoss. I have both on the python side, but now i want to use it also in my c++ code. Do you have any ideas?</p>
<p>Have a nice day!</p>

username: ptrblck
cooked: <p>You could take a look at the internal implementations e.g. for <a href="https://github.com/pytorch/pytorch/blob/4cb534f92ef6f5b2ec99109b0329f93a859ae831/torch/csrc/api/src/nn/modules/loss.cpp#L37-L47"><code>MSELoss</code></a> and the corresponding <a href="https://github.com/pytorch/pytorch/blob/4cb534f92ef6f5b2ec99109b0329f93a859ae831/aten/src/ATen/native/Loss.cpp#L421-L462">functional API via <code>mse_loss</code></a>.</p>
How to solve the 1 positional argument but 2 were given error when extracting intermediate feature as output?

username: JimW
cooked: <p>I am new to Pytorch, and I obtained a pertained model only (without model definition in python), and I can load it using command:</p>
<pre><code class="lang-auto">mnet=torch.hub.load(...)
</code></pre>
<p>which was successful, and I can pass input data to mnet. Now I hope to use the feature from the 3rd last layer as output, so I am doing this:</p>
<pre><code class="lang-auto">mnet = torch.hub.load(...)
self.list_block=list(mnet.children())[:-2]
self.encoder= nn.Sequential(*self.list_block)
</code></pre>
<p>However, when I pass input data to encoder, I got the following error:</p>
<pre><code class="lang-auto">TypeError: forward() takes 1 positional argument but 2 were given
</code></pre>
<p>Could someone provide some guidance on how to fix the above issue?</p>
<p>The input is just a tensor with shape [16, 3, 256, 256], and it worked well for all earlier experiments with mainstream structures. For the new experiments, I just replaced the CNN with a pretrained mnet (also worked well). But the problem is with the “extracting intermediate feature” operation. I searched online, and “take 1 positional argument but 2 were given” error is usually because forward() of self.encoder expects additioin input “self” (which I don’t understand…). I don’t have the model definition, but it was automatically loaded by torch.hub.load(…) command.</p>
How to define a custom loss with pure numpy operation?

username: nickhuangxinyu
cooked: <p>I want to define a custom loss function, with a little modification to RMSE</p>
<p>I want to calculate the rmse of pred head and tail</p>
<p>here is the code:</p>
<pre><code class="lang-auto">from util import *
import torch.optim as opt 
import torch.nn as nn
import torch

def getxy(seed=100):
  np.random.seed(100)
  x = np.random.rand(100, 2)
  w = np.array([3, 7]) 
  b = 5 
  y = (x * w).sum(axis=1) + b  ### remember, keep the shape same with network output!!!
  # y = 3 * x1 + 7* x2 + 5
  return x, y.reshape(-1,1)

class MLP(nn.Module):
  def __init__(self):
    super(MLP, self).__init__()
    self.l1 = nn.Linear(2, 1, bias=True)

  def forward(self, x): 
    return self.l1(torch.tensor(x).double())

class myloss(nn.Module):
  def __init__(self):
    super().__init__()

  #TODO: HOW CAN I MAKE THIS WORK??
  def forward(self, x, y): 
    xarray = x.detach().numpy()
    mask = (xarray &gt; np.quantile(xarray, 0.9, axis=0)) + (xarray &lt; np.quantile(xarray, 0.1, axis=0))
    xarray = np.where(mask, xarray, y)
    l = mask.sum()
    #return ((xarray - y) ** 2).mean(axis=1)
    #return torch.tensor(torch.sum(torch.square(torch.tensor(xarray) - torch.tensor(y))) / l)
    return torch.mean(torch.square(torch.tensor(xarray) - torch.tensor(y)))

  #def backward(grad_output):
    #return grad_output

mlp = MLP().double()
loss = myloss()
o = opt.Adam(mlp.parameters(), lr=0.1)
#o = opt.SGD(mlp.parameters(), lr=0.05)

def train(n):                                                                                                                      
  x, y = getxy()
  for i in range(n):
    pred = mlp(x)
    o.zero_grad()
    los = loss(pred, y)
    los.backward()
    o.step()
    print('%dth: loss = %.3f'%(i, los.item()), mlp.l1.weight.detach().numpy(), mlp.l1.bias.detach().numpy())

if __name__ == '__main__':
  train(10000)
</code></pre>
<p>Can anyone help on this? or provide a demo which can use numpy in custom loss function freely?</p>

username: eqy
cooked: <p>You can implement your own loss function by overriding an existing loss function class like what is done in the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#CrossEntropyLoss" rel="noopener nofollow ugc">loss module</a>, as long as you use only torch operations that are differentiable. I suspect all of the “numpy” operations needed in your custom lost function are implemented in PyTorch.</p>
Using DataParallel in jupyter notebook

username: wasabi
cooked: <p>Hi, I try to use dataparallel in jupyter notebook and I was following the instruction <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Optional: Data Parallelism — PyTorch Tutorials 1.8.1+cu102 documentation</a>. It’s not clear to me how the multiple gpu version works. On top of the page it writes <code>device = torch.device("cuda:0")</code>, isn’t it sending all tensors to the first gpu? Then later it writes <code>model.to(device)</code>, which also sends to the first gpu. Thanks for any clarification.</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> will allocate the model, inputs, and outputs on the default device (or whichever is used in the <code>to()</code> operation) and will then scatter and gather the parameters, data, gradients etc. as described in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a>.<br>
The communication overhead could cause a slowdown compared to <code>DistributedDataParallel</code> using a single process per GPU, which is why we generally recommend to use the latter approach.</p>
How can I get the mid-level output of a trained model?

username: Pimentaw
cooked: <p>Hi I’m a newbie in DL and pytorch and I have some questions.</p>
<p>I want to deploy a DL model in an Android app. (suppose it’s an app about image classification…)</p>
<p>Typically,<br>
we can take a picture to get our input and transform it to tensor type (noted as <code>mInputTensor</code>),<br>
we can use <code>Module.load()</code> method to load our script model (noted as <code>mModule</code> ),<br>
we can use <code>mModule.forward(IValue.from(mInputTensor)).toTensor()</code> method to get our output (noted as <code>mOutputTensor</code>).</p>
<p>However,I just want to get some specfic layer’s output rather than the final output <code>mOutputTensor</code>. (E.g convolution layer’s outout feature map)<br>
So if there a method that I can get such mid-level outputs or if there is a way I can terminate the <code>forward()</code> method before or after a specific layer?</p>

username: JuanFMontesinos
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://blog.paperspace.com/favicon.png" class="site-icon" width="16" height="16">
      <a href="https://blog.paperspace.com/pytorch-hooks-gradient-clipping-debugging/" target="_blank" rel="noopener nofollow ugc" title="12:00PM - 07 July 2019">Paperspace Blog – 7 Jul 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:130/103;"><img src="https://blog.paperspace.com/content/images/2019/07/1_1weEyJRIXCzRlYYQhdZCgA.jpeg" class="thumbnail" width="130" height="103"></div>

<h3><a href="https://blog.paperspace.com/pytorch-hooks-gradient-clipping-debugging/" target="_blank" rel="noopener nofollow ugc">Debugging and Visualisation in PyTorch using Hooks</a></h3>

<p>In this post, we cover debugging and Visualisation in PyTorch. We go over PyTorch hooks and how to use them to debug our backpass, visualise activations and modify gradients.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Update model's parameter with moving average in DDP

username: YutongZheng
cooked: <p>Consider a model training with DDP, all parameters are updated with gradient, except one that is only updated to keep recording the moving average of a value (generated after each forward pass) during training.</p>
<p>Is there an easy way to implemented it in DDP?</p>
<p>Thanks!</p>

username: rvarm1
cooked: <p>Hi,</p>
<p>Do you mean a separate parameter that is not part of <code>model.paramters()</code> which needs to keep track of a moving average cross all workers?</p>
<p>If so, you should be able to just use <code>all_reduce</code> to get the average value of this variable across all workers, and use any typical running mean algorithm to update this on the desired worker.</p>
Exponential Moving Average (EMA) of model weights does not work with Dynamic Networkarchitecture

username: Erica_Zheng
cooked: <p>I implement Exponential Moving Average, it works with a static network architecture. But not for <strong>Dynamic</strong> ones e.g. <em>Stochastic Depth</em>. My experiments shows no weight update at all.<br>
Thank you in advance.</p>
<pre><code class="lang-auto">    model_ema = copy.deepcopy(model)
    for param in model_ema.parameters():
        param.detach_()


def update_ema_variables(model, ema_model, alpha, global_step):
    alpha = min(1 - 1 / (global_step + 1), alpha)
    for ema_param, param in zip(ema_model.parameters(), model.parameters()):
        ema_param.data.mul_(alpha).add_(1 - alpha, param.data)

</code></pre>
Hello everyone! Where is the calculation source code of nn.conv2d?

username: YZW-explorer
cooked: <p>Hello everyone! Where is the calculation source code of nn.conv2d?</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/yzw-explorer">@YZW-explorer</a> , you can find it here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py#L301" class="inline-onebox" rel="noopener nofollow ugc">pytorch/conv.py at master · pytorch/pytorch · GitHub</a></p>
Nonuniform batch size with CUDA MAGMA

username: cam123
cooked: <p>This paper <a href="https://arxiv.org/pdf/2103.13744.pdf" rel="noopener nofollow ugc">KiloNeRF</a> executes many networks in PyTorch with variable batch sizes simultaneously using a MAGMA routine. I need to perform a similar application. I can’t find any details regarding which routine specifically performs this. Can anyone point me in the right direction to using it?</p>
<p>Thank you</p>
How to rotate 2D image using TF.rotate

username: clissa
cooked: <p>Hi, I was trying to write an augmentation pipeline for images and masks in a segmentation task and I got stuck with TF.rotation. In particular, I get an error when rotating the 2D mask. Here’s a reproduction of the error:</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms.functional as TF

shape_image = (3,512,512)
shape_mask = (512, 512)
image = torch.rand(shape_image)
mask = torch.rand(shape_mask)

TF.rotate(image, 15) # runs fine
TF.rotate(mask, 15)
</code></pre>
<blockquote>
<p>RuntimeError: grid_sampler(): expected 4D or 5D input and grid with same number of dimensions, but got input with sizes [1, 512, 512] and grid with sizes [1, 512, 512, 2]</p>
</blockquote>
<p>I have to say I’m new to pytorch so it may be I’m just missing something. Anyway any help would be much appreciated.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: eqy
cooked: <p>This might be because rotate expects inputs to have at least 3 dimensions and the mask just has two. Can you try <code>mask = torch.unsqueeze(mask, 0)</code> to add a dummy “color” dimension to the mask before rotating it?</p>

username: clissa
cooked: <p>Thanks, that work! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
QAT backend for training

username: eefahd
cooked: <p>Hi,<br>
With a quantized model, it’s necessary to set the correct backend (fbgemm or qnnpack) for inference.</p>
<p>But in the quantization aware training, does this backend affect the training?</p>
<p>For instance, can I train the quantized model using fbgemm backend and then use it with the qnnpack in the inference phase!</p>

username: Vasiliy_Kuznetsov
cooked: <p>Hi <a class="mention" href="/u/eefahd">@eefahd</a> ,</p>
<p>There are a couple of things to keep in mind:</p>
<ol>
<li>default qconfigs have different settings for <code>qnnpack</code> and <code>fbgemm</code>.  One setting in particular, <code>reduce_range</code>, if set to False only works correctly in <code>qnnpack</code> and leads to potential overflow in <code>fbgemm</code>.</li>
<li>when weights are packed, the global backend setting is used to determine whether to pack for <code>fbgemm</code> or for <code>qnnpack</code>.</li>
</ol>
Unable to add CUDA to CPATH

username: aneeshaasc
cooked: <p>I’m following installation instructions at: <a href="https://rusty1s.github.io/pytorch_geometric/build/html/notes/installation.html" rel="noopener nofollow ugc">https://rusty1s.github.io/pytorch_geometric/build/html/notes/installation.html</a></p>
<p>Problem:<br>
At part 2 of step 2, <em><strong>Add CUDA to  <code>$PATH</code>  and  <code>$CPATH</code>  (note that your actual CUDA path may vary from  <code>/usr/local/cuda</code> ):</strong></em></p>
<pre><code class="lang-auto">$ CPATH=/usr/local/cuda/include:$CPATH
$ echo $CPATH
&gt;&gt;&gt; /usr/local/cuda/include:...
</code></pre>
<p>The above is what I’m supposed to see  on echo. But I’m running this on Google colab, and an echo doesn’t show anything for me. Its like CPATH is empty. I’ve tried to use export to set the path as well. But still no luck. I’ve also checked that  /usr/local/cuda exists.</p>
<p>Please advise.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/19a93b4290f22ad8940569e8b19d6390b1ba0477.png" alt="cpath%20err" data-base62-sha1="3F0vQ42jU42ItnXM0XEGsrDJwub" width="531" height="196"></p>

username: el_youssfi_azeddine
cooked: <p>Are yous ure that “/usr/local/cuda/include” is your own cuda path?</p>
Custom Backward for `nn.Module` Function other than `forward`

username: rslz
cooked: <p>I am re-implementing the <a href="https://arxiv.org/abs/1811.00995" rel="noopener nofollow ugc">Invertible Residual Networks</a> architecture. This model has, beyond the forward pass, also an <code>inverse</code> function:</p>
<pre><code class="lang-python">class iResNetBlock(nn.Module):
    def __init__(self, input_size, hidden_size):
        self.bottleneck = nn.Sequential(
            LinearContraction(input_size, hidden_size),
            LinearContraction(hidden_size, input_size),
            nn.ReLU(),
        )
        
    def forward(self, x):
        return x + self.bottleneck(x)
    
    def inverse(self, y):
        x = y.clone()
        while not converged:
            # fixed point iteration
            x = y - self.bottleneck(x)
        return x
</code></pre>
<p>I want to add a custom backward pass to the <code>inverse</code> function. Since it is a fixed point iteration, one can make use of the implicit function theorem to avoid unrolling of the loop, and instead compute the gradient by solving a linear system. This is for example done in the <a href="https://arxiv.org/abs/1909.01377" rel="noopener nofollow ugc">Deep Equilibrium Models</a> architecture.</p>
<p>How do I register my custom backwards pass for this function?  I would have hoped that there is some decorator (this is for example <a href="https://jax.readthedocs.io/en/latest/notebooks/Custom_derivative_rules_for_Python_code.html" rel="noopener nofollow ugc">possible in Jax</a>) or other simple way of telling PyTorch to use a certain function as the backward of another function.</p>
<pre><code class="lang-python">class iResNetBlock(nn.Module):
    ....

    def inverse_backwards(self, grad_output)
         # How to tell pytorch to use this as the backward for the .inverse?
</code></pre>
<p>In particular, I want that, when I later define some loss such as  <code>r = loss(y, model.inverse(other_model(model(x))))</code>, that <code>r.backwards()</code> correctly uses my custom gradient for the inverse call.</p>
<p>Ideally the solution should be <code>torchscript</code>-compatible. A colleague suggested putting a <code> inverse=False</code> switch in the forward pass, but (a) I am not sure if this key-val would get passed to the backward pass and (b) this would force me to also implement a <code>backward</code> for the <code>forward</code> function, which I do not want to do. Another Idea came up to create an “Inverse” module, but it seems the parameter sharing between modules is not possible.</p>
Dataloader with clustered batches

username: gebrahimi
cooked: <p>I need to write a dataloader that is customized and I want it to batch some examples that I clustered before using kmeans or any clustering method, together. I am just looking for ideas on how to approach this. Not sure if collate_fn helps here. I appreciate any help. Thanks.</p>

username: ariG23498
cooked: <p>Hey <a class="mention" href="/u/gebrahimi">@gebrahimi</a></p>
<p>I am thinking of storing the examples in a pandas dataframe with a specific column <code>cluster_number</code> to designate the cluster number that the example belongs to.</p>
<p>We can write a CustomDataset for this so that the <code>__getitem__</code> gives us the example as a torch tensor and the cluster it belongs to.</p>
<p>Then comes in the CustomDataLoader that can be used to wrap the CustomDataset.</p>
<blockquote>
<p>The examples would be hard to store in a pandas dataframe if they are anything other than 1D arrays. I would personally save 2D or more in seperate folder and use the <a href="https://pytorch.org/vision/stable/datasets.html#imagefolder" rel="noopener nofollow ugc">ImageFolder</a> dataset.</p>
</blockquote>
Debugging F.softplus() for nan gradient

username: zfzhang
cooked: <p>Hi,<br>
In my multi-layer network, F.softplus(x) gives me nan gradient, and I want to know what x value &amp; incoming gradient is causing it.<br>
What is the best approach to debug?</p>
<p>Thanks!</p>

username: eqy
cooked: <p>Have you tried registering a <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=backward_hook#torch.nn.Module.register_full_backward_hook" rel="noopener nofollow ugc">backward hook</a> to the gradient function to inspect the values?</p>

username: zfzhang
cooked: <p>Thanks!<br>
I am currently trying the variable hook instead of the module hook, plus using lambda function to capture the input x. Seems to work.</p>
Visualizing the texture map in PyTorch3D

username: mohit117
cooked: <p>I was going through the following wonderful Colab Notebook,</p>
<p><a href="https://pytorch3d.org/tutorials/render_textured_meshes" rel="noopener nofollow ugc">PyTorch3D · A library for deep learning with 3D data</a></p>
<p>This colab only has the code without the outputs. My personal copy of this Colab Notebook with outputs can be found below,</p>
<p><a href="https://colab.research.google.com/drive/1dNj7dSsKwNV8Y3Wc4bLhAfufRan3BgW2?usp=sharing#scrollTo=Mm1ecHPaqZeO" rel="noopener nofollow ugc">Copy of render_textured_meshes.ipynb - Colaboratory (google.com)</a></p>
<p>And when I executed the cell</p>
<pre><code class="lang-auto">plt.figure(figsize=(7,7))
texture_image=mesh.textures.maps_padded()
plt.imshow(texture_image.squeeze().cpu().numpy())
plt.grid("off");
plt.axis("off");
</code></pre>
<p>It gave the following output,</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b261fa30878ad0c441930d8e499c8b1be3ea63f6.png" alt="image" data-base62-sha1="ps2U7n2lYBOCkUn5qEHJK3iPsKW" width="394" height="394"></p>
<p>and when I ran the following cell,</p>
<pre><code class="lang-auto">plt.figure(figsize=(7,7))
texturesuv_image_matplotlib(mesh.textures, subsample=None)
plt.grid("off");
plt.axis("off");
</code></pre>
<p>it gave the folowing,</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf62e0a7873a706351021def7f25ef168d2fcff8.png" alt="image" data-base62-sha1="tACGc6WTWr5aIqq4QqtyaXnDBxe" width="395" height="395"></p>
<p>Can someone explain what these output images mean or denote?</p>
<p>Thankyou very much.</p>
How to convert to long while maintaining the learnability

username: MauroGentile
cooked: <p>Hello,<br>
the output of my net is a regression expressing pixels spatial offsets Delta X from the closest marker. I need to use such Delta X as index of a matrix used in the customised loss function. The problem is that if I convert such Delta X into long, the requires_grad of Delta_X  is switch automatically to False, I guess because this is not a linear operation. And as such, the net does not learn. On the other hand, if I don’t converting Delta X into a long, I obviously get an error because the index of a matrix cannot be a float.</p>
<p>Any suggestion on how I can bypass this problem?</p>
<p>Thanks for helping</p>

username: eqy
cooked: <p>This sounds related to quantization; have you tried cloning your “Delta X” into a separate tensor that you convert to long for your indexing? You can use <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.detach" rel="noopener nofollow ugc">detach</a> to keep the clone separate from the original float tensor.</p>

username: googlebot
cooked: <p>if your “customized loss function” is piecewise constant, as integer indexing implies, you’ll have zeros  for loss derivatives, i.e. there is no “right” direction to minimize loss if you discretize. I don’t know what’s your lookup matrix doing, but maybe you can switch to using loss multipliers, so that float Delta_X doesn’t disappear (akin to delta * table[delta.long()] and the answer above)</p>

username: MauroGentile
cooked: <p>Thank you guys I will try and let you know</p>
How do extract features from a new architecture?

username: danielp3011
cooked: <p>I am trying to extract features from a new multi-task learning architecture called MTI-net ( <a href="https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch" rel="noopener nofollow ugc">SimonVandenhende/Multi-Task-Learning-PyTorch: PyTorch implementation of multi-task learning architectures, incl. MTI-Net (ECCV2020). (github.com)</a> ).<br>
Sadly, I only found tutorials for generally used architectures like Resnet …</p>
<p>Can some one point me into the right direction?</p>
<p>Thanks,</p>
Is this usage of MSELoss correct?

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python">crit = nn.MSELoss(reduction='sum').to(setting.device)
 ...
# Do I need to define two criterions 'crit1' and 'crit2'?
# Or just only one criterion 'crit' as below?
extra_loss += crit(A, target1.detach())
extra_loss += crit(B, target2.detach())
</code></pre>
<p>Any help is appreciated.</p>

username: pascal_notsawo
cooked: <p>One is enough. The loss here has no (learnable) parameter (in case you define it as a module), it is just a function (<a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.mse_loss" rel="noopener nofollow ugc">F.mse_loss</a>).</p>
<p>By the way, when you define it as a pytorch module, it is always F.mse_loss that is called in <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/loss.py#L528" rel="noopener nofollow ugc">forward</a> : <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/loss.py#L465" class="inline-onebox" rel="noopener nofollow ugc">pytorch/loss.py at master · pytorch/pytorch · GitHub</a><br>
One of the purposes of the module is that you can do <code>loss.to("device_name")</code> as you did.</p>
While trying to execute the below code, getting the error which I am not able to understand. Plz suggest

username: Dr_Akash_Pethekar
cooked: <p>model = Net()<br>
optimizer = optim.Adam(model.parameters(), lr=0.001)<br>
loss_func = nn.NLLLoss()</p>
<p>epochs = 20<br>
loss_list = []</p>
<p>model.train()<br>
for epoch in range(epochs):<br>
total_loss = []<br>
for batch_idx, (data, target) in enumerate(dataset):<br>
optimizer.zero_grad()<br>
# Forward pass<br>
output = model(data)<br>
# Calculating loss<br>
loss = loss_func(output, target)<br>
# Backward pass<br>
loss.backward()<br>
# Optimize the weights<br>
optimizer.step()</p>
<pre><code>    total_loss.append(loss.item())
loss_list.append(sum(total_loss)/len(total_loss))
print('Training [{:.0f}%]\tLoss: {:.4f}'.format(
    100. * (epoch + 1) / epochs, loss_list[-1]))
</code></pre>
<hr>
<p>NameError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 model = Net()<br>
2 optimizer = optim.Adam(model.parameters(), lr=0.001)<br>
3 loss_func = nn.NLLLoss()<br>
4<br>
5 epochs = 20</p>
<p> in <strong>init</strong>(self)<br>
7         self.fc1 = nn.Linear(256, 64)<br>
8         self.fc2 = nn.Linear(64, 1)<br>
----&gt; 9         self.hybrid = Hybrid(qiskit.Aer.get_backend(‘qasm_simulator’), 100, np.pi / 2)<br>
10<br>
11     def forward(self, x):</p>
<p>NameError: name ‘Hybrid’ is not defined</p>

username: pascal_notsawo
cooked: <p>It seems that you are using a module / function (Hybrib) that you did not import or define in your scripts.</p>
<p>Format your code a bit better by putting it between “``”.<br>
<code>print("my formatted code")</code></p>