akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is small boundary loss an indicator of overfitiing on the boundary data in PINNs?

username: DK13
cooked: <p>I use the PINN approach to solve elliptic BVPs. The loss function I exploit is a weighted sum of the MSE losses of a set of collocation points in a domain $D$ and of a set of boundary points on on the boundary B, i.e. $L=L_D+w * L_B$. Both $L_D$ and $L_B$ become sufficiently small after a number of iterations (epochs), however the loss L_B is 10 to 100 smaller than $L_D$. On the other hand though, because $w$ has to be around $10-100$ for the network to describe well the boundary, the term $w L_B$ is of the same order of $L_D$. Is this an indicator of overfitting on the boundary data or not?</p>
How to use convolutional operations inside pytorch?

username: drogor
cooked: <p>Hello and thanks for your help in advance!</p>
<p>As this is my first time posting here, I’ll try to make my post as clear as possible and am greatly sorry if this has already been answered in the past or is common knowledge.</p>
<p>So, I have just recently started getting into ML and have already written some code with tensorflow, but am switiching over to pytorch right now. At some point inside my code, I would like to convolve a spectrum with a coiflet filter (Reconstruction low/high-pass filter), so a linear convolution of two one-dimensional sequences is needed. Previously, I would do that by using <strong>tf.nn.convolution</strong>, which takes both the spectra and filters as inputs. Pytorch on the other hand uses <strong>torch.nn.Conv1d</strong> with the number of channels as input.</p>
<p>A similar function to what I would be looking for is <strong>numpy.convolve</strong>, but here I am faced with the issue that I have to use a number of .cpu().detach().numpy() operations as I am working on a server, which I would like to avoid. Also this seemes to mess up the backpropagation (must likely a bug in my code).</p>
<p>I hope, I was able to state my problem clearly and sorry for any confusions. Mabe someone could help me understand how to adept torch.nn.Conv1d for my needs or how to avoid detaching/switching the data between server and cpu when using the numpy routine. I am pretty new to pytroch and ML, so any help is appreciated!</p>
<p>Some sample code …<br>
Tensorflow:<br>
<code>dataset_new = tf.nn.convolution(dataset, coiflet, padding='SAME')</code><br>
pytorch:</p>
<pre><code class="lang-auto">for i in range(batch):
        dataset_new [i,:] = torch.from_numpy(np.convolve(dataset[i,:].cpu().detach().numpy(), coiflet, mode='same')).to(device)
</code></pre>
<p>So, my question would be:</p>
<ul>
<li>Is there an efficient way to convolve spectra with predefined filters utilizing only pytorch commands?</li>
</ul>

username: ptrblck
cooked: <p>You could use the functional API via:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.nn.functional as F

data = ... # input data
weight = ... # conv filters
bias = ... # conv bias
out = F.conv2d(data, weight, bias, padding=...)
</code></pre>
<aside class="quote no-group" data-username="drogor" data-post="1" data-topic="137480">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/258eb7/48.png" class="avatar"> drogor:</div>
<blockquote>
<p>Also this seemes to mess up the backpropagation (must likely a bug in my code).</p>
</blockquote>
</aside>
<p>I would guess it’s not a bug in your code but a known limitation, as using numpy operations will break the computation graph since Autograd is not aware of these ops and cannot track them.</p>
Reusing a GPU by multiple distributed processes

username: prabhuteja12
cooked: <p>Hi,<br>
I’m training a model that doesn’t use the full GPU, but I need to train on multiple GPUs. Is there any way to use <code>torch.distributed</code> to accomplish this?</p>
<p>To illustrate, I’ve the following example. It runs on a node with 8GPUs, but I would like to run 12 processes. I launch this using</p>
<pre><code class="lang-python">torchrun --nproc_per_node=12 --standalone gpu_reuse_test.py
</code></pre>
<p>where the file <code> gpu_reuse_test.py</code> is as follows:</p>
<pre><code class="lang-python">import os
from torchvision.models import resnet34
import torch.distributed as dist
import torch
import torch.nn as nn

def main2():
    global_rank = dist.get_rank()
    local_rank = global_rank % 8
    torch.cuda.set_device(local_rank)
    def init_weights(m):
        if isinstance(m, nn.Linear):
            torch.nn.init.xavier_uniform_(m.weight)
            m.bias.data.fill_(0.01 * global_rank)

    a = resnet34().cuda()
    a.apply(init_weights)
    sync_model(a)

    print(f"Local_rank: {local_rank}")


def sync_model(model):
    for p in model.parameters():
        dist.all_reduce(p)

if __name__ == "__main__":
    local_rank = int(os.environ['LOCAL_RANK']) % 8
    dist.init_process_group('nccl', rank=local_rank, world_size=int(os.environ['WORLD_SIZE']))
    main2()
</code></pre>
<p>The code fails with <code>return TCPStore( RuntimeError: Address already in use</code> error.</p>
<p>If I replace the <code>init_process_group</code> call with a plain <code>dist.init_process_group('nccl')</code>,  the <code>all_reduce</code> call fails with</p>
<pre><code class="lang-auto">RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1634272068694/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3
ncclSystemError: System call (socket, malloc, munmap, etc) failed.
</code></pre>
<p>What should I do to fix this error? Can someone help me with this?</p>

username: rvarm1
cooked: <p>Hi, the distributed package assumes that the user will use one process per GPU, so you’d want to use 8 processes in this case. Is there any reason you’re trying to use 12 processes for 8 GPUs? The underlying root cause is that libraries like NCCL won’t work well if multiple processes try to use the same GPU, resulting in deadlocks, hangs etc.</p>

username: prabhuteja12
cooked: <p>I’m trying to benchmark some algorithms which need more than 8 workers, but I have only 8 GPUs for myself.</p>
Sparsemax in PyTorch

username: taha
cooked: <p>Is there any PyTorch implementation of Sparsemax [1]? I found the following implementation in Torch:<br>
<a href="https://github.com/gokceneraslan/SparseMax.torch" rel="nofollow noopener">SparseMax in Torch</a></p>
<p>[1] Martins, Andre, and Ramon Astudillo. “From Softmax to Sparsemax: A sparse model of attention and multi-label classification.” In <em>ICML 2016</em>.</p>

username: jsm
cooked: <p>This might be useful:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/msobroza/SparsemaxPytorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/15339805?s=400&amp;v=4" class="thumbnail onebox-avatar" width="201" height="201">

<h3><a href="https://github.com/msobroza/SparsemaxPytorch" target="_blank" rel="nofollow noopener">msobroza/SparsemaxPytorch</a></h3>

<p>SparsemaxPytorch - SparseMax activation function implementation (ICML 2016) (PyTorch)</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: addisonklinke
cooked: <p>FYI there is also a <a href="https://pypi.org/project/sparsemax/" rel="noopener nofollow ugc">pip package</a> now</p>
Target label images categorical single channel or one hot encoded multichannel

username: YASJAY
cooked: <p>Working on a segmentation problem, converting single channel categorial coded image label to one hot encoded form in the data loader is taking considerable runtime,</p>
<p>does it make sense to create a one hot encoded label dataset?</p>
<p> → How to save single channel label image as png with multichannel one hot encoded form?</p>

username: KFrank
cooked: <p>Hi Srinath!</p>
<aside class="quote no-group" data-username="YASJAY" data-post="1" data-topic="137494" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yasjay/48/12858_2.png" class="avatar"> YASJAY:</div>
<blockquote>
<p>does it make sense to create a one hot encoded label dataset?</p>
</blockquote>
</aside>
<p>In general, I don’t think using one-hot encoding with pytorch makes<br>
sense.</p>
<p>I’ve never been able to think of a non-contrived use case (other than<br>
if you’re working with preexisting one-hot encoded data that you don’t<br>
want to bother transforming or with preexisting code that uses one-hot<br>
encoding that you don’t want to modify) where one-hot encoding would<br>
be preferred.</p>
<p>Pytorch’s workhorse loss criterion for classification problems,<br>
<code>CrossEntropyLoss</code>, expects categorical integer class labels as its<br>
<code>target</code>, so that’s the straightforward choice when using pytorch.</p>
<p>Best.</p>
<p>K. Frank</p>
Why is pytorch tensor addition giving wrong value in the following situation?

username: ADONAI_TZEVAOT
cooked: <p>In a small code snippet in a project…I have a for loop which iterates through 3 tensor values ,  A = torch.tensor([-3. , 6. , -3]).</p>
<pre><code class="lang-auto">det =  torch.tensor(0)
for i in range(3):
    det += A[i]
return det
</code></pre>
<p>Im getting the answer as, det = 2.3842e-07,  all the values in A and det are of type ‘torch.float64’</p>

username: albanD
cooked: <p>On which platform are you testing? I cannot reproduce with:</p>
<pre><code class="lang-python">import torch

def foo(A, acc_dtype):
    det =  torch.tensor(0, dtype=acc_dtype)
    for i in range(3):
        det += A[i]
    print(det)

A = torch.tensor([-3. , 6. , -3])
foo(A, acc_dtype=torch.float)
foo(A, acc_dtype=torch.double)
A = torch.tensor([-3. , 6. , -3], dtype=torch.double)
foo(A, acc_dtype=torch.float)
foo(A, acc_dtype=torch.double)
</code></pre>

username: ptrblck
cooked: <p>Same here. I get a <code>0</code> error in <code>1.11.0.dev20211101+cu113</code>.</p>

username: ADONAI_TZEVAOT
cooked: <p>I tried the following snippet in google colab<br>
Code:</p>
<pre><code class="lang-auto">a = torch.tensor([2.0, 3.0])
b = torch.tensor([5.0, 6.0])

c = torch.stack([a,b], dim = 1)
print(c)
torch.det(c)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([[2., 5.],
        [3., 6.]])
tensor(-2.9999995232)
</code></pre>
<p>The answer should be -3. For my problem statement the outputted answer won’t help, I need it to generate -3. Please help.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is expected behavior. Floating point arithmetic is not exact unfortunately: <a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic" class="inline-onebox">Floating-point arithmetic - Wikipedia</a></p>
Missing some header files in Conda package

username: Benjamin_Bolte1
cooked: <p>Hi! I’ve installed PyTorch using the standard Conda route (showing this version on my machine: <code>pytorch=1.10.0=py3.8_cuda11.3_cudnn8.2.0_0</code>)</p>
<p>I am trying to include this file:</p>
<pre><code class="lang-auto">/path/to/conda/env/lib/python3.8/site-packages/torch/include/torch/csrc/jit/python/pybind_utils.h
</code></pre>
<p>However, when I try to compile my project, it seems to be missing this: <code>torch/csrc/distributed/rpc/py_rref.h</code></p>
<p>It seems as if the wheel was built with <code>USE_DISTRIBUTED=1</code> but then the files are not packaged. What is the correct thing to do in this case?</p>

username: ptrblck
cooked: <p>Could you describe your use case a bit more, please?<br>
Where do you want to include this file and which application claims the <code>py_rref.h</code> is missing?<br>
It sounds as if you would like to build another application using the pre-built conda binaries?</p>

username: Benjamin_Bolte1
cooked: <p>Ah, I solved this by setting some flags in various places. Now getting a different error here → <a href="https://discuss.pytorch.org/t/pybind11-doesnt-recognize-torch-c-scriptmodule-as-torch-module/137367" class="inline-onebox">PyBind11 doesn't recognize `torch._C.ScriptModule` as `torch::jit::Module`</a></p>
The CrossEntropy loss for the train dataset alwas styas around 0.7

username: mmdbrdrn
cooked: <p>Hello,</p>
<p>I have a problem and I hope someone could help me through this.<br>
I am training a resnet34 to classify the gender (male or female) in the CelebA dataset, and I use the nn.CrossentropyLoss() for this task. However, the training loss always stays around 0.7 !! I cannot even overfit my train dataset.</p>
<ul>
<li>I have checked the dataset (images match the annotations).</li>
<li>I have checked the networks. Parameters.require_grad. they are all True.<br>
…<br>
I have also added the network information if it helps. My classification problem is a part of the whole multi-task problem (classification and regression).<br>
I really don’t know where I have missed what.<br>
Thank you so much.</li>
</ul>

username: mmdbrdrn
cooked: <p>class ClassifierLocalizer(nn.Module):</p>
<pre><code>def __init__(self, model_name, num_classes=2):

    super(ClassifierLocalizer, self).__init__()

    self.num_classes = num_classes

  

    # create cnn model

    model = getattr(models, model_name)(pretrained=True)

  

    # remove fc layers and add a new fc layer

    num_features = model.fc.in_features

    model.fc = nn.Linear(num_features, 6) # classifier + localizer

    self.model = model



def forward(self, x):

    x = self.model(x)                    # extract features from CNN

    scores = x[:, :self.num_classes]     # class scores

    coords = x[:, self.num_classes:]     # coordinates

    return scores, coords</code></pre>
Recurrent dropout available?

username: rajarsheem
cooked: <p>I am looking for recurrent dropout for RNN networks. Is this available or any alternative way of implementing it ?</p>

username: SimonW
cooked: <p>Use RNN cells and use dropout layer in between?</p>

username: rajarsheem
cooked: <p>Yes, that’s a workaround. It would be nice if something comes up with the multi-layer networks too.</p>

username: SimonW
cooked: <p>You can also stack those cells together to form a multiplayer rnn though.</p>

username: rajarsheem
cooked: <p>I know right ? <img src="//discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"><br>
I am making a point for convenience. Just think of adding backward recurrence.</p>

username: sushmit_roy
cooked: <p>Can anyone of you please post a working example of this?</p>
Get RuntimeError when using `register_full_backward_hook`

username: Coinc1dens
cooked: <p>Hello, I encountered an error (or to say that’s a bug) when I’m using hook on the pretrained vgg16 model (from torchvision).</p>
<p>I’m trying to get the grads of the intermediate layer (specifically, model.features[15]) while it just raiseError. I googled for a long time and found a temporary solution: using <code>forward_hook</code> and hook the output again to get the grad, it just returns <code>None</code>.</p>
<p>I don’t know if there’s some wrong operations of mine or it may be a possible phenomenon?</p>
<p>Here is my code demo:</p>
<pre><code class="lang-auto">import numpy as np
import torch
from torchvision import models

def get_layers_with_logits(x, model):
    operations = []
    '''
    def hook(module, input, output):
        def h(grad):
            operations.append(grad)
        output.register_hook(h)
    handle = model.features[15].register_forward_hook(hook)
    '''
    def hook(module, input, output):
        print(output)
        operations.append(output)
    handle = model.features[15].register_full_backward_hook(hook)
    
    logits = model(x)
    handle.remove()
    return logits, operations

def main():
    test = torch.rand((1, 3, 224, 224))
    test.requires_grad = True
    with torch.enable_grad():
        model = models.vgg16(pretrained=False, num_classes=1000)
        model.load_state_dict(torch.load('./models_torch/vgg16-397923af.pth'))
        model.eval()
        
        logits, operations = get_layers_with_logits(test, model)
        feature = operations[0]

main()
</code></pre>
<p>and I am using:</p>
<pre><code class="lang-auto">PyTorch: 1.10.0
torchvision 0.11.1
Python 3.7.9
OS: Windows 10</code></pre>

username: ptrblck
cooked: <p>When running your code I’m getting:</p>
<pre><code class="lang-python">RuntimeError: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.
</code></pre>
<p>which seems to point to the inplace <code>nn.ReLU</code>. Replacing the inplace version with an out-of-place version via:</p>
<pre><code class="lang-python">model.features[15] = nn.ReLU()
</code></pre>
<p>seems to solves this issue.<br>
However, I don’t quite understand your use case, as you are registering a backward hook while you are wrapping the code into a <code>no_grad()</code> context (which would disallow a <code>backward</code> call) and are also removing the hook before a <code>backward</code> call is even used.<br>
Are you looking for forward hooks to get the output of <code>features[15]</code>?</p>

username: Coinc1dens
cooked: <p>Thank you for your reply! Now I understand what mistake I’ve made due to the lackness of knowledge using Pytorch…</p>
Is it possible to do a graph surgery on torchscript?

username: pyberry
cooked: <p>Hello,<br>
We are exploring the possibility of graph surgery on a jit scripted model.<br>
The scripted model has a graph object. The graph object has blocks, nodes and variables.<br>
If the graph has 10 nodes, is it possible to create two graph objects with 5 nodes each?</p>

username: Michael_Suo
cooked: <p>This is indeed possible, although if you are looking to do some lightweight graph transformations I would recommend torch.fx</p>

username: robeast
cooked: <p>Hello <a class="mention" href="/u/michael_suo">@Michael_Suo</a>, thanks for your idea. Unfortunately we are bound to work on the JIT scripted model. So we can’t use fx for our purposes.</p>
<p>Do you think this is possible without fx using only the JIT API?</p>
<p>Thanks &amp; Best regards,<br>
RB</p>

username: Michael_Suo
cooked: <p>certainly it’s possible, you can look at some of the passes in <code>torch/crsc/jit/passes</code> as reference</p>

username: leslie-fang-intel
cooked: <p>Hi <a class="mention" href="/u/michael_suo">@Michael_Suo</a>, I am thinking about how to auto add some fork/wait into the graph to enable the interop parallelism. Does the jit-trace/jit-script/FX already have this kind of feature or is it in the plan?</p>
TypeError: torch.nn.parameter.Parameter is not a Module subclass

username: jef
cooked: <p>My model has a custom bias list like below. But it faced an error</p>
<pre><code class="lang-auto">nn.ModuleList([nn.Parameter(torch.randn(1, out_chs, 1, 1)) for _ in range(n_layers)])

--&gt;
TypeError: torch.nn.parameter.Parameter is not a Module subclass
</code></pre>
<p>So I just declare as a list like this. But now it looks it is not affected by <code>model.cuda()</code>. So how to pass the list of <code>nn.Parameter</code> to gpu?</p>
<pre><code class="lang-auto">[nn.Parameter(torch.randn(1, out_chs, 1, 1)) for _ in range(n_layers)]

--&gt;
didn't match because some of the arguments have invalid types:  │ (torch.FloatTensor)
</code></pre>

username: chenyuntc
cooked: <pre><code class="lang-auto">nn.ParameterList
</code></pre>
<p>see docs:</p>
<p><a href="http://pytorch.org/docs/master/nn.html?highlight=parameterlist#torch.nn.ParameterList" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/master/nn.html?highlight=parameterlist#torch.nn.ParameterList</a></p>

username: jef
cooked: <p>Thank you! It works!</p>

username: wangyilan
cooked: <p>wow, so cool! Does the ParameterList is with gradient?</p>

username: ptrblck
cooked: <p><code>nn.ParameterList</code> will make sure to register all parameters properly, so e.g. that they will be pushed to the device, if you call <code>model.to('cuda:0')</code>.<br>
Each <code>nn.Parameter</code> should require gradients by default. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: yuri
cooked: <p>the same happens also with nn.ModuleDict and should be with nn.ParameterDict() replaced</p>
Licence of pre-built libtorch versions

username: allinall
cooked: <p>Dear all,</p>
<p>we would like to use libtorch commercially. For this we aim to use the pre-built binarys, given through the links from the pytorch homepage such as</p>
<ul>
<li><a href="https://download.pytorch.org/libtorch/cu101/libtorch-win-shared-with-deps-1.4.0.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu101/libtorch-win-shared-with-deps-1.4.0.zip</a></li>
</ul>
<p>Unfortunately there is no main license document given in these zip files. However in the official github repository [<a href="https://github.com/pytorch/pytorch/blob/master/LICENSE" class="inline-onebox" rel="noopener nofollow ugc">pytorch/LICENSE at master · pytorch/pytorch · GitHub</a>] there is a leading BSD-license document.</p>
<p>The following questions arise:</p>
<ul>
<li>Are we correct in the assumption, that these pre-built binaries are published with the same license as the leading license given in the github repository?</li>
<li>In the latter case, would it be possible to include the license documents in the pre-built zips? Or is there any specific reason for not including license information in these pre-built zips?</li>
</ul>
<p>Thanks</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/jspisak">@jspisak</a> who would know the licensing of these pre-built libs.</p>

username: jspisak
cooked: <p>Yes, they should be under the same license. <a class="mention" href="/u/malfet">@malfet</a> - can we add a license here as well?</p>

username: allinall
cooked: <p>Thank you for your responses.</p>
<blockquote>
<p>Yes, they should be under the same license. <a class="mention" href="/u/malfet">@malfet</a> - can we add a license here as well?</p>
</blockquote>
<p>Is there any news in this regard?</p>

username: Dennis_Hugle
cooked: <p>+1 to this request. It would make my life much easier.</p>
<p>Cheers</p>
Why gradients of `eigvalsh` always numerically stable?

username: desmondlzy
cooked: <p>Hi,</p>
<p>The doc (v1.10.0) says the gradient of the function <code>eigvalsh</code> is always numerically stable – I’m wondering why it’s the case?</p>
<p>I saw <code>eigvalsh</code> internally invokes <code>eigh</code> of which the backward pass is problematic when there’re repeated eigenvalues. So why the problem just goes away when we only want the derivatives of eigenvalues?</p>
<p>In other words, why only differentiating eigenvalues is ok, both theory-wise and implementation-wise?</p>
<p>Can someone enlighten me?</p>
Inplace modification of constants

username: hwalker
cooked: <p>Hi, I am currently trying to differentiate a function that contains Legendre functions. I define these recursively in a matrix. I wrote a simple code for computing the geometric series that reproduces the error:</p>
<pre><code class="lang-auto">import torch as t

def powers_of_x(N,X):
    Y = t.ones((N,))
    for I in range(1,N):
        Y[I] = Y[I-1]*X
    return Y

X = t.tensor([2.0], requires_grad=True)
SERIES = powers_of_x(9,X).sum()

SERIES.backward()
X.grad
</code></pre>
<p>I guess torch.autograd doesn’t like the fact that I’m overwriting the ones vector, i.e. constant variables. How do I surpass this? I would greatly appreciate any help.</p>

username: ZimoNitrome
cooked: <p>I think a better implementation of <code>powers_of_x(9,X)</code> could be <code>t.pow(X, t.arange(9))</code>:</p>
<p>Updated code:</p>
<pre><code class="lang-python">import torch as t

X = t.tensor([2.0], requires_grad=True)
SERIES = t.pow(X, t.arange(9)).sum()

SERIES.backward()
X.grad
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">tensor([1793.])
</code></pre>

username: hwalker
cooked: <p>Hi, thanks for the reply. What I am trying to figure out, however, is how to use autograd in a recursive function.</p>

username: ZimoNitrome
cooked: <p>I understand. I think torch is complaining about the index assignment…</p>
<p>This version might be slower but works and bases the next value upon the previous one:</p>
<pre><code class="lang-python">import torch as t

def powers_of_x(N,X):
    Y = [t.tensor([1])]
    for _ in range(1,N):
        Y.append(Y[-1] * X)
    return t.stack(Y)

X = t.tensor([2.0], requires_grad=True)
SERIES = powers_of_x(9,X).sum()

SERIES.backward()
X.grad
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">tensor([1793.])
</code></pre>

username: hwalker
cooked: <p>I see. I guess this is an option, though unfortunately not a particularly elegant one. Thank you!</p>

username: ZimoNitrome
cooked: <p>Another option is to just clone the previous item in the assignment:</p>
<pre><code class="lang-python">import torch as t

def powers_of_x(N,X):
    Y = t.ones(N)
    for i in range(1,N):
        Y[i] = Y[i-1].clone()*X
    return Y

X = t.tensor([2.0], requires_grad=True)
SERIES = powers_of_x(9,X).sum()

SERIES.backward()
X.grad
</code></pre>
<p>output:</p>
<pre><code class="lang-python">tensor([1793.])
</code></pre>

username: hwalker
cooked: <p>Thanks, this is precisely what I was looking for.</p>

username: tom
cooked: <p>Note that the stack solution can be quite a bit more efficient once you have substantial computation and perhaps a batch of X to process at the same time.<br>
(And if you have elementwise computation, this is probably not good anyways…)</p>
<p>Best regards</p>
<p>Thomas</p>
Torch.fx cannot trace torch.Size() properly

username: shengchunnan
cooked: <p>torch.fx cannot trace the model properly if there are manipulations of torch.Size() types.<br>
This is the example to try to reconstruct a size type object from the input tensor’s first and second dimensions.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.fx as fx

class Example (nn.Module):
    def __init__(self) -&gt; None:
        super(Example, self).__init__()

    def forward(self, input):

        batch_size = input.size(0)
        ch_size = input.size(1)

        return torch.Size([batch_size, ch_size])

if __name__ == "__main__":

    model = Example()
    y = model(torch.randn(16, 5, 3, 3))
    print(y)

    fx_model = fx.symbolic_trace(model)
    fx_graph = fx_model.graph

    fx_graph.print_tabular()
    print(fx_model.code)
</code></pre>
<p>The terminal output here:</p>
<pre><code class="lang-auto">torch.Size([16, 5])
Traceback (most recent call last):
  File "trace_bug.py", line 23, in &lt;module&gt;
    fx_model = fx.symbolic_trace(model)
  File "/home/shengcn/anaconda3/envs/torch110/lib/python3.7/site-packages/torch/fx/_symbolic_trace.py", line 907, in symbolic_trace
    graph = tracer.trace(root, concrete_args)
  File "/home/shengcn/anaconda3/envs/torch110/lib/python3.7/site-packages/torch/fx/_symbolic_trace.py", line 615, in trace
    self.create_node('output', 'output', (self.create_arg(fn(*args)),), {},
  File "trace_bug.py", line 14, in forward
    return torch.Size([batch_size, ch_size])
TypeError: torch.Size() takes an iterable of 'int' (item 0 is 'Proxy')
</code></pre>

username: Crazyai
cooked: <p>I am a freshman to FX. Maybe <code>fx.wrap</code> can help you.</p>
<pre><code class="lang-auto">@fx.wrap
def get_size(x, y):
    return torch.Size([x, y])

class Example (nn.Module):
    def __init__(self) -&gt; None:
        super(Example, self).__init__()

    def forward(self, input):

        batch_size = input.size(0)
        ch_size = input.size(1)
        # return torch.Size([batch_size, ch_size])
        return get_size(batch_size, ch_size)
</code></pre>

username: Crazyai
cooked: <p>From the documentation, <code>torch.fx</code> marks the method as <code>leaf method</code>, which will be recorded in the graph, instead of being traced.</p>
Init parameter depend on data during first forward

username: Nick_Young
cooked: <p>i have a parameter, and it’s initialized during first forward by:</p>
<pre><code class="lang-auto">def forward(self, x):
    if not self.initialized:
        a = nn.Parameter(torch.mean(x))
        self.initialized = True
</code></pre>
<p>note that, <em>x</em> is input data,</p>
<p>so the problem is,</p>
<p>when using ddp, batch data is different on each devices, so parameter <em>a</em> is initialized differently on each devices. this can not be solved by setting random seed, i think. it’s more like a sync problem.</p>
<p>how can i initialize them all the same?</p>

username: Nick_Young
cooked: <p>thanks for your help!</p>
<p>i added more information. i think it cannot be solved by setting random seed.</p>

username: junb
cooked: <p>Sorry I have misinterpreted your case. I though <code>a</code> was referring to a module parameter but <code>x</code> refers to a  batch of your dataset.</p>
<p>I don’t see any straightforward solution as the batch of inputs will inevitably vary per device. Although, you could get a rough estimate by doing like so:</p>
<ul>
<li>register <code>a</code>  as a parameter when instantiating your model instead of defining it in the forward method:</li>
</ul>
<pre><code class="lang-auto">class Model(torch.nn.Module):

    def __init__(self, a_init):
        super().__init__()
        self.a = torch.nn.Parameter(a_init)
</code></pre>
<ul>
<li>instantiate <code>a</code> while initializing your model through the following :</li>
</ul>
<pre><code class="lang-auto">a_init = torch.mean(x) / n_batch
model = Model(a_init)
</code></pre>
<p>where <code>n_batch</code> denotes your number of batches and <code>x</code> is your whole dataset. This way you will instantiate <code>a</code> as an average of your dataset, rescaled by the number of batches.</p>

username: rvarm1
cooked: <p>Distributed collectives such as broadcast/allreduce can help you initialize them in the same way across different workers.</p>
<p>For example, you can use <code>broadcast</code> to make the value the same as rank 0’s across all ranks, or allreduce to take the mean value across all ranks. The different collectives are documented here: <a href="https://pytorch.org/docs/stable/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.10.0 documentation</a></p>

username: Nick_Young
cooked: <p>thanks, this solved my problem. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=10" title=":+1:" class="emoji" alt=":+1:"></p>
<pre><code class="lang-auto">def forward(self, x):
    if not self.initialized:
        mean_x = torch.mean(x)
        torch.distributed.all_reduce(mean_x, torch.distributed.ReduceOp.SUM)
        a = nn.Parameter(mean_x / torch.distributed.get_world_size())
        self.initialized = True
</code></pre>
Pytorch equivalent of activation=Linear from Keras

username: Mukesh1729
cooked: <p>What is the equivalent of the activation= linear in Pytorchfrom keras? Thanks</p>
<pre><code class="lang-auto">x_out = Conv2DTranspose(output_channels, kernel_size=3, strides=2, padding="same", activation="linear", kernel_initializer="glorot_normal")
</code></pre>

username: Eta_C
cooked: <p>It seems that <code>activation.linear</code> means forwarding without activation?</p><aside class="onebox githubblob" data-onebox-src="https://github.com/keras-team/keras/blob/5c4074a820a08505cf87562f60d21adfdc2dfb05/keras/activations.py#L459-L477">
  <header class="source">

      <a href="https://github.com/keras-team/keras/blob/5c4074a820a08505cf87562f60d21adfdc2dfb05/keras/activations.py#L459-L477" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/keras-team/keras/blob/5c4074a820a08505cf87562f60d21adfdc2dfb05/keras/activations.py#L459-L477" target="_blank" rel="noopener nofollow ugc">keras-team/keras/blob/5c4074a820a08505cf87562f60d21adfdc2dfb05/keras/activations.py#L459-L477</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="459" style="counter-reset: li-counter 458 ;">
          <li>@keras_export('keras.activations.linear')</li>
          <li>@tf.__internal__.dispatch.add_dispatch_support</li>
          <li>def linear(x):</li>
          <li>  """Linear activation function (pass-through).</li>
          <li>
          </li>
<li>  For example:</li>
          <li>
          </li>
<li>  &gt;&gt;&gt; a = tf.constant([-3.0,-1.0, 0.0,1.0,3.0], dtype = tf.float32)</li>
          <li>  &gt;&gt;&gt; b = tf.keras.activations.linear(a)</li>
          <li>  &gt;&gt;&gt; b.numpy()</li>
          <li>  array([-3., -1.,  0.,  1.,  3.], dtype=float32)</li>
          <li>
          </li>
<li>  Args:</li>
          <li>      x: Input tensor.</li>
          <li>
          </li>
<li>  Returns:</li>
          <li>      The input, unmodified.</li>
          <li>  """</li>
          <li>  return x</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Mukesh1729
cooked: <p>Hey  Eta_C, thanks for cofirming.</p>
Cross entropy for diff shaped input and target

username: YASJAY
cooked: <p>My input is of shape torch.Size([16, 4, 224, 224]) and target is of shape  torch.Size([16, 224, 224]), does cross entropy function in pytorch take them as intended inputs?</p>

username: ptrblck
cooked: <p>Yes, as described in <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss">the docs</a>.</p>
Bounding Box returns and data augmentation functions

username: Glerk
cooked: <p>Most of the data augmentation library appears to be useless for modern vision development since it doesn’t take in bounding box and segmentation information or return the parameters of a random transform.</p>
<p>Will this ever be addressed?</p>

username: ptrblck
cooked: <p>I’m not sure which data augmentation libraries you are referring to, so would recommend to take a look at <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">this Detection Tutorial</a> using <code>torchvision</code> and <a href="https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/">these docs for <code>albumentations</code></a>, which are both using bounding boxes in their transformation.<br>
Each <code>torchvision.transforms</code> provides the <code>get_params</code> method, which will return the parameters used for the current setup in the transformation.<br>
In case you are still seeing missing functionalities, I’m sure the code owners would appreciate a pull request. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>
The loss remains same in pytorch nn

username: mmuneeburahman
cooked: <pre><code>class Network(nn.Module):
 
def __init__(self, inputs,hidden_layers, outputs):
    super(Network, self).__init__()
    print(hidden_layers)
    self.fc1 = nn.Linear(inputs, hidden_layers[0])
    self.fc2 = nn.Linear(hidden_layers[0], hidden_layers[1])
    self.fc3 = nn.Linear(hidden_layers[1], hidden_layers[2])
    self.fc4 = nn.Linear(hidden_layers[2], hidden_layers[3])
    self.fc5 = nn.Linear(hidden_layers[3], hidden_layers[4])
    self.fc6 = nn.Linear(hidden_layers[4], hidden_layers[5])
    self.fc9 = nn.Linear(hidden_layers[5], outputs)
    
def forward(self, x):
    out = torch.relu(self.fc1(x))
    out = torch.relu(self.fc2(out))
    out = torch.sigmoid(self.fc3(out))
    out = torch.sigmoid(self.fc4(out))
    out = torch.sigmoid(self.fc5(out))
    out = torch.sigmoid(self.fc6(out))
    out = torch.sigmoid(self.fc9(out))
    return out

inputs = SIZE*SIZE*2
print(inputs)
x = inputs
hidden_layers = []
while x&gt;=2:
    x/=4
    if(x&gt;=2):
        hidden_layers.append(int(x))
print(hidden_layers)
outputs = 1

model = Network(inputs, hidden_layers, outputs)

criterion = nn.MSELoss()
    optimizer = torch.optim.SGD(
    model.parameters(), 
    lr=0.0001, 
    momentum=0.9)'
N_EPOCHS = 100
for epoch in range(N_EPOCHS):
    optimizer.zero_grad()
    model.train()
    for i,(im1, im2, labels) in enumerate(train_dl):
        i1 = torch.flatten(im1,1)
        i2 = torch.flatten(im2,1)
        inp = torch.cat([i1,i2],1)
    
        b_x = Variable(inp) # batch x
        b_y = Variable(labels) # batch y
        y_ = model(b_x).squeeze()
        y_ = (y_&gt;0.5).float()
    
        print(y_)
        print(l)
        loss = criterion(y_,b_y)
        loss = Variable(loss, requires_grad = True)
        print(loss.item())
    
        loss.backward()
        optimizer.step()</code></pre>

username: ptrblck
cooked: <p>You are rewrapping the <code>loss</code> into a new <code>Variable</code> in:</p>
<pre><code class="lang-python">loss = Variable(loss, requires_grad = True)
</code></pre>
<p>and are thus breaking the computation graph.<br>
Remove this line of code and all <code>Variable</code>s as well, since they are deprecated since PyTorch <code>0.4</code>.</p>
How to plot images with unnormalized?

username: GamarChoba
cooked: <p>I have written some code and it is working nicely, but it is normalized. What I want is unnormalized. I do not know how to achieve that.</p>
<pre><code class="lang-auto">def imshow(img, mean, std):
  mean = torch.tensor(mean).reshape(1,3,1,1)
  std = torch.tensor(std).reshape(1,3,1,1)
  return img * std + mean

def show_img(trainloader):
  classes = ["airplane","automobile","bird","cat","deer","dog","frog","horse","ship","truck"]
  dataiter = iter(trainloader)
  batch = next(dataiter)
  labels = batch[1][0:16]
  images = batch[0][0:16]

  for i in range(16):
    print(classes[labels[i]])
    image = images[i].numpy()
    plt.imshow(np.rot90(image.T, k=3))
    plt.show()
</code></pre>

username: ptrblck
cooked: <p>Your custom <code>imshow</code> method seems to remove the normalization already. Assuming you would like to plot the tensor as a uint8 image tensor, you could multiply it with <code>255</code> and transform to <code>torch.unit8</code>.</p>
How to get results (bonding boxes, class IDs, confidences) of Object detection (Yolo v5) in TensorRT

username: Yuki_Nat
cooked: <p>Hello, I’m trying to do object detection(Yolov5) in TensorRT.<br>
But I’m not sure how to get results.</p>
<p>My workflow:<br>
Model is trained with Yolo v5. It works correctly in Pytorch framework.<br>
Convert the model to ONNX format in Ubuntu PC.<br>
Convert the ONNX-format Model to TensorRT in Jetson nano.</p>
<p>Problem:<br>
I inferred with the TensorRT model. But it returns array of [nan, nan, nan, … ,nan].<br>
How to get bounding boxes, confidences, class IDs?</p>
<p>(I post same article in Jetson forum <a href="https://forums.developer.nvidia.com/t/how-to-get-results-bonding-boxes-class-ids-confidences-of-object-detection-yolo-v5-in-tensorrt/195816" rel="noopener nofollow ugc">https://forums.developer.nvidia.com/t/how-to-get-results-bonding-boxes-class-ids-confidences-of-object-detection-yolo-v5-in-tensorrt/195816</a>)</p>
<pre><code class="lang-auto">def main():
    SHOW_IMAGE=True

    data_root="../Image/gratin_od/images"
    files = glob.glob(data_root+"/*.bmp")

    print("Got file names")
    files.sort()
    print(files[75])
    all_judges=np.empty(0)


    # Image shape expected by the post-processing

    with get_engine("./models/gratin.onnx", "./models/gratin.trt") as engine, \
        engine.create_execution_context() as context:

        for i in range(1):
            x_cv = cv2.imread(str(files[75]))
            start = time.time()
            x_show = cv2.cvtColor(x_cv, cv2.COLOR_BGR2RGB)


            inputs[0].host = x_show
            output= common.do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)

            print(output[0])   
            #print&gt;&gt; [nan nan nan ... nan nan nan]
            #How to get bounding box, class id and confidence?


            end = time.time()
            ms = 1000.0*(end-start)
 

            if(SHOW_IMAGE):
                cv2.imshow("image", x_cv)
                cv2.waitKey(0)&amp;0xFF

    print("END")
</code></pre>
How to output an image from a desaturated tensor? In pytorch

username: MACHOCAPTCHA
cooked: <p>I’m trying to load my training data. I do it this way.</p>
<pre><code class="lang-auto">    batch_sizeDif = len (imagDif_Tenzor)
    batchDif = torch.zeros (batch_sizeDif, 3, 493, 733, dtype = torch.uint8)
        
    batch_sizeD = len (imagD_Tenzor)
    batchD = torch.zeros (batch_sizeD, 3, 493, 733, dtype = torch.uint8)
    
    for i, filename in enumerate (imagDif_Tenzor):
     batchDif [i] = torchvision.io.read_image (os.path.join (imagDif_path, filename))
    
    batchDif = setup_aug_tfms ([Saturation (max_lighting = 0.0, p = 1.0, draw = 0.0)])
    
    for i, filename in enumerate (imagD_Tenzor):
     batchD [i] = torchvision.io.read_image (os.path.join (imagD_path, filename))
</code></pre>
<p>Notice the third line from the end. I desaturate the tensor. And then the following code stops working.</p>
<pre><code class="lang-auto">    fig = plt.figure ()
    ax1 = fig.add_subplot (2,2,1)
    ax1.imshow (batchDif [0] .permute (1, 2, 0))
    ax2 = fig.add_subplot (2,2,2)
    ax2.imshow (batchD [0] .permute (1, 2, 0))
    ax3 = fig.add_subplot (2,2,3)
    ax3.imshow (batchDif [20] .permute (1, 2, 0))
    ax4 = fig.add_subplot (2,2,4)
    ax4.imshow (batchD [20] .permute (1, 2, 0))
    plt.show ()
</code></pre>
<p>there is an error in line 3 from the top. ‘Saturation’ object has no attribute ‘permute’.</p>
<p>What attribute to use to display the image? Or do we act differently? If you do not desaturate the images are displayed without errors.</p>

username: MACHOCAPTCHA
cooked: <p>I made some progress on this. Now I use the Grayscale command (num_output_channels = 1) to get a gray picture. However, she still has 3 channels. I have to use resize_ to get what I want [139,1,493,733] instead of [139,3,493,733]. If I have 3 channels, then matplotlib displays the image as gray. If 1 channel is greenish. This is fine?</p>
Resuming checpoint leads to a higher accuracy than keep it running

username: silvester
cooked: <p>Hello experienced ones,</p>
<p>When the training is progressing, I see the test accuracy does not change a lot and is almost steady. However, when I save the checkpoint and resume the iteration from the checkpoint the training show a jump in the test accuracy with a significant margin.</p>
<p>I wonder if you guys know what is going on and possibly provide an explanation for such a random behavior.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: eqy
cooked: <p>It’s hard to understand the situation in detail without any code; are you referring to test accuracy that is run on a separate test dataset?</p>
<p>It might be useful to verify that the test accuracy is consistent when saving a checkpoint vs. when loading the same checkpoint (without any additional training).</p>

username: silvester
cooked: <p>Yes, it is accuracy on the test set.</p>
<p>The test accuracy at the event of saving is equal to the one at the event of resuming.</p>

username: eqy
cooked: <p>Are the other hyperparameters the same upon resuming? It might be easier to understand if there were some concrete code/outputs shown.</p>

username: silvester
cooked: <p>The hyperparameters are the same. Preparing a code snippet due to semi-supervised nature of the training which is performed.</p>
Edge Classification for Heterogeneous Graphs

username: avijit_dasgupta
cooked: <p>I am trying to classify the edges of a heterogeneous graph. I am not sure how to pass the labels for that. I have already created the graph following <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/heterogeneous.html" rel="noopener nofollow ugc">this</a>. Can anyone please point me to some examples for edge classification in Hetero-graphs?</p>
Model Overfits One Feature And Underfits Other Feature. Different Divergence Speed

username: milan_kalkenings
cooked: <p>I am currently working on a multimodal(text input and image input) transformer model.</p>
<p>Trained only on the text input the model converges after 15 epochs</p>
<p>Trained only on the image input the model converges after 30 epochs and performs poorly after 15 epochs.</p>
<p>When I train on both features, the results are close to what the “only on text” model produces</p>
<p>Any idea on how to make my model learn faster on the image component to make the model learn equally fast on both modalities?</p>
Knowledge distillation on bdd100k dataset using Resnet-50 as teacher and my own architecture as student

username: aman_goyal
cooked: <p>Greetings,</p>
<p>This is Aman Goyal. I am currently pursuing research at MSU on knowledge distillation. I had explored various methods but was unable to find any method which distills knowledge on bdd100k detection dataset. I want to use teacher as Resnet-50 and student as my own architecture.</p>
<p>It would be really great if anyone could guide me on the same.</p>
<p>Thanks a lot</p>
How to clear GPU memory after using model?

username: laro
cooked: <p>I’m trying to free up GPU memory after finishing using the model.</p>
<ul>
<li>I checked the <code>nvidia-smi</code> before creating and trainning the model: <code>402MiB /  7973MiB</code>
</li>
<li>After creating and training the model, I checked again the GPU memory status with  <code>nvidia-smi</code>: <code>7801MiB /  7973MiB</code>
</li>
<li>Now I tried to free up GPU memory with:</li>
</ul>
<pre><code class="lang-auto">del model
torch.cuda.empty_cache() 
gc.collect()
</code></pre>
<p>and checked again the GPU memory:  <code>2361MiB /  7973MiB</code></p>
<ul>
<li>As you can see not all the GPU memory was released.</li>
<li>I can only relase the GPU memory via terminal (<code>sudo fuser -v /dev/nvidia*</code>  and <code>kill</code> pid)</li>
</ul>
<p>Is there a way to free up the GPU memory after I done using the model ?</p>

username: ptrblck
cooked: <p>Your code should generally work to free all allocated memory so I guess you might be missing to delete all references to all tensors stored on the GPU.<br>
Once this is done, note that the CUDA context will still use memory to hold all the kernels which might take up to ~1GB depending on the device, CUDA, PyTorch version etc.</p>
Memory problems

username: gokijam
cooked: <p>First Problem:</p>
<blockquote>
<p>RuntimeError: expected scalar type Float but found Double</p>
</blockquote>
<p>I tried to fixed this by adding this:</p>
<blockquote>
<p>i = i.float()<br>
recon = model(i)  <span class="hashtag-raw">#Training</span> model</p>
</blockquote>
<p>After this changing, I received:</p>
<blockquote>
<p>not enough memory: you tried to allocate 10616832000 bytes.</p>
</blockquote>
<p>It is about 10GB, I worked on CPU because my GPU is GTX 970 but i have 32GB of RAM and always have a amount of free 25 GB.</p>
<p>I received suggestion to force this to work on double (float64), is a any chance for that?<br>
My input data is numpy array with matrices 480x480 (MRI data)</p>
<p>CPU: Intel(R) Core™ i7-6700K CPU @ 4.00GHz   4.01 GHz<br>
RAM: 32,0 GB<br>
GPU: GeForce GTX 970</p>
<p>This is university course project, so I don’t want to spend money for cloud GPU.</p>

username: ptrblck
cooked: <p>Based on the error message I don’t think you are running out of memory on your GPU but on your host, so you might want to double check the memory requirements of your script for host RAM.</p>