akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How does deep learning cope with missing values?

Cells_Panda: <p>When performing LSTM time series prediction task, why the input dataset one has missing values nan, the results obtained by the model for each block are all nan tensor, is there any way to make the model ignore these missing values when learning?</p>

ptrblck: <p>Models use mathematical operations in each layer, which are often e.g. matrix multiplications. If any value of your input contains invalid values, these operations would output these values and could create result tensors full of NaNs. You would have to remove these invalid input samples e.g. by replacing them with zeros or any value which would not forward the NaNs.</p>

J_Johnson: <pre><code class="lang-auto">model_input[torch.isnan(model_input)|torch.isinf(model_input)]=0.0
</code></pre>
<p>The above code line will automatically change <code>NaN</code> and <code>inf</code> values to zeroes.</p>
Is it possible to use model parallelism as a blackbox?

wasabi: <p>I would like to perform model parallelism, as the model size is too big to even fit in one GPU. Based on <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="noopener nofollow ugc">Single-Machine Model Parallel Best Practices — PyTorch Tutorials 2.0.0+cu117 documentation</a>, these model parallelism techniques need me to modify the model class. Is there any way to directly parallelize these models as black boxes without modifying their class? Thanks.</p>

ptrblck: <p>You could take a look at <a href="https://pytorch.org/tutorials/intermediate/FSDP_adavnced_tutorial.html?highlight=fsdp">FSDP</a>, which might need less mofications to your code.</p>
How to run a subprocess along a main process in PyTorch?

yxchng1: <pre><code class="lang-auto">import torch.multiprocessing as mp

import torch

def main(local_rank):
    p = mp.Process(target=dummy)
    p.start()

    x = torch.ones(1).cuda()
    y = torch.ones(1).cuda()
    while True:
        x*y
        print("main")
    
    p.join()

def dummy():
    x = torch.ones(1).cuda()
    y = torch.ones(1).cuda()
    while True:
        print("dummy")
        x*y



if __name__ == "__main__":
    #main()
    mp.spawn(main, nprocs=torch.cuda.device_count())
</code></pre>
<p>This code gives the following error</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>which is very weird. How can this small tensor gets out of memory?</p>
<p>I need a subprocess that will not block my main process which uses DDP.</p>

ptrblck: <p>I cannot reproduce the issue and get:</p>
<pre><code class="lang-auto">dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
dummy
main
...
</code></pre>

yxchng1: <p>Sorry. I pasted the wrong example. Should be the following</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

import torch

def main(local_rank):
    p = mp.Process(target=dummy)
    p.start()

    model = torch.nn.Linear(1,1)
    model = torch.nn.DistributedDataParallel(model)
    x = torch.ones(1).cuda()
    y = torch.ones(1).cuda()
    while True:
        x*model(x)
        print("main")
    
    p.join()

def dummy():
    x = torch.ones(1).cuda()
    y = torch.ones(1).cuda()
    while True:
        print("dummy")
        x*y



if __name__ == "__main__":
    #main()
    mp.spawn(main, nprocs=torch.cuda.device_count())
</code></pre>
<p>The main process is not running. I need to use DDP along with a subprocess processing CUDA tensors.</p>
<p>The behavior also looks different when run with CUDA_VISIBLE_DEVICES=0 … and CUDA_VISIBLE_DEVICES=0,1,2,3 …</p>
Cuda works on Numba but not on Torch

Sanjit_Sarda: <p>Hi all,</p>
<p>I am a bit new to torch and have only used it on VMs before. I am now trying to use it on my computer and it doesn’t seem to be working.</p>
<p>When I <code>print(pycuda.driver.Device.count())</code> I get <code>1</code>, however when I <code>print(torch.cuda.is_available())</code> I get <code>None</code>.</p>
<p>I tried reinstalling both torch and cuda and I still get this issue. Can anyone point me in the right direction?</p>

ptrblck: <p>You might have installed the CPU-only binary of PyTorch. Could you check what <code>print(torch.version.cuda)</code> returns?</p>

Sanjit_Sarda: <p>That releases None. Could you point me to how to install a GPU version of PyTorch? I tried uninstalling and reinstalling a cuda version, but I’m unsure if I am entirely cleaning PyTorch.</p>

ptrblck: <p>Thanks for the update as it confirms you’ve installed the CPU-only binary.<br>
Refer the <a href="https://pytorch.org/get-started/locally/">these install instructions</a>, select the desired CUDA runtime (11.7 or 11.8), copy/paste the install command, and make sure the PyTorch binary indicates the CUDA version in its name.<br>
Also, I would recommend installing PyTorch into a clean and empty environment to avoid conflicts with already installed PyTorch versions.<br>
E.g. for <code>2.0.0+cu117</code> a simple <code>pip install torch</code> would work.</p>

Sanjit_Sarda: <p>Sorry for the late reply, but that has worked. Thank you so much!</p>
How to use torch.autograd.grad together with torch.no_grad() in model forward?

paulrev: <p>Hello there!</p>
<p>I want to compute the gradients of a model that has a torch.no_grad operation in the forward function.<br>
The resulting grad in this example is None although x.requires_grad is True when I debug the forward.</p>
<p>Here is an example script:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Network(nn.Module):
    def __init__(self, in_dim=10, out_dim=1):
        super(Network, self).__init__()
        
        self.instancenorm   = nn.InstanceNorm1d(in_dim)
        self.fc = nn.Linear(in_dim, out_dim)

    def forward(self, x):
        
        with torch.no_grad():
            x = self.instancenorm(x)

        x = self.fc(x)
       
        return x


model = Network(in_dim=10, out_dim=1)
model.eval()

x = torch.rand(1,10)
x.requires_grad_(True)

score = model(x)


grad = torch.autograd.grad(
                outputs=score,
                inputs=x,
                allow_unused=True)[0]

print(grad)
</code></pre>
<p>When I remove</p>
<pre><code class="lang-auto">with torch.no_grad(): ...
</code></pre>
<p>it works, however setting x.requires_grad_(True) after torch.no_grad() also results in None.</p>
<p>What causes this and how can I solve it? And why is it no problem in training?</p>
<p>Thanks!</p>

srishti-git1110: <p>Hey, the behaviour is expected.</p>
<p><code>torch.no_grad()</code> essentially makes autograd (PyTorch’s automatic differentiation engine) to “look away”.</p>
<p>Hence, by using it in your forward pass, you are causing the computation graph to break which is why the gradients are None.</p>

paulrev: <p>Thx for your answer! How can I solve this if I want to keep the instance norm layer in the model?</p>
<p>and why is there no problem computing the gradient with backward() during training?</p>

soulitzer: <p>Why are you using no-grad with instance norm?</p>

srishti-git1110: <p>You should remove the <code>no_grad</code> context manager for this to work. As <a class="mention" href="/u/soulitzer">@soulitzer</a> pointed out, what’s the reason you are using instance norm with no_grad?</p>
CUDA does not run

AYAD: <p>Please what do I need to do for the CUDA to run see the picture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57c93e2b3eab570b148cbce01ac772e9c71262cc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/57c93e2b3eab570b148cbce01ac772e9c71262cc" title="torch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57c93e2b3eab570b148cbce01ac772e9c71262cc_2_690x259.jpeg" alt="torch" data-base62-sha1="cwAHrEddHHb9RqzCvmoK8XWqzQU" width="690" height="259" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57c93e2b3eab570b148cbce01ac772e9c71262cc_2_690x259.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/57c93e2b3eab570b148cbce01ac772e9c71262cc_2_1035x388.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/57c93e2b3eab570b148cbce01ac772e9c71262cc.jpeg 2x" data-dominant-color="EDECE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch</span><span class="informations">1192×449 107 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You would need to install a PyTorch binary (pip wheel or conda binary) with CUDA support using the install instructions from <a href="https://pytorch.org/get-started/locally/">here</a>.</p>
Public dynamo regression trackers for nightly

anwang2009: <p>Hi, I’m wondering if there’s a place to track dynamo regressions across nightly torch releases?</p>

marksaroufim: <p>There’s a few relevant PRs</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/issues/96953" class="inline-onebox" rel="noopener nofollow ugc">New PT2 Dashboard Tracker · Issue #96953 · pytorch/pytorch · GitHub</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues/93667" class="inline-onebox" rel="noopener nofollow ugc">14k github models on PyTorch 2.0 pass rates dashboard · Issue #93667 · pytorch/pytorch · GitHub</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues/93794" class="inline-onebox" rel="noopener nofollow ugc">TorchDynamo Performance DashBoard · Issue #93794 · pytorch/pytorch · GitHub</a></li>
</ul>
Is it possible to encounter the "unused parameters" error in DDP despite all parameters participating in loss?

EachOneChew: <p>I believe the error is one that has been brought up already, many times:</p>
<pre><code class="lang-auto">Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by 
making sure all `forward` function outputs participate in calculating loss. 
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 0: 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319
 In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error
</code></pre>
<p>Usually this is because someone has some parameter that doesn’t participate in the forward output, or doesn’t use some part of the forward output. However, I have checked every parameter listed in the error (302-319) AND every other parameter in the model after the backward pass - all have their <code>.grad</code> populated. Somehow a specific subset of parameters are not getting reduced by DDP despite being given gradients by autograd.</p>
<p>Parameters 302-319 also happen to be exactly the parameters of a projection and prediction MLP head. I am sure this is not a coincidence. I try to come up with a minimal reproducible example if I can, but for now I would like to know if the above situation is documented.</p>

irisz: <aside class="quote no-group" data-username="EachOneChew" data-post="1" data-topic="176073">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eachonechew/48/58832_2.png" class="avatar"> EachOneChew:</div>
<blockquote>
<pre><code class="lang-auto">Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by 
</code></pre>
</blockquote>
</aside>
<p>Have you tried passing find_unused_parameters=True when wrapping the model?</p>
<p>If this is not helpful, please file a github issue with a minimal repro.</p>

EachOneChew: <p>I will try this, thank you</p>
How to avoid defragmentation?

mahesh_bhosale: <blockquote>
<p>CUDA out of memory. Tried to allocate 6.85 GiB (GPU 0; 23.69 GiB total capacity; 9.79 GiB already allocated; 2.73 GiB free; 16.13 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>The message says that the reserved memory is significantly large than the allocated memory, we should handle the fragmentation. What needs to be done to handle the fragmentation?</p>

mahesh_bhosale: <p>Can someone please suggest how to avoid this issue? I have already tried freeing the cache and I have blocked the splitting of the blocks by export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128. But it doesn’t help.</p>

irisz: <aside class="quote no-group" data-username="mahesh_bhosale" data-post="1" data-topic="174866">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahesh_bhosale/48/30773_2.png" class="avatar"> mahesh_bhosale:</div>
<blockquote>
<p>CUDA out of memory. Tried to allocate 6.85 GiB</p>
</blockquote>
</aside>
<p>You may wanna get info from how GPU memory is allocated first by using</p>
<pre><code class="lang-auto">torch.cuda.memory_summary(device=None, abbreviated=False)
</code></pre>
<p>You can also try reduce your model size, batch size…</p>
QAT : Could not run 'aten::add.out'

PierreId: <p>Hello,</p>
<p>How is it possible that a simple addition is not working out of the box in QAT with Pytorch 2.0 ?<br>
If I take the QAT example from “<a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.1 documentation</a>” and only add a skip connection :</p>
<pre><code class="lang-auto">def forward(self, x):
        x = self.quant(x)
        _x = x # NEW
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        x = x + _x # NEW
        x = self.dequant(x)
        return x
</code></pre>
<p>I get the error :</p>
<blockquote>
<p>NotImplementedError: Could not run ‘aten::add.out’ with arguments from the ‘QuantizedCPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::add.out’ is only available for these backends: [CPU, CUDA, Meta, …].</p>
</blockquote>
<p>Did I miss something ?</p>

supriyar: <p>you would need to use <code>FloatFunctional</code> to wrap the add operation. The QAT tutorial has more details on how to do this <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#model-architecture" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 2.0.0+cu117 documentation</a></p>
How to do inference/validation with model trained with gumbel-softmax

JimW: <p>I am trying a policy network with gumbel-softmax provided by pytorch.</p>
<blockquote>
<p>r_out = myRNNnetwork(x, h, c)<br>
Policy = F.gumbel_softmax(r_out, temperature, True)</p>
</blockquote>
<p>In the above implementation， r_out is the output from RNN which represents the variable before sampling. It’s a 1x2 float tensor like this: [-0.674, -0.722], and I noticed r_out [0] is always larger than r_out[1].<br>
Then, I sampled policy with gumbel_softmax, and the output will be either [0, 1] or [1, 0] depending on the input signal.</p>
<p>Although r_out [0] is always larger than r_out[1],  the network seems to really learn something meaningful (i.e.  generate correct [0,1] or [1,0] for specific input x).  This actually surprised me. So my first question is:   Is it normal that r_out [0] is always larger than r_out[1] but policy is correct after gumbel-softmax sampling?</p>
<p>In addition,  what is the correct way to perform inference or validation with a model trained like this?  Should I still use gumbel-softmax during inference, which my worry is that it will introduce randomness?  But if I just replaced gumbel-softmax sampling and simply do deterministic r_out.argmax(),  the return is always fixed to [1, 0], which is still not right.</p>
<p>Could someone provide some guidance on this?</p>
Why in DDPG TD3 the critical's loss function decreases and the actor's increases

chamovalera: <p>Why in DDPG TD3 the critical’s loss function decreases and the actor’s increases</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f2d58f62f33c8aa30a45e80ae6d81a76acc0869.png" data-download-href="https://discuss.pytorch.org/uploads/default/7f2d58f62f33c8aa30a45e80ae6d81a76acc0869" title="Captura de pantalla (72)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f2d58f62f33c8aa30a45e80ae6d81a76acc0869_2_690x388.png" alt="Captura de pantalla (72)" data-base62-sha1="i93NjfMxHGD2ND6KmKk8uRxBAcF" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f2d58f62f33c8aa30a45e80ae6d81a76acc0869_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f2d58f62f33c8aa30a45e80ae6d81a76acc0869_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f2d58f62f33c8aa30a45e80ae6d81a76acc0869_2_1380x776.png 2x" data-dominant-color="F8FAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Captura de pantalla (72)</span><span class="informations">1920×1080 166 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
CTC Loss Problems

raraz15: <p>Hello, I unfortunately have to deal with the problematic CTC Loss.</p>
<p>I have a Bidirectional RNN custom module followed by 3 Fully connected layers and I am trying to implement a Speech Recognizer (Based on Deep Speech 2).</p>
<p><code>criterion = torch.nn.CTCLoss(reduction="sum", zero_infinity=True)</code></p>
<p>My Batch Size is 16.<br>
The input sizes are fixed(N_features) but sequence lengths are different between each mini-batch.<br>
I apply BatchNorm1D to my input tensors and not in between layers.<br>
I use torch.int32 for tensors related to targets, and torch.float32 for my input tensors as required by cuDNN.<br>
I also enable cuDNN and tried benchmark mode on and off.</p>
<p>My first problem is when my model is fairly simple, training loss gets stuck around a certain value at couple of epochs. The model predicts all blanks or random characters from the alphabet.</p>
<p>Second problem is that when I increase model complexity, the loss becomes NaN after a few iterations.  This should have been fixed probably by the zero_infinity parameter, but it turns out that it doesn’t.</p>
<p>I know that these are common problems, but I couldn’t find any solutions to it.</p>
<p>UPDATE:</p>
<p>I started using a BatchNorm1D before every non-linearity in my network. Now the network is making not all blank predictions, though they do not make sense in terms of the language. The loss however grew 10 times compared with a singe BatchNorm for the input layer. I am more optimistic as maybe with enough epochs this model can converge.</p>

raraz15: <p>UPDATE:</p>
<p>I run the comet.ml’s adaptation of my model using native Pytorch CTCLoss and it gave reasonable results in no time. I am not able to understand why my approach did not work. If I find out in the future, I will share it here, otherwise; I think the CTCLoss works fine.</p>

bruggy: <p>Any update on this? I’m running into a similar issue</p>
Where can I find tensor.view() source code?

ChaeunLee: <p>Hi,</p>
<p>I am trying to inherit <code>torch.Tensor</code> class to define a new type of <code>MyTensor</code> class which has additional attributes rather than those defined in <code>torch.Tensor</code> class. Please refer to the reference topic: <a href="https://discuss.pytorch.org/t/subclassing-torch-tensor/23754/5">LINK</a>.</p>
<p>But, the problem is that some functions such as <code>torch.Tensor.permute()</code>, <code>torch.Tensor.view()</code>, <code>torch.Tensor.transpose()</code> and others return a new tensor instance which does not include the values of additional attributes. For instance,</p>
<pre><code class="lang-python">a = MyTensor([1.], extra_attr="a")
b = a.view(-1)
</code></pre>
<p>then, though <code>type(b)</code> is <code>MyTensor</code>, it does not inherit <code>extra_attr</code> attributes. So, I am trying to overwrite these functions, but I cannot find the source code for these functions and cannot know the exact arguments for those functions.</p>

soulitzer: <p>To directly answer your question, the source code can be found here <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/TensorShape.cpp#L3647" class="inline-onebox" rel="noopener nofollow ugc">pytorch/TensorShape.cpp at master · pytorch/pytorch · GitHub</a></p>
<p>You might be better of just looking at function signatures though which can be all be found in <a href="https://github.com/pytorch/pytorch/blob/1d08b5b1034c273fb64123c095aa2ab41eae3ea0/aten/src/ATen/native/native_functions.yaml#L7484" class="inline-onebox" rel="noopener nofollow ugc">pytorch/native_functions.yaml at 1d08b5b1034c273fb64123c095aa2ab41eae3ea0 · pytorch/pytorch · GitHub</a></p>
<p>Also, are you looking at <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-with-a-tensor-like-type" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 2.0 documentation</a> for subclassing Tensor?</p>

ChaeunLee: <p>awesome! That’s what I’ve been looking for. thanks.</p>
<p>One more question about <code>.view()</code> implementations. Then, how can <code>.view()</code> function be bound as python class method under <code>torch.Tensor</code> class? I guess that binding just <code>Tensor</code> class in <code>ATen</code> torch c++ frontend implementation is enough?</p>

soulitzer: <p>yeah, the torch.Tensor class inherits from a codegened tensor class written in cpp</p>
Libtorch Makefile

Nick95: <p>Dear all,<br>
I want to use the libtorch libraries inside a kernel module. I wrote a simple Makefile to compile the kernel module:</p>
<pre><code class="lang-auto">obj-m += firewall.o
SRC := $(shell pwd)
KERNEL_SRC ?= /lib/modules/$(shell uname -r)/build/

all:
	$(MAKE) -C $(KERNEL_SRC) M=$(SRC) modules
install:
	cp firewall.ko /lib/modules/$(shell uname -r)/
	modprobe firewall
clean:
	$(MAKE) -C $(KERNEL_SRC) M=$(SRC) clean
</code></pre>
<p>However, I get the following error:</p>
<pre><code class="lang-auto">firewall.c:1:10: fatal error: linux/init.h: No such file or directory
    1 | #include &lt;linux/init.h&gt;         // Macros
</code></pre>
<p>How may I solve?</p>
What provides faster training times? torch.compile() OR custom C++ and cuda extensions?

Munich: <p>I want to decrease my training times for my model. I’ve followed the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html" rel="noopener nofollow ugc">Performance Tuning Guide</a> but I want to increase my performance a lot more. I am thinking my next options are either to use torch.compile() in 2.0 or develop my <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">custom C++ code</a> and use pybind11 to call my C++ pytorch functions in python. Does anyone have any insight as to what is more performant?</p>
<p>Any insight/advice is appreciated.</p>

marksaroufim: <p><code>torch.compile()</code> is an easier thing to try out and will likely give you some speedups, I personally wouldn’t bother with custom c++ code unless you already have a bunch experience. We don’t explicitly compare torch.compile to custom c++ code but instead compare it to eager pytorch code</p>

Munich: <p>Yes, torch.compile() is much easier to try out. I’ll give it a go and see the results. I have a lot of C++ experience so I could write the custom C++ code, but it would take much longer to implement. Since my custom loss functions are a bit complicated with lots of operations, a for loop, and other function calls I suspect that going the custom C++ would be more optimal.</p>
Torchrl, Replay Buffer

Samvel_Safaryan: <p>Let a = ReplayBuffer(), how I can a.add(tuple of (state, action, reward, next_state, done))?</p>

vmoens: <p>With the most recent version of the library, you should be able to do this</p>
<pre><code class="lang-auto">from torchrl.data import ReplayBuffer, ListStorage
rb = ReplayBuffer(batch_size=4, collate_fn=lambda x: x, storage=ListStorage(10))
s, a, sp, r, d = range(5)
rb.add((s, a, sp, r, d))
s, a, sp, r, d = range(5, 10)
rb.add((s, a, sp, r, d))
s, a, sp, r, d = range(10, 15)
rb.add((s, a, sp, r, d))

print(rb.sample())
</code></pre>
<p>(removing the collate function will cause the buffer to call torch.stack on the tuples, which won’t work. The ListStorage is just there to show you how to control the size of the buffer)</p>
<p>which results in</p>
<pre><code class="lang-auto">[(5, 6, 7, 8, 9), (0, 1, 2, 3, 4), (0, 1, 2, 3, 4), (5, 6, 7, 8, 9)]
</code></pre>
<p>ie 4 samples of your replay buffer.</p>
<p>If you want to make the best of the replay buffer, use <code>TensorDictReplayBuffer</code> with a LazyTensorStorage, which will be much faster to write and sample.</p>
<p>Hope that helps!</p>
Torchrl. Import problem

Samvel_Safaryan: <p>Hello everybody. Please help me for solution this problem. I install torchrl use the command<br>
pip install torchrl but after when I want import this library it take me the error<br>
ImportError: /home/whoiam/Desktop/ML/shad_env/lib/python3.10/site-packages/torchrl/_torchrl.so: undefined symbol: _ZNK3c107SymBool10guard_boolEPKcl<br>
I search solution of this one in stackoverflow and other sites but not found.</p>

vmoens: <p>Hi <a class="mention" href="/u/samvel_safaryan">@Samvel_Safaryan</a><br>
May I ask which version of pytorch you’re using?</p>
<p>EDIT:<br>
The binaries should be used with torch 2.0<br>
For other versions, you can still do <code>pip install git+https://github.com/pytorch/rl@v0.1.0</code><br>
For now we want the library to be compatible with previous versions of torch, this is why the pytorch version is not enforced in the setup.py.</p>
Unexpected results with torch.nn.functional.layer_norm

hello-fri-end: <p>Consider the following inputs:</p>
<pre><code class="lang-auto">a = torch.tensor([-3.], dtype=torch.float64)
weight = torch.tensor([1e10], dtype=torch.float64)
bias = torch.tensor([-1], dtype=torch.float64)
normalaxis = (1,)
eps = 0.1
torch.nn.functional.layer_norm(a, normalaxis, weight= weight, bias=bias, eps=eps)
</code></pre>
<p>This returns -1. as expected. In fact, when the weight is between <code>1e0</code> to <code>1e10</code>, the return value is -1. However, when the weight is further increased from <code>1e10</code>, the return value changes (first decreases and then increases). This makes no sense to me since the mean in every case is -3 and therefore, the weight will always get multiplied by 0 according to the layer_norm update equation which means the answer should always be -1 (equal to bias). What am I missing here?</p>

ptrblck: <p>If I’m not mistaken, <a href="https://github.com/pytorch/pytorch/blob/f1978b18f970bf9f92ad6cd73b8ded88ad03fe4b/aten/src/ATen/native/layer_norm.cpp#L202">this code</a> should be called on the CPU and I don’t see any obvious issues there.<br>
Could you create an issue on GitHub so that the code owners could double check and reproduce it, please?</p>

hello-fri-end: <p>Sure, thanks for the reply.</p>
MaskRCNN: Training independent models connected to an ROI Head

CINJ: <p>I’m trying to figure out the appropriate way to train a MaskRCNN model that has key points.</p>
<p>The basis for my approach is here:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://haochen23.github.io/2020/06/fine-tune-mask-rcnn-pytorch.html#.ZCaDRaTZU0F">
  <header class="source">
      <img src="https://haochen23.github.io/images/icon2.png" class="site-icon" width="128" height="128">

      <a href="https://haochen23.github.io/2020/06/fine-tune-mask-rcnn-pytorch.html#.ZCaDRaTZU0F" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 20 June 2020">Eric Chen's Blog – 20 Jun 20</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://haochen23.github.io/2020/06/fine-tune-mask-rcnn-pytorch.html#.ZCaDRaTZU0F" target="_blank" rel="noopener nofollow ugc">Fine-tune PyTorch Pre-trained Mask-RCNN</a></h3>

  <p>This time, we are using PyTorch to train a custom Mask-RCNN. And we are using a different dataset which has mask images (.png files) as . So, we can practice our skills in dealing with different data types. Without any futher ado, let's get into it.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>You can see that in that approach where we fine tune the model, we add a fastrcnn and a MaskRCNN predictor to the ROI Head. In this case we are fine tuning both masks and bounding box.</p>
<p>What I have is a complete different dataset that contains key points - for pose estimation. The reason is simple, Coco provide masks, bounding box, labels and key points for humans, but all other classes only have masks, bounding box and label.</p>
<p>So, how can I add a keypoint predictor to the ROI Head and only train that? Does anyone have experience with this?</p>
<p>Appreciate and advice.</p>
How to inference dictionary data by libtorch API?

Marc_Joshua: <p>Hi, I am trying to inference structured data by libtorch API, the network is similar to vectorNet. And, i want to inference dict data on gpu. The cpu version c++ code as follow, how to modify to gpu version in data construct part?</p>
<pre><code class="lang-auto">// load script model
torch::jit::script::Module module = torch::jit::load(model_path);

// construct fake input data[0]
std::unordered_map&lt;std::string, std::vector&lt;torch::Tensor&gt;&gt; actors_data;
actors_data["types_int"] = {torch::randn({12, 1}), torch::randn({23, 1})};                      // [5, 1]
actors_data["history_speeds"] = {torch::randn({12, 16, 2}), torch::randn({23, 16, 2})};         // [5, 16, 2]
actors_data["history_headings"] = {torch::randn({12, 16, 1}), torch::randn({23, 16, 1})};       // [5, 16, 1]
actors_data["history_trajs"] = {torch::randn({12, 16, 3}), torch::randn({23, 16, 1})};          // [5, 16, 3]
actors_data["history_trajs_relative"] = {torch::randn({12, 16, 3}), torch::randn({23, 16, 3})}; // [5, 16, 3]

// construct fake input data[1]
std::unordered_map&lt;std::string, std::vector&lt;std::vector&lt;torch::Tensor&gt;&gt;&gt; lanes_data;            //[0][32, 6]   [1][32, 6, 60, 2]
lanes_data["centerlines"] = {{torch::randn({10,2}), torch::randn({10,2}), torch::randn({10,2}), torch::randn({10,2}), torch::randn({10,2})}, 
        {torch::randn({10,2}), torch::randn({10,2}), torch::randn({10,2}), torch::randn({10,2}), torch::randn({10,2}), torch::randn({10,2}), torch::randn({10,2})}};

// construct torch::jit::IValue
std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(actors_data);
inputs.push_back(lanes_data);

// do inference
auto outputs = module.forward(inputs).toTuple();
</code></pre>

ptrblck: <p>You could either move the tensors to the GPU via:</p>
<pre><code class="lang-auto">tensor = tensor.to({at::kCUDA, 1});
</code></pre>
<p>or specify the device directly during tensor creation via:</p>
<pre><code class="lang-auto">at::randn({12, 1}, at::kCUDA);
</code></pre>

Marc_Joshua: <p>Thanks for your reply. I have solved this problem by only move dict’s value to gpu, key left. I was confused by how to move full entity(including key and value) of dict to gpu, yesterday.</p>
Difference in inference between pytorch model and exported onnx model

Konrad_X: <p>Hi,</p>
<p>I’ve got a problem with running inference on onnx model exported from torch model by <em>torch.model.export</em>. I am running my code in docker container on multiple machines. Work is as follows:<br>
I create a dummy pytorch model with two hidden layers. Then I export this model into onnx model with some dummy input shape. And then I run inference on both of them on the same data and compare those results. On almost every machine result are completely the same(comparing them with just “==” works perfectly) but on one there are differences up to 1e-05 atol. I’ve tried to run <em>torch.onnx.verification.find_mismatch</em> to somehow compare them but honestly I don’t understand how this function works - it accepts torch model and input shape but where exactly am I supposed to provide the exported onnx model? Also do you maybe have any idea why there are difference only on some machines? Libraries versions, opset version are the same on every machine.</p>
<p>Thanks in advance.</p>

ptrblck: <p>Small numerical mismatches are expected on different setups as internally different algorithms can be used. These algorithms could use a different order of operations which will result in these small errors due to the limited floating point precision.<br>
Here is a small example showing it for two approaches calculating the <code>sum</code> of a tensor:</p>
<pre><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0)
print((s1 - s2).abs())
# tensor(2.2888e-05)
</code></pre>
Is using a single GPU with DDP same as not using DDP?

mahesh_bhosale: <p>I am using DDP for multi-GPU training. I was wondering if I have set up the DDP correctly. To validate, I was going to compare the results of multi-GPU and single-GPU training. Reverting to a single GPU code is cumbersome. So I was wondering, is it the same to use a single GPU in the DDP setting as to not using DDP at all?</p>

cxv0519: <p>No, It’s not exactly same.<br>
Empirically, there are some more overheads in a single learning wrapped with DDP.</p>
<p>But computationally, it’s the same.<br>
It means have no effect on convergence</p>

mahesh_bhosale: <p>Thanks, let me revert back once I have the results. I have started the training.</p>
Column Wise Dot-Product torch.einsum not matching torch.sum(torch.mul(), axis=0)

Shawn_K: <p>I am trying the perform a dot product between the columns of two tensors. I am trying to do this in the most efficient way possible. However, my two methods are not matching up.</p>
<p>My first method using <code>torch.sum(torch.mul(a, b), axis=0)</code> gives me my expected results, <code>torch.einsum('ji, ji -&gt; i', a, b)</code> (take from <a href="https://stackoverflow.com/questions/69702937/efficient-method-to-compute-the-row-wise-dot-product-of-two-square-matrices-of-t" class="inline-onebox" rel="noopener nofollow ugc">Efficient method to compute the row-wise dot product of two square matrices of the same size in PyTorch - Stack Overflow</a>) does not. The reproducible code is below:</p>
<pre><code class="lang-auto">import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
torch.manual_seed(0)

a = torch.randn(3,1, dtype=torch.float).to(device)
b = torch.randn(3,4, dtype=torch.float).to(device)

print(f"a : \n{a}\n")
print(f"b : \n{b}\n")
print(f"Expected:    {a[0,0]*b[0,0] + a[1,0]*b[1,0] + a[2,0]*b[2,0]}")

c = torch.sum(torch.mul(a, b), axis=0)
print(f"sum and mul: {c[0].item()}")

d = torch.einsum('ji, ji -&gt; i', a, b)
print(f"einsum:      {d[0].item()}\n")

print(torch.eq(c,d))
</code></pre>
<p>The output is:<br>
<a href="https://i.stack.imgur.com/b9VBE.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/b9VBE.png" alt="enter image description here" width="624" height="329"></a></p>
<p>Notes:<br>
On the CPU (all I did was remove the <code>.to(device)</code>) the last line <code>torch.eq(c,d)</code> is all true however, I need the tensors to be on the GPU.</p>
<p>Also for some seeds such as <code>torch.manual_seed(100)</code> the tensor are equal…</p>
<p>I feel like it has to be something with einsum because I can get my expect answer other ways.</p>

soulitzer: <p>I think these small discrepancies will be expected given that float32 only has 7-8 digits of precision anyway. You should check that the difference is much smaller with float64.</p>

Shawn_K: <p>Yes, using <code>dtype=torch.float64</code> made the difference smaller but do you happen to know why there is a discrepancy at all between using <code>torch.sum(torch.mul(a, b), axis=0)</code> and <code>torch.einsum('ji, ji -&gt; i', a, b)</code>?</p>

soulitzer: <p>torch.einsum computes the same quantity using a different sequence of operations, in this case - viewing reshaping + batched matrix multiply.</p>
Could anyone help me understand an output of global_mean_pool(x,batch) in torch_geometric.nn

dfdf_pakr: <p>Mathematically, I don’t understand exactly what it means global_mean_pool in torch_geometric.nn.</p>
<p>My data is loaded by dataloader(batch_size=32)</p>
<p>like " x = global_mean_pool(x,batch)". In this case, a shape of x is (900,6) and the shape of batch is (900). As a result, the shape of x=global_mean_pool(x,batch) is (32,6).</p>
<p>what is mean… could anyone help me plz…</p>
Multi-output Classification?

mansoor_sh: <p>I have a dataset consisting of 1 X (Textual Data) and 5 different y (Topics of each text) and each y can take values from 0 to 5. I need to develop a deep learning model to take text and predict all y. I really appreciate it if you could guide me or provide me with some resources to deal with this kind of problem in PyTorch. Also, Am I correct that this problem is considered a multi-output classification? Thanks</p>

AbdulsalamBande: <p>Hello, I believe you can use an LSTM or a transformer ( you can use a pre-trained model for 2 class classification from huggingFace) and add an extra layer for 5 class classification. This can help you<br>
<a href="https://www.youtube.com/watch?v=iCL1TmRQ0sk&amp;ab_channel=Rohan-Paul-AI" rel="noopener nofollow ugc">Multi-class classification</a></p>

akuysal: <p>There are some terminologies called multi-class classification and multi-label classification. If the each document in your dataset belongs to only one topic and the number of topics are more than two, you are talking about multi-class classification. If each document in your dataset may belong to one or more classes at the same time, your problem can be defined as multi-label classification.</p>
<p>I assume that your problem is just multi-class text classification. You can use  AutoModelForSequenceClassification in huggingface pipelines. It is a kind of wrapper for PyTorch and TensorFlow. So, you may produce PyTorch models via huggingface.</p>
<p>As another solution, you can directly employ Pytorch seq2seq models. You may probably use FastText easily. Here is related topic that I created before. For sentiment classification including just two topics, I used FastText before as a deep learning approach.</p><aside class="quote quote-modified" data-post="1" data-topic="167251">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akuysal/48/52484_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/implementation-of-seq2seq-models-for-sentiment-classification/167251">Implementation of Seq2Seq models for sentiment classification</a> <a class="badge-category__wrapper " href="/c/nlp/8"><span data-category-id="8" data-drop-close="true" class="badge-category " title="Topics related to Natural Language Processing"><span class="badge-category__name">nlp</span></span></a>
  </div>
  <blockquote>
    I’m trying to use Seq2Seq models for sentiment classification. I can execute an implementation using torchtext with FastText model on IMDB dataset. It works as expected and I can obtain predictions. However, when I try to replace FastText model with a BiRNN model , I’m getting an error about input sizes as below. Can anyone explain how to correct the error about input sizes? If you can provide a suitable implementation of a Seq2Seq model that I can adapt, it will be ok for me also. I provide the…
  </blockquote>
</aside>
Train 10 models using 4 gpus

yyyyyyy: <p>Hi,</p>
<p>I want to train 10 models simultaneously using multiple gpus. First,  I split the training data into 10 parts, so each models randomly collect a batch of data from one part at each iteration, and they update their parameters independently with the data they receive. Then, after the local update, I need to compute the average of all models and then give this average parameter to all models. So compared with traditional training, the only difference now is we need to compute the average of the parameters of all models after each local update. It is not very difficult to write the code for single gpu, but as we train 10 models here, it takes a lot of time. Anyone knows how to use multiple gpus to accelerate the training?</p>
<p>Here I attach my code which works well in single GPU:</p>
<pre><code class="lang-auto">K = 10#number of models, here it should be 10
models = [Net().to(device) for _ in range(K)]
opts = [opt.SGD(models[i].parameters(), lr) for i in range(K)]

for _ in range(runs):#epoch
        for data in train_loader:
            for i in range(K): 
                #gradient descent over all agents
                x,y = data[i][0].to(device), data[i][1].to(device)# data for i_th model
                models[i].train()
                yp = models[i](x)
                loss = nn.CrossEntropyLoss()(yp,y)
                opts[i].zero_grad()#sgd
                loss.backward()
                opts[i].step()
                models[i].eval()
            
            with torch.no_grad():

                ##combine the parameters of all agents
                net_comb_state_dict_temp = [net.state_dict() for net in models]
                net_comb_state_dict = copy.deepcopy(net_comb_state_dict_temp)# record the w_{i-1}
                weight_keys = list(net_comb_state_dict[0].keys())
                for j in range(K):
                    updated_state_dict = collections.OrderedDict()#construct a dictionary which tracks the order of the added items
                    for key in weight_keys:
                        key_sum = 0
                        for n in range(len(net_comb_state_dict)):
                            key_sum = key_sum + 1/K*net_comb_state_dict[n][key]
                        updated_state_dict[key] = key_sum
                    models[j].load_state_dict(updated_state_dict)
</code></pre>

irisz: <p>We have existing distributed solutions for this.</p>
<p>If you are simply doing data parallel, you may want to look into DistributedDataParallel. This does training in a SPMD style on multiple GPUs. <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.0.0+cu117 documentation</a></p>
Dreamer nan losses

ksridhar: <p>I recently tried running the dreamer implementation in torch rl. But the reconstruction loss (in loss_model_reco.csv), kl divergence loss (in loss_model_kl.csv), the world model loss (in loss_world_model.csv), reward loss (in loss_model_reward.csv), and the world model gradients (in grad_world_model) are all <strong>nan</strong>s. Is this normal? It would be great if there were plots available for the torch rl implementation to compare with. <a class="mention" href="/u/vmoens">@vmoens</a></p>

vmoens: <p>Looking into this thanks for reporting!<br>
Can you open an issue on the repo?</p>

ksridhar: <p>Sure, will open an issue. Thank you for looking into this.</p>
Submodule buffers not being reassigned when moving module to cuda()

nchen9191: <p>Hi,</p>
<p>I have a module that have multiple submodules as attributes. I have wrapped the submodules in nn.ModuleList and nn.ModuleDict. When I call .cuda() on the parent module, the buffers would be copied over to the GPU, but the submodules attributes that originally was the same object as the buffers, is not reassigned to the buffers on the GPU. How do I best fix this? Below is a snippet of a toy code</p>
<pre><code class="lang-auto">import torch


class InnerToyModule(torch.nn.Module):
    def __init__(self, x):
        super(InnerToyModule, self).__init__()
        self.ten = torch.tensor(x)
        self.register_buffer("ten_" + str(x), self.ten)


class ToyModule(torch.nn.Module):

    def __init__(self):
        super(ToyModule, self).__init__()
        self.inner = torch.nn.ModuleList([InnerToyModule(1), InnerToyModule(2)])


if __name__ == '__main__':
    toy = ToyModule()
    print(dict(toy.named_buffers()))
    print(toy.inner[0].ten is dict(toy.named_buffers())["inner.0.ten_1"])
    toy.cuda()
    print(toy.inner[0].ten is dict(toy.named_buffers())["inner.0.ten_1"])
    print(dict(toy.named_buffers())["inner.0.ten_1"])


output:
{'inner.0.ten_1': tensor(1), 'inner.1.ten_2': tensor(2)}
True
False
tensor(1, device='cuda:0')
</code></pre>

AbdulsalamBande: <p><a class="mention" href="/u/nchen9191">@nchen9191</a><br>
When you call <code>.cuda()</code> on a parent module, it recursively applies <code>.cuda()</code> to all of its submodules, but this does not update any references to the original objects. To fix this issue, you can reassign the attributes of the parent module with the CUDA versions of the submodules after calling <code>.cuda()</code> . Here is an example modification of your toy code that demonstrates this:</p>
<blockquote>
<p>example</p>
</blockquote>
<pre><code class="lang-auto">import torch.nn as nn

class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.buffer1 = nn.Parameter(torch.randn(10, 10))
        self.buffer2 = nn.Parameter(torch.randn(10, 10))
        self.module_list = nn.ModuleList([nn.Linear(10, 10) for _ in range(2)])
        self.module_dict = nn.ModuleDict({"linear": nn.Linear(10, 10)})
        
    def forward(self, x):
        x = self.module_list[0](x)
        x = self.module_dict["linear"](x)
        return x
    
model = MyModule()
model.cuda()

# Reassign the attributes with the CUDA versions of the submodules
model.module_list = nn.ModuleList([module.cuda() for module in model.module_list])
model.module_dict["linear"] = model.module_dict["linear"].cuda()

</code></pre>
Idea for CV Project, is it realistic and best approach?

setsailforfail: <p>Hi there!</p>
<p>I love building models from scratch, or I love building stuff in general. I recently built a conditional GAN model, where the trained generator takes a label and creates a fake image, that should resemble the class associated with that label.<br>
Since I do want to transform this little project into something bigger and visually pleasing, I thought about training that model on animals, that is, the generator can create a lion, dog, or whatever it was trained on when given that class as an input. This is something I should be able to do with some time and effort.<br>
However, my main goal is to let the user create a zoo, that means, the user can name some animals, like monkey and lion, and the final picture should be a monkey and a lion in a zoo, in realistic positions. That is, if there is a tree, the monkey should sit in a realistic position on the tree, and the lion should lie on the grass.<br>
Those are just some stupid ideas, but I wonder, if that is achievable at all? If so, where should I start?<br>
Any help is greatly appreciated!</p>