akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is Dynamo a good fit for high-level graph manipulation?

username: kmaeng
cooked: <p>Dear experts:</p>
<p>I have been using torch.fx for high-level graph manipulation using its proxy/retracing (<a href="https://pytorch.org/docs/stable/fx.html#proxy-retracing" class="inline-onebox" rel="noopener nofollow ugc">torch.fx — PyTorch 2.2 documentation</a>).</p>
<p>I found that Dynamo support graph replacement using its subgraph_rewriter in here: <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf#scrollTo=WAIG9-A9VOfF" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a><br>
I am curious if this is strictly superior and is intended to replace torch.fx’s proxy/retracing.</p>
<p>Originally, I thought this is strictly better because now I don’t have to write my own detection and replacement logic. However, I realized that the subgraph_rewriter wants me to define operators like below:</p>
<pre><code class="lang-auto">mul_2 = torch.ops.aten.mul.Tensor(x, 0.5)
mul_3 = torch.ops.aten.mul.Tensor(x, 0.7071067811865476)
erf = torch.ops.aten.erf.default(mul_3)
add_1 = torch.ops.aten.add.Tensor(erf, 1)
mul_4 = torch.ops.aten.mul.Tensor(mul_2, add_1)
</code></pre>
<p>Using torch.fx’s proxy, I was able to express my logic using plain PyTorch language (e.g., <code>0.5 * x * (1 + torch.erf(x / sqrt(2))</code>).</p>
<p>Is Dynamo rewriter only useful for low-level maniputation and should I keep using torch.fx for high-level manipulation? Or is can I use a similar proxy trick inside Dynamo as well?<br>
I am not really interested in optimizing for a specific backend (i.e., not interested in low-level stuffs). I am only interested in playing around at the IR level.</p>
NotOpenSSLWarning when using PyTorch

username: deependujha
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f134e384932604e326610be2e3a0780f71fa5b9.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f134e384932604e326610be2e3a0780f71fa5b9" title="Screenshot 2024-02-21 at 4.05.31 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f134e384932604e326610be2e3a0780f71fa5b9_2_690x203.png" alt="Screenshot 2024-02-21 at 4.05.31 PM" data-base62-sha1="29mwufDUyFsvlsUrvittUKrvaJP" width="690" height="203" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f134e384932604e326610be2e3a0780f71fa5b9_2_690x203.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f134e384932604e326610be2e3a0780f71fa5b9_2_1035x304.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f134e384932604e326610be2e3a0780f71fa5b9_2_1380x406.png 2x" data-dominant-color="262A31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-02-21 at 4.05.31 PM</span><span class="informations">2226×658 79.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How do I fix the error <code>NotOpenSSLWarning</code>?</p>
<p>Error:<br>
NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the ‘ssl’ module is compiled with LibreSSL 2.8.3</p>

username: deependujha
cooked: <p>For someone, having the same issue, you can fix it by using an older version of urllib3 that is compatible.</p>
<p>Here’s the link of the original answer: <a href="https://stackoverflow.com/a/76187415/16022393" rel="noopener nofollow ugc">StackOverFlow NotOpenSSLWarning</a></p>
<p>Crux:</p>
<ol>
<li>Use virtualenv</li>
<li>Install the older version of urllib</li>
</ol>
<pre data-code-wrap="bash"><code class="lang-bash">pip install urllib3==1.26.6
</code></pre>
Determine which CUDA is in use in a Conda environment

username: IcarusWizard
cooked: <p>Hi,</p>
<p>I am a big fan of Conda and always use it to create virtual environments for my experiments since it can manage different versions of CUDA easily. Normally, I will install PyTorch with the recommended conda way, e.g. <code>conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia</code>. It has been working for years without any problem.</p>
<p>But recently, I have some RL experiments that need to run in a docker container which is based on <code>nvidia/cudagl:11.4.2-devel-ubuntu20.04</code> that has a system-level cuda 11.4 install in it. Somehow, the conda install solution doesn’t work anymore. The <code>torch.cuda.is_available</code> always return false. However, when I try to install PyTorch with pip, it seems like it also installs some cuda-related package and make <code>torch.cuda.is_available</code> return true.</p>
<p>Now I am confused. If I have multiple cuda installation from different source, e.g. system-level, conda and pip, which cuda is actually in use? I see there are some post say one can use <code>torch.version.cuda</code> to check the cuda version, but I feel it always returns the version that the code is compiled with, even when <code>torch.cuda.is_available</code> returns false. How exactly is PyTorch indexing these dependencies?</p>

username: ptrblck
cooked: <p>The PyTorch binaries ship with their own CUDA dependencies, which are selected/specified in the install command. Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.</p>

username: IcarusWizard
cooked: <p>Thanks for the reply. So you mean that for the pip installation, some CUDA dependencies is installed <em>inside</em> PyTorch package, so that it never misuses other versions. Is this understanding correct?</p>
<p>But why does conda fail in this case? It should also use the CUDA it shipped with right?</p>

username: ptrblck
cooked: <p>Yes, the pip wheels depend on the CUDA libs hosted on PyPI (e.g. <code>nvidia-cuda-runtime-cu12</code>, <code>nvidia-cudnn-cu12</code>, etc.) while the conda binaries use their corresponding packages from the <code>nvidia</code> conda channel.</p>
<p>I don’t know why the conda binaries do not work inside your container, but outside they seem to work.</p>
/torch/optim/lr_scheduler.py:28): UserWarning: The verbose parameter is deprecated

username: barista
cooked: <p>How can I get rid of this warning? I’m on <code>2.2.0+cu121</code>.</p>
<p><code>/Lib/site-packages/torch/optim/lr_scheduler.py:28): UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate. warnings.warn("The verbose parameter is deprecated. Please use get_last_lr() "</code></p>

username: ptrblck
cooked: <p>Don’t pass the <code>verbose</code> argument to the scheduler as explained in the warning.</p>

username: Bocharick
cooked: <p>Tell me, please, how can I now receive information in the console that ReduceLROnPlateau has processed and changed the LR?<br>
I’m using lightning, and implementing the configure_optimizers method this way:</p>
<pre data-code-wrap="python"><code class="lang-python">    def configure_optimizers(self):
        optimizer = torch.optim.AdamW(lr=self.lr, params=self.parameters())
        return {
            "optimizer": optimizer,
            "lr_scheduler": {
                "scheduler": ReduceLROnPlateau(
                    optimizer=optimizer,
                    verbose=True,
                    min_lr=0.000000001,
                    patience=self.lr_scheduler_patience,
                    threshold=0.001,
                    factor=0.1
                ),
                "monitor": "val_loss",
                'interval': "step",  # "epoch",  # 'step' or 'epoch'
                'frequency': self.lr_scheduler_freq,
            },
        }
</code></pre>
<p>Before the update, verbose worked as expected and, after N validations without improvements, the LR changed, as was written in the console.<br>
Now, when I start a training, I get a warning as described above. The loss change graph shows when ReduceLROnPlateau is working, but no logging occurs in the console. To be honest, it is not clear how to return such logging with new changes.</p>

username: barista
cooked: <aside class="quote no-group" data-username="Bocharick" data-post="3" data-topic="196634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bocharick/48/67573_2.png" class="avatar"> Bocharick:</div>
<blockquote>
<p>To be honest, it is not clear how to return such logging with new changes.</p>
</blockquote>
</aside>
<p>Did you figure it out?</p>

username: Bocharick
cooked: <p>Nope, still waiting <a href="https://discuss.pytorch.org/u/ptrblck">ptrblck’s</a> answer</p>

username: ptrblck
cooked: <p><code>.get_last_lr()</code> is the new and recommended way to check the current learning rate. I don’t know enough about lightning and how they are printing the lr during the training.</p>
Mask-RCNN: can't move training data to GPU

username: sjfrank
cooked: <p>I’m training maskrcnn_resnet50_fpn and creating a Dataset as follows:</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, root_dir):
        self.root_dir = root_dir
        self.image_files = os.listdir(os.path.join(root_dir, 'images'))

    def __len__(self):
        return len(self.image_files)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir, 'images', self.image_files[idx])
        mask_name = os.path.join(self.root_dir, 'masks', self.image_files[idx])

        #image = Image.open(img_name).convert('RGB').resize((1024,1024))
        image = read_image(img_name, mode=ImageReadMode.RGB)
        image = T.Resize((1024,1024))(image)
        mask = Image.open(mask_name).convert('L').resize((1024,1024))

        # Normalize pixel values to [0, 1]
        image = torch.tensor(np.array(image)) / 255.0
        mask = torch.tensor(np.array(mask)) / 255.0
        mask = mask.unsqueeze(0)

        # Apply horizontal flip with 50% probability
        if torch.rand(1) &lt; 0.5:
            image = F.hflip(image)
            mask = F.hflip(mask)

        # Apply vertical flip with 50% probability
        if torch.rand(1) &lt; 0.5:
            image = F.vflip(image)
            mask = F.vflip(mask)

        boxes = masks_to_boxes(mask)
        labels = torch.tensor([1])  # Class label 1
        target = {
            'boxes': boxes,
            'labels': labels,
            'masks': mask
        }
        
        return image, target
</code></pre>
<p>The dataloaders are set up as follows:</p>
<pre><code class="lang-auto">train_dataset = CustomDataset(trainpath)
train_data_loader = DataLoader(train_dataset, batch_size=8, shuffle=True, collate_fn=collate_fn)
valid_dataset = CustomDataset(validatepath)
valid_data_loader = DataLoader(valid_dataset, batch_size=1, shuffle=True, collate_fn=collate_fn)
</code></pre>
<p>where collate_fn is defined as:</p>
<pre><code class="lang-auto">def collate_fn(batch):
    return tuple(zip(*batch))
</code></pre>
<p>That works fine on the CPU.  When I try to move the images and targets to the GPU using <code>images, targets = images.to(device), targets.to(device)</code>, I get the following error:</p>
<pre><code class="lang-auto">---&gt; 56     images, targets = images.to(device), targets.to(device)
AttributeError: 'tuple' object has no attribute 'to'
</code></pre>
<p>Trying to move the items separately,</p>
<pre><code class="lang-auto">images = images.to(device)
targets = targets.to(device)
</code></pre>
<p>throws the same error.  The problem doesn’t seem to be that <code>collate_fn</code> is returning a tuple, and I’ve tried various tweaks to <code>collate_fn</code> but these throw different errors.</p>
<p>Advice appreciated.</p>

username: ptrblck
cooked: <p>Check which of the objects, <code>images</code> or <code>targets</code>, is a <code>tuple</code> and unwrap it. Then move the tensors (assuming the <code>tuple</code> contains tensors) to the device separately.</p>
Changing learning rate after checkpoint

username: Vaibhav_Chahar
cooked: <p>During training of Neural networks in PyTorch, I save a checkpoint with a learning rate 0.005 but when I started the training again from that checkpoint I changed the learning rate to 0.05. Does it affect the training?</p>

username: ptrblck
cooked: <p>Yes, increasing the learning rate will affect the training. E.g. if it’s too high it could “reset” the training again.</p>

username: Vaibhav_Chahar
cooked: <p>If i use a scheduler which reduces my learning rate in every 5 thousand epochs by a factor 0.2, for eg. if i start with lr=0.05, and then it becomes 0.01 then 0.002 and then 0.0004. Does it have affect on training?</p>

username: ptrblck
cooked: <p>Yes, all changes of the learning rate will affect the training.</p>
Using PyTorch in MIG Environment

username: RicketyRick
cooked: <p>Hi!</p>
<p>I am struggling with a virtualization issue. We set up a H100 with 5 partitions. There is no other GPU installed, so only 1 physical device. We are not able to set more than one vGPU as CUDA_VISIBLE_DEVICES. The first one is used, no matter what. I read a lot about pytorch is only able to see one physical gpu if one is in MIG mode and the others aren’t. But holds this true for a single GPU with virtualization?</p>
<p>Examples:</p>
<blockquote>
<p>CUDA_VISIBLE_DEVICES=“MIG-f7267b59-7dd4-5be3-8d25-16ceda48e4ba,MIG-c0d67c52-1804-5a4f-bae7-752caa2b8f21” python test.py</p>
</blockquote>
<p>Is cuda available? True<br>
Device count? 1</p>
<blockquote>
<p>CUDA_VISIBLE_DEVICES=“0,1” python test.py</p>
</blockquote>
<p>Is cuda available? True<br>
Device count? 1</p>
<p>where test.py is</p>
<p>import torch<br>
print(“Is cuda available?”, torch.cuda.is_available())<br>
print(“Device count?”, torch.cuda.device_count())</p>
<p>Thank you very much!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="RicketyRick" data-post="1" data-topic="197445">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/977dab/48.png" class="avatar"> RicketyRick:</div>
<blockquote>
<p>We are not able to set more than one vGPU as CUDA_VISIBLE_DEVICES.</p>
</blockquote>
</aside>
<p>This is expected as MIG allows you to use a single compute instance only.</p>
Torch.cuda.is_available() gives False

username: External_happy
cooked: <p>nvcc -V gives me:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2022 NVIDIA Corporation<br>
Built on Wed_Sep_21_10:33:58_PDT_2022<br>
Cuda compilation tools, release 11.8, V11.8.89<br>
Build cuda_11.8.r11.8/compiler.31833905_0</p>
<p>I tried to install the most recent PyTorch version compatible with CUDA 11.8, but still, “torch.cuda.is_available()” gives False. What is the problem?</p>

username: ptrblck
cooked: <p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension. You would need to properly install a current NVIDIA driver and PyTorch via the provided install instructions from <a href="https://pytorch.org/get-started/locally/">here</a>.</p>

username: Rodrigo_Santos
cooked: <p>I have had the same problem. The easiest way to work around it (<strong>for me</strong>), was install cuda toolkit with conda (I assume you are using a conda installation):</p>
<p><a href="https://hatchjs.com/conda-install-cudatoolkit-cudnn/" rel="noopener nofollow ugc">How to Install CUDA Toolkit and cuDNN with Conda (hatchjs.com)</a></p>
<p><strong>Important advice: Create a virtual env to install all this DL packages …</strong></p>
<p>conda activate &lt;virtual_environment_name&gt;</p>
<p>conda install -c conda-forge cudatoolkit=&lt; CHECK YOUR VERSION &gt; cudnn=&lt; CHECK YOUR VERSION &gt;</p>
What is the recommended way to use (PyTorch naïve quantization) when deploying to int8 TensorRT?

username: Michelvl92
cooked: <p>Hi, the following question is unclear to me, and I was not able to find an answer in Pytorch and on the forum. I have found a similar question on this forum but it is still unclear: <em>“what-is-the-difference-between-pytorch-quantization-and-torch-ao-quantization”</em></p>
<p>What is the recommended and <strong>future-proof</strong> way to use (PyTorch naïve quantization) when deploying to int8 TensorRT?<br>
I am curious about two types of “TensorRT deployment”:</p>
<ol>
<li>The model should be deployed and able to be used with Nvidia tools such as NVIDIA Triton Inference Server</li>
<li>Torch-TensorRT</li>
</ol>
<p>Currently, I am using <a href="https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/index.html" rel="noopener nofollow ugc">Nvidia’s PyTorch Quantization tool</a> which is also in PyTorch’s documentation: <a href="https://pytorch.org/TensorRT/_notebooks/vgg-qat.html" rel="noopener nofollow ugc">DEPLOYING QUANTIZATION AWARE TRAINED MODELS IN INT8 USING TORCH-TENSORRT</a>. With this tool, I can deploy models to Torch-TensorRT or via ONNX →  to TensorRT (although there are some limitations).</p>
<p>But currently, Nvidia’s PyTorch Quantization tool gives a depreciated warning and suggests using PyTorch naive quantization. But from PyTorch Quantization it is unclear which future-proof route I should follow, they are all focused on CPU deployment. For me, it is not a problem if I first need to convert the model to ONNX and then to TensorRT. Or should I now (still) stay at Nvidia’s PyTorch Quantization tool?</p>
Speeding up tensor computation with different size

username: SorrowAir
cooked: <p>Hi,</p>
<p>I’m trying to use a differentiable alignment algorithm (<a href="https://github.com/flatironinstitute/deepblast" class="inline-onebox" rel="noopener nofollow ugc">GitHub - flatironinstitute/deepblast: Neural Networks for Protein Sequence Alignment</a>) to align protein sequence. As the lengths of protein are different, the alignment matrix has different size. As a result, I can’t batch all of these alignment matrices and the computation is very slow. Then, I want to speed up the computation with torch.multiprocessing, but the algorithm requires autograd which is not supported by torch.multiprocessing. So is there any other method to speed up the tensor computation with different size?</p>
Train only one network out of two

username: Salman_Abbasi
cooked: <p>Hi<br>
I have two networks. The input data goes into network 1(to be trained), and then outputs data which goes to network 2(already trained).</p>
<p>The output of network 2 is the loss data, which is compared with the input data to network 1. How can I train network 1 in pytorch, so that the gradients of network 1 are only updated while the loss still backpropagates from network 2 to network 1?</p>
<p>I appreciate simple explanation, with an example code, if possible</p>
<p>Thanks<br>
Sal</p>

username: ptrblck
cooked: <p>You can simply pass the data as explained from network 1 to 2:</p>
<pre data-code-wrap="python"><code class="lang-python"># setup
network1 = nn.Linear(10, 5)
network2 = nn.Linear(5, 10)
optimizer = torch.optim.Adam(network1.parameters(), lr=1e-3)

# freeze network2
for param in network2.parameters():
    param.requires_grad = False

# input data
x = torch.randn(1, 10)

# training
for epoch in range(1000):
    optimizer.zero_grad()
        
    # forward pass
    out = network1(x)
    out = network2(out)
    
    # calculate loss
    loss = F.mse_loss(out, x)
    
    # calculate gradients
    loss.backward()
    
    # weight update
    optimizer.step()
</code></pre>
<p>Note that these networks are just a single linear layer and you would of course need to replace them with real models.</p>

username: Salman_Abbasi
cooked: <p>Thankyou for thd simple example code.It definitely helped.This is what I was hoping to seek.</p>
Why does PyTorch's Transformer Encoder implementation have a norm argument?

username: Goldname
cooked: <p>The TransformerEncoder has this argument: <strong>encoder_layer: an instance of the TransformerEncoderLayer() class (required).</strong></p>
<p>But TransformerEncoderLayer already has LayerNorm built in. Why does TransformerEncoder take in another norm arg: <strong>norm: the layer normalization component (optional).</strong></p>
Conv weights changed after exporting from pytorch (.pt) model to onnx model

username: manix
cooked: <p>Hi Everyone,</p>
<p>I exported a pytorch model to onnx using torch.onnx.export function().<br>
The model has a few Conv Nodes and after exporting observed that the model weights has changed.</p>
<p>I have attached sample conv weights from both torch(1.13.0) and Onnx(1.15.0)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac843f903ba129bd4ef16577d5e2e59c8233a34b.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac843f903ba129bd4ef16577d5e2e59c8233a34b" title="Onnx Conv weights"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac843f903ba129bd4ef16577d5e2e59c8233a34b.png" alt="Onnx Conv weights" data-base62-sha1="oC9sRH89zaPGZuWJHxIfsPZL4QH" width="433" height="500" data-dominant-color="252525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Onnx Conv weights</span><span class="informations">681×786 18.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b891cc44c99f839e5d50c360c3e2be8190467c20.png" data-download-href="https://discuss.pytorch.org/uploads/default/b891cc44c99f839e5d50c360c3e2be8190467c20" title="PT Conv Weight"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b891cc44c99f839e5d50c360c3e2be8190467c20.png" alt="PT Conv Weight" data-base62-sha1="qkMdXzCuLxynC0GKMQ3n0P48P4Y" width="616" height="500" data-dominant-color="1F1F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PT Conv Weight</span><span class="informations">956×775 17.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Observation</strong> : [0][0] of all the 8 3x3 matrix has significantly changed</p>
<p>If we are thinking about inference or input data/activations, I agree with the variance.<br>
But we are looking at model weights here.</p>
<p>I am trying to understand what is happening under the hood of torch.onnx.export,</p>
<p>is there any way to keep weights without much changes,(or very minimal precision changes)? And what is the actual reason for us to have difference in the  weights when we are exporting <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> ?</p>
Adamw param.mul_(1 - lr * weight_decay) RuntimeError: result type ComplexFloat can't be cast to the desired output type Float

username: bigtree
cooked: <p>I use the adamw as the optimizer and after the training run a day I got this problem:</p>
<pre><code class="lang-auto">[epoch][s/s_per_e/gs]: [99][304/319/31899], lr: 0.000001000796, loss: 0.251922130585
[epoch][s/s_per_e/gs]: [99][305/319/31900], lr: 0.000001000000, loss: 0.198185890913
Traceback (most recent call last):
File "train_main.py", line 745, in
main()
File "train_main.py", line 740, in main
main_worker(0, ngpus_per_node, args)
File "train_main.py", line 591, in main_worker
optimizer.step()
File "/home/bigtree/miniconda3/envs/color/lib/python3.7/site-packages/torch/optim/optimizer.py", line 89, in wrapper
return func(*args, **kwargs)
File "/home/bigtree/miniconda3/envs/color/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/bigtree/miniconda3/envs/color/lib/python3.7/site-packages/torch/optim/adamw.py", line 121, in step
group['eps'])
File "/home/bigtree/miniconda3/envs/color/lib/python3.7/site-packages/torch/optim/&lt;em&gt;functional.py", line 122, in adamw
param.mul&lt;/em&gt;(1 - lr * weight_decay).

RuntimeError: result type ComplexFloat can't be cast to the desired output type Float
</code></pre>
<p>Please help.</p>

username: bigtree
cooked: <p>It turns out that the lr goes to very small &lt; 1e-6. After fixed this the problem solved.</p>

username: ptrblck
cooked: <p>That sounds quite weird. Do you have a minimal code snippet to reproduce the error message by changing the learning rate?</p>

username: Suraj_Patni
cooked: <p>I was getting the same issue at line <code>optimizer.step()</code>. The error was:<br>
<code>RuntimeError: result type ComplexFloat can't be cast to the desired output type Float</code>.<br>
<strong>Solution:</strong><br>
The problem was that the learning rate was taking complex values. So once I fixed that, the issue was resolved.<br>
I was setting the learning rate using:</p>
<pre data-code-wrap="python"><code class="lang-python">for param_group in optimizer.param_groups:
	current_lr = &lt;custom function to calc learning rate&gt;
    param_group['lr'] = current_lr
</code></pre>
Unusual compiler behavior

username: AidanGoldfarb
cooked: <p>A compiled function will hang forever without a print statement. When the print statement in <code>mul()</code> is uncommented, function will run fine at any size input (n in main). If the print statement is left commented, only n &lt;= 8 will work. Here is a reduced example.</p>
<pre><code class="lang-auto">import torch,random

def mul(a, b):
    #print("") #uncomment to unhang 
    n = len(a)
    c = [[[0]*n]*n]*n
    for i in range(n):
        for j in range(n):
            for k in range(n):
                c[i][j][k] = 0
                for l in range(n):
                    c[i][j][k] += a[i][j][l] * b[i][j][k]
    return c

def main():
    n = 16 #change to 8 to unhang
    t1,t2 = [[[random.random() for _ in range(n)] for _ in range(n)] for _ in range(n)],[[[random.random() for _ in range(n)] for _ in range(n)] for _ in range(n)]

    fun = torch.compile(mul)
    fun(t1,t2)

if __name__ == "__main__":
    main()```</code></pre>
Back propogation after applying a function to output

username: alireza_samadi
cooked: <p>I am training a unet and my model output is [B,C,D,H,W]. in my forward pass I apply a complex mathematical formula to my output and I wish to compute loss over the output of this function after prediction and my ground truth.This formula could be something like:<br>
output_from_channel0*( -  torch.exp(-10/ output_from_channel1))*torch.exp(-5/ output_from_channel2)<br>
i.e. :<br>
outputs = model(input)</p>
<p>outputs2=a_complex_mathematical_formula_on_tensors(outputs)</p>
<p>my_loss=loss(ground_truth  ,  outputs2)<br>
my_loss.backward()<br>
optimizer.step()<br>
but apparantly this doesnt work<br>
I see <a href="https://discuss.pytorch.org/t/apply-a-skimage-or-any-function-to-output-before-loss/80097">here</a> is mentioned Backward needs to be implemented manually. How this should be done? The documentation of AUTOGRAD seems high level and quite complicated.</p>

username: JuanFMontesinos
cooked: <p>You have an example here<br>
<a href="https://pytorch.org/tutorials/intermediate/custom_function_conv_bn_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/custom_function_conv_bn_tutorial.html</a></p>
Correctly changing precision in DLRM

username: Rishabh_Jain
cooked: <p>Hello,</p>
<p>I would like to change the precision used in DLRM inference. I observed that the default datatype used in the original code is “float32”, in the <a href="https://github.com/facebookresearch/dlrm/blob/9830654a2b33f0ff11900fb1622e095a23eb2cab/dlrm_s_pytorch.py#L284" rel="noopener nofollow ugc">create_emb()</a> method. Is changing the dtype to any of the <a href="https://pytorch.org/docs/stable/tensor_attributes.html" rel="noopener nofollow ugc">supported values</a> enough to correctly change the precision?</p>
<p>I believe the generated embedding table would be directly used in the <a href="https://github.com/facebookresearch/dlrm/blob/9830654a2b33f0ff11900fb1622e095a23eb2cab/dlrm_s_pytorch.py#L456" rel="noopener nofollow ugc">apply_emb()</a> function.</p>
<p>Thanks for the help.<br>
Rishabh Jain</p>

username: HDCharles
cooked: <p>for some changes, like fp32 to fp16 or bf16 that is sufficient if those ops support those dtypes, for actual quanitzation you need to quantize your model which takes more effort, see: <a href="https://pytorch.org/docs/stable/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.2 documentation</a> for more details</p>
Pytorch Equivalent for kernel_regulariser in Tensorflow

username: Sai_Kishore
cooked: <p>I am trying to convert a Tensorflow Code to Pytorch. I am stuck at implementing this line.</p>
<pre><code class="lang-auto">Conv3DTranspose(nb_channels, kernel_size=(3, 3, 3), kernel_regularizer=l2(0.001), strides=_strides, padding='same')(y)
</code></pre>
<p>How to add a kernel_regulariser in pytorch.</p>

username: ptrblck
cooked: <p>You could either calculate the L2 norm of these weights manually and add it to the loss as seen e.g. <a href="https://discuss.pytorch.org/t/how-to-implement-pytorch-equivalent-of-keras-kernel-weight-regulariser/99773/2">here</a> or you could add this parameter with <code>weight_decay</code> to the optimizer.</p>

username: Sai_Kishore
cooked: <p>Thanks a Lot, Peter, a small doubt. I have several layers which need kernel_regularisation. Will it suffice to add just one model.layer.weight parameter??.</p>

username: ptrblck
cooked: <p>No, you would have to add all layers which should be regularized to the loss calculation or to the optimizer’s parameter list containing the <code>weight_decay</code>.</p>

username: Sai_Kishore
cooked: <p>Any Links for the weight Decay Solution??</p>

username: ptrblck
cooked: <p>The <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">Optimizer - Per-parameter options docs</a> give you an example how to pass different parameter groups to an optimizer which can be used to e.g. spcify weight decay only for a specific group.</p>

username: Sai_Kishore
cooked: <p>Thanks a ton, Peter.</p>

username: th3gentlecod3r
cooked: <p>Been learning PyTorch since last week. I have a doubt. Why do we have to do everything imperatively in PyTorch as compared to TensorFlow where we just declare what we want and that’s it? Like in case of regularizer, we can just use <code>tf.keras.regularizers.L2</code>.</p>
Initialising a DDP model twice slows down training

username: Raider
cooked: <p>Hi,<br>
When I accidentally wrapped the model <strong>twice</strong> in the DistributedDataParallel Class, each epoch started taking longer to complete when compared to wrapping it <strong>once</strong>.</p>
<p>For Example:</p>
<pre><code class="lang-auto">model = DistributedDataParallel(model)
# Maybe load some checkpoint
model = DistributedDataParallel(model)
</code></pre>
<p>is slower than</p>
<pre><code class="lang-auto">model = DistributedDataParallel(model)
# Maybe load some checkpoint
</code></pre>
<p>My Setup:<br>
4 Nodes with 1 gpu in each.<br>
I know there is no reason for wrapping a model twice but I was wondering why is it taking longer, in fact, significantly longer, in my case.</p>
<p>Thanks</p>

username: Raider
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> if you could shed some light on this, it’ll be insightful.<br>
thanks.</p>

username: ptrblck
cooked: <p>I don’t know what would happen if you wrap <code> DDP</code> module again.<br>
Could you create a full (Nsight Systems) profile and compare the timelines between both runs to check if anything stands out?</p>
Why not multiprocess pin_memory in data loader?

username: mengwanguc
cooked: <p>In PyTorch data loader, there is a separate thread for pinning memory.<br>
However, pinning memory can be CPU-intensive as it needs to copy the tensors. When the tensors are very large and model computations are very fast, this pin_memory thread might become the bottleneck.</p>
<p>Why don’t we do multiprocessing for pin_memory as well, just like for workers?<br>
Is there any specific reason, or challenge?</p>

username: albanD
cooked: <p>Hi!</p>
<p>This is because, to send the Tensor between threads, we need to put it in shared memory so that both processes can access it. Unfortunately, it is not possible to have memory that is both shared and pinned. So we need to send it to the final process first and only then can be pin it.</p>
<p>If it is a bottleneck for you, it should be relatively simple to have multiple pinning threads in the main process to speed it up when you have a lot of worker threads.</p>
Any chance to torch.export DTensor Module

username: Vremold
cooked: <p>Hi. Recently, I’m using torch.export to export a DNN module expressed in DTensor.<br>
It raises the following error:</p>
<pre><code class="lang-auto">RuntimeError: aot_export is not currently supported with traceable tensor subclass.
</code></pre>
<p>To repro the error, you may need to slightly change the <code>detect_fake_mode</code> API in <code>torch/_guards.py</code> to enable it to detect FakeTensorMode in wrapper tensor subclasses, like</p>
<pre><code class="lang-auto">def detect_fake_mode(inputs: Any = None):
    ...
    for i, flat_input in enumerate(flat_inputs):
        from torch.utils._python_dispatch import is_traceable_wrapper_subclass
        if isinstance(flat_input, FakeTensor):
            fake_modes.append((flat_input.fake_mode, "fake tensor input", i))
        elif is_traceable_wrapper_subclass(flat_input):
            attrs, _ = flat_input.__tensor_flatten__()
            for attr in attrs:
                inner_tensor = getattr(flat_input, attr)
                if isinstance(inner_tensor, FakeTensor):
                    fake_modes.append((inner_tensor.fake_mode, "fake inner tensor input", i))
</code></pre>
<p>I am posting this post to ask is there any chance to torch.export nn.Module expressed in DTensor. If so, may I know a general plan or approach? Thanks ahead for any reply.</p>

username: wanchaol
cooked: <p>Hi <a class="mention" href="/u/vremold">@Vremold</a> Thanks for the question here! I wonder what’s your use case with torch.export + DTensor? We haven’t planned on torch.export with DTensor yet.</p>
<p>As of today’s nightly we have torch.compile + DTensor already working, would that work for you?</p>

username: Vremold
cooked: <p>Thanks for the reply. This is for the following consideration. After we have trained a model expressed in DTensor, compared to torch.compile, I guess torch.export is more suitable for model deployment. That’s why I want to torch.export a DTensor model.</p>
[WinError 1455] The paging file is too small for this operation to complete

username: Sarmad_GTU
cooked: <p>[WinError 1455] The paging file is too small for this operation to complete. Error loading “C:\Users\gtu\anaconda3\envs\Deeplearning\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll” or one of its dependencies.</p>
<p>How can I overcome this error?</p>

username: ptrblck
cooked: <p>I don’t use Windows, but <a href="https://stackoverflow.com/questions/64837376/how-to-efficiently-run-multiple-pytorch-processes-models-at-once-traceback">this issue</a> came as a direct result searching for your error message.</p>

username: cobryan05
cooked: <p>I have managed to mitigate (although not completely solve) this issue.  I posted a more detailed explanation at the <a href="https://stackoverflow.com/questions/64837376/how-to-efficiently-run-multiple-pytorch-processes-models-at-once-traceback" rel="noopener nofollow ugc">StackOverflow link</a> given by ptrblck, but basically try this:</p>
<p>Download:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/cobryan05/7d1fe28dd370e110a372c4d268dcb2e5">
  <header class="source">

      <a href="https://gist.github.com/cobryan05/7d1fe28dd370e110a372c4d268dcb2e5" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/cobryan05/7d1fe28dd370e110a372c4d268dcb2e5" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/cobryan05/7d1fe28dd370e110a372c4d268dcb2e5</a></h4>

  <h5>fixNvPe.py</h5>
  <pre><code class="Python"># Simple script to disable ASLR and make .nv_fatb sections read-only
# Requires: pefile  ( python -m pip install pefile )
# Usage:  fixNvPe.py --input path/to/*.dll

import argparse
import pefile
import glob
import os
import shutil
</code></pre>
    This file has been truncated. <a href="https://gist.github.com/cobryan05/7d1fe28dd370e110a372c4d268dcb2e5" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Install dependency:<br>
<code>python -m pip install pefile</code></p>
<p>Run (for OPs paths) (NOTE: THIS WILL MODIFY YOUR DLLS [although it will back them up]):<br>
<code>python fixNvPe.py --input C:\Users\gtu\anaconda3\envs\Deeplearning\lib\site-packages\torch\lib\*.dll</code></p>

username: mebelmag36
cooked: <p>Thanks. Your solution is really helped to me</p>

username: azj.n
cooked: <p>I tried the script and it is giving me the same error, I also tried changing the torch version, does it mean that my memory is just not enough?</p>
Lazy loading in ConcatDataset

username: jsetty
cooked: <p>Hello, I have around 400 GB of image data which I have stored in n .nc files (Working with xarrays). I instantiate n Dataset objects, one for each file and use ConcatDataset. I do not want to load them all on memory together. Therefore, I lazy load the dataset onto memory using the logic below inside <code>__getitem__</code>:</p>
<pre><code class="lang-auto">if not self.isDataLoaded:
    self.data = xr.load_dataset(self.data_file, engine="h5netcdf")
    self.isDataLoaded= True
</code></pre>
<p>With <code>shuffle=False</code>, my expectation was that only one file will be loaded and iterated before going to the next one. However, all the files are loaded onto memory in the beginning of the epoch leading to OOM issues. I would understand if the memory usage increased gradually during the epoch because I still have not figured out how to identify end of iteration for that single dataset and delete the loaded memory. Could anyone spot an issue here? Thanks!</p>

username: JinroTorch
cooked: <p>Hi! hope you are doing well:</p>
<p>Regarding your issue, it is because <code>xr.load_dataset</code>, in the documentation, says:</p>
<blockquote>
<p>“Open, load into memory, and close a Dataset from a file or file-like object.”</p>
</blockquote>
<p>Unless you have more than 400GB of ram, this will not be possible. Instead, you would like to try <code>xr.open_dataset</code> which performs a <strong>lazy load</strong> by using chunks (dask package is required).</p>
<p>If you are trying to load multiple <code>.nc</code> files, use <code>xr.open_mfdataset</code> and specify the coordinates to concatenate the file.</p>
<p>Note: not related to the question, this just happened to me. If you are installing dask along with xarray in a new environment and it is not recognized by the xarray library, try reinstalling dask after the environment is created.</p>
Stylegan2 Training NaN for gp and loss_critic resulting to black pictures

username: Leandro_Francia
cooked: <p>I’ve recreated a code from a guide and follow every step and when the time that I try to train the model with my own pictures which are in 128x128 the resulting process for the gp and loss_critic results to a NaN. So I am guessing there might be some problems in my data or the equations inside the code. Asking for more insights on this problem. Below are the lines from gp and loss_critic</p>
<pre><code class="lang-auto">gp = gradient_penalty(critic, real, fake, device=DEVICE)
</code></pre>
<pre><code class="lang-auto">loss_critic = -(torch.mean(critic_real) - torch.mean(critic_fake)) + LAMBDA_GP * gp + (0.001 * torch.mean(critic_real ** 2))
</code></pre>
<p>and below is the guide that I am following. Thank you in advance.<br>
<a href="https://blog.paperspace.com/implementation-stylegan2-from-scratch/#load-all-dependencies-we-need" rel="noopener nofollow ugc">https://blog.paperspace.com/implementation-stylegan2-from-scratch/#load-all-dependencies-we-need</a></p>
Changing number of parameters in optimizer during training

username: mbutt
cooked: <p>Hello everyone</p>
<p>I am trying to optimize images (not model) so I have an optimizer instantiated like</p>
<p>optimizer_img = torch.optim.SGD([image, ], lr=args.lr_img, momentum=0.5)</p>
<p>where image is 100x3x32x32 tensor.</p>
<p>For next iteration, I want to use image tensor of shape 50x3x32x32.</p>
<p>At loss.backward() I am getting “RuntimeError: Function SliceBackward0 returned an invalid gradient at index 0 - got [50, 3, 32, 32] but expected shape compatible with [100, 3, 32, 32]”</p>
<p>I have tried reinitializing the optimizer (which I don’t want to do).</p>
<p>Tried to update like this “optimizer_img.param_groups[0][‘params’][0].data = new_image_tensor”</p>
<p>Also tried, partail function solution here <a href="https://github.com/pytorch/pytorch/issues/97603" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/97603</a></p>
<p>The error persists in all cases. Is there any possible solution?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>Why don’t you want to reinitialize the optimizer given you are optimizing entirely new samples in each iteration?</p>

username: mbutt
cooked: <p>Actually the samples are not entirely new, they are the samples optimized in previous iteration and then we process them so the number of samples become half of what it was.</p>
<p>Other than that I am still getting the same error even if I am reinitializing the optimizer in each iteration.</p>
Evaluating PyTorch for 2D to 3D Product Model Rendering from Brochures

username: onlyriccardo
cooked: <p>Hello PyTorch Community,</p>
<p>I’m delving into a project with the aim to develop a system capable of automatically generating <strong>3D models</strong> of products using <strong>2D images</strong> from <strong>brochures</strong>. The challenge is to design a scalable solution starting from a modest collection of 2D images, which may also include specific dimensions to aid in achieving accurate scale in the rendered models.</p>
<p>I am specifically looking at PyTorch as the potential backbone for this project due to its widespread adoption and the extensive support network. Before I commit, I would greatly appreciate your insights on a few key points:</p>
<ul>
<li>Is PyTorch suited for managing the 2D to 3D conversion process, particularly with a <strong>limited number of images</strong>?</li>
<li>Can PyTorch effectively <strong>integrate physical dimensions</strong> into the model to ensure the final 3D renderings are to scale?</li>
<li>Given PyTorch’s capabilities, what is the expected <strong>quality</strong> of the 3D model output considering the constraints of initial data input?</li>
</ul>
<p>Additionally, if there are particular PyTorch libraries, extensions, or APIs that you think would be especially beneficial for this application, I would be eager to learn about them.</p>
<p>I understand the complexity of what I’m proposing and am seeking your expert advice on whether PyTorch is the right choice for this endeavor. If there are alternative tools or frameworks that you believe are more suited to these specific requirements, I’m open to suggestions and would value your recommendations.</p>
<p>Thank you for sharing your expertise and helping guide this project in the right direction!</p>
Which AT_DISPATCH_ - function can we take for mixed long and float type tensors (CUDA extension)?

username: Max_Unhold
cooked: <p>There is a list of all DispatchTypes in <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/Dispatch.h" rel="noopener nofollow ugc">DispatchTypes</a> but it lacks any<br>
but since I could not find any documentation on when to use wich AT_DISPATCH_ function, my question is the following:</p>
<p><em>How do I select the right function to dispatch both long and float tensors for CUDA?</em></p>
<p>There are older but finally unanswered question here<br>
<a href="https://discuss.pytorch.org/t/cuda-extensions-scalar-t-float-and-int-mix/70470">CUDA extensions, scalar_t float and int mix - PyTorch Forums</a> and in <a href="https://discuss.pytorch.org/t/aten-cuda-kernel-dispatch-for-type/18990">ATen cuda kernel dispatch for type - PyTorch Forums</a>.</p>
<p>It is actually quite important if you want to write a CUDA kernel with more than just one argument tensor type.</p>
<hr>
<p>Potential solutions:<br>
If we use AT_DISPATCH_SWITCH as</p>
<pre><code class="lang-auto">    AT_DISPATCH_SWITCH(self.scalar_type(), "op_name",
        AT_DISPATCH_CASE_INTEGRAL_TYPES([&amp;] {
          op_integral&lt;scalar_t&gt;(iter);
        })
        AT_DISPATCH_CASE_FLOATING_TYPES([&amp;] {
          op_floating&lt;scalar_t&gt;(iter);
        })
        AT_DISPATCH_CASE(kBool, [&amp;] {
          op_bool(iter);
        })
    );
</code></pre>
<p>we appear to have three different operations (op_integral,op_floating,op_bool) instead of one of different dtypes.</p>
CUDA extensions, scalar_t float and int mix

username: MartinBagge
cooked: <p>Hello,</p>
<p>So basically i want to port this project <a href="https://github.com/ayanc/fdscs" rel="nofollow noopener">https://github.com/ayanc/fdscs</a> to pyorch.<br>
As the project has custom kernel written for CUDA Tensorflow I would need to get these CUDA kernels to compile with pytorch instead.<br>
I have been following the tutorial on how to write a CUDA extension for pytorch, but since it seems to only mention floats I can’t seem to get it to compile properly with integers and floats mixed.</p>
<p>I’m creating the template&lt;typename scalar_t&gt; but as I see it this basically gets translated to double internally. So what I need is a way to represent an integer which I guess should also be with some internal template?</p>
<p>And when dispatching ie. AT_DISPATCH_ALL_TYPES to allow all types which type should I write as the first argument of the function?</p>
<p>Hope you can help, and at least thanks for looking at my problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: MartinBagge
cooked: <p>I think I found out a bit about scalar_t.</p>
<p>So I think that all arrays should be defined with scalar_t and AT_DISPATH_ALL_TYPES defines that scalar_t can be any type and should be checked for any array. If instead of ALL_TYPES are AT_DISPATCH_FLOAT_TYPES scalar_t is only checked against floating types. What is used if you only have integer types I have no idea.</p>

username: ignaciogavier
cooked: <p>I think you should use the AT_DISPATCH_INTEGRAL_TYPES that is defined at <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/Dispatch.h" rel="noopener nofollow ugc">Dispatch.h</a> if you only have integers.</p>
<p>And in a mixed situation, AT_DISPATCH_ALL_TYPES should do the work.</p>

username: Max_Unhold
cooked: <aside class="quote no-group" data-username="ignaciogavier" data-post="3" data-topic="70470">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ignaciogavier/48/52479_2.png" class="avatar"> ignaciogavier:</div>
<blockquote>
<p>AT_DISPATCH_ALL_TYPES</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ignaciogavier">@ignaciogavier</a> : That is a good idea and thank you for providing the file link. Unfortunately, it is still not working with <code>AT_DISPATCH_ALL_TYPES</code> since you have to give a type as a first argument which apparently sets the type. Did you find a solution for all types in the mean time?</p>