akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Does nn.PixelShuffle support 3D data?

username: tianmaliuxing
cooked: <p>Does nn.PixelShuffle support 3D data? In other words, the input is 5D.</p>

username: ptrblck
cooked: <p>Based on the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.PixelShuffle">docs</a> a 4-dimensional input tensor is expected and you should also get a <code>RuntimeError</code>, if your input tensor has a different number of dimensions.</p>

username: Natan_Semyonov
cooked: <p>is there any solution to 3d data in nn.PixelShuffle or will be in the future?</p>

username: ptrblck
cooked: <p><a href="https://github.com/pytorch/pytorch/pull/49187">This PR</a> might have already added the support.<br>
You could use the latest nightly binary or build from source and check, if it’s working. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Natan_Semyonov
cooked: <p>Thanks.<br>
The latest nightly binary solved the problem of nn.PixelShuffle for input with more than 4 dimensions.</p>

username: dengluokk8
cooked: <p>Is that supported for 3D data now?</p>

username: ptrblck
cooked: <p>The MR was merged in Dec 2020 so I think it should be enabled. Did you try it and if so did you seeany issues?</p>

username: Furina58
cooked: <p>Maybe you can refer to this.<br>
<a href="https://github.com/kuoweilai/pixelshuffle3d" rel="noopener nofollow ugc">pixelshuffle3d</a></p>
How can I get more knowledge about optimized pytorch code?

username: Mahdi_Amrollahi
cooked: <p>I was reading some tutorial about <code>torch.compile</code> to understand what is happening under the hood and how it converts the pytorch codes to optimized kernels.<br>
Can someone tell me, what are the topics should I know more about?<br>
For example, <a href="https://pytorch.org/docs/stable/torch.compiler_get_started.html" rel="noopener nofollow ugc">here</a> it says that we can optimize the code and substitute the code with 2 reads and 2 writes with 1 read and one write.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
def fn(x, y):
    a = torch.cos(x).cuda()
    b = torch.sin(y).cuda()
    return a + b
new_fn = torch.compile(fn, backend="inductor")
input_tensor = torch.randn(10000).to(device="cuda:0")
a = new_fn(input_tensor, input_tensor)
</code></pre>
<p>So, can somebody tell me how we can categorize the optimization ways? How can I know more about this topic? Do we have a equivalent optimized kernel for each pytorch command like <code>nn.Linear()</code>? Or there may be somewhere to apply custom optimization?</p>
Batched tensor creation inside vmap

username: harsanyidani
cooked: <p>I want to create a tensor with <code>torch.zeros</code> based on the shape of an input to the function. Then I want to vectorize the function with <code>torch.vmap</code>.</p>
<p>Something like this:</p>
<pre><code class="lang-auto">poly_batched = torch.tensor([[1, 2, 3, 4], [1, 2, 3, 4]])

def polycompanion(polynomial):
    deg = polynomial.shape[-1] - 2
    companion = torch.zeros((deg+1, deg+1))
    companion[1:,:-1] = torch.eye(deg)
    companion[:,-1] = -1. * polynomial[:-1] / polynomial[-1]
    return companion

polycompanion_vmap = torch.vmap(polycompanion)
print(polycompanion_vmap(poly_batched))
</code></pre>
<p>The problem is that the batched version will not work, because <code>companion</code> won’t be a <code>BatchedTensor</code>, unlike <code>polynomial</code>, which was the input.</p>
<p>There is a workaround:</p>
<pre><code class="lang-auto">poly_batched = torch.tensor([[1, 2, 3, 4], [1, 2, 3, 4]])

def polycompanion(polynomial,companion):
    deg = companion.shape[-1] - 1
    companion[1:,:-1] = torch.eye(deg)
    companion[:,-1] = -1. * polynomial[:-1] / polynomial[-1]
    return companion

polycompanion_vmap = torch.vmap(polycompanion)

print(polycompanion_vmap(poly_batched, torch.zeros(poly_batched.shape[0],poly_batched.shape[-1]-1, poly_batched.shape[-1]-1)))
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([[[ 0.0000,  0.0000, -0.2500],
         [ 1.0000,  0.0000, -0.5000],
         [ 0.0000,  1.0000, -0.7500]],

        [[ 0.0000,  0.0000, -0.2500],
         [ 1.0000,  0.0000, -0.5000],
         [ 0.0000,  1.0000, -0.7500]]])
</code></pre>
<p>But this is ugly.</p>
<p>Is there a solution for this? Will this be supported in the future?</p>
<p>Note: If you use <code>torch.zeros_like</code> on an input to the function it works and creates <code>BatchedTensor</code> but this doesn’t help me here.</p>
<p>Thanks in advance for the help!</p>
Any resource that has grouped Core ATen Ops into categories?

username: kgbounce
cooked: <p>For example, <a href="https://tvm.apache.org/docs/reference/langref/relay_op.html" class="inline-onebox" rel="noopener nofollow ugc">Relay Core Tensor Operators — tvm 0.15.dev0 documentation</a> groups the operators by levels and functionalities. Thanks.</p>
<p>Similarly, I’m looking for some document that has grouped the core aten ops into categories, such as non-linear functions, memory operators, etc.</p>

username: Martin_Yuan
cooked: <p>Currently we don’t have the grouping documentation yet. We’ll look into the categories in the link you mentioned and see if it’s applicable for core ATen ops.</p>
Training Variational Autoencoder (VAE) from multiple input tensors

username: Rahul_Gulia
cooked: <p>Hi team,</p>
<p>I am new to the VAE implementation. And I want to predict SINR (RBG) image from multiple tensors such as Euclidean distance image, 3D distance image, Permittivity image, and Conductivity image (all in RBG).</p>
<p>Please let me know if there is any sample work done in such topic, or a headstart that anyone can give me. I could not find any relevant work online that could help me with the architecture.</p>
<p>Looking forward to any kind of suggestion.</p>
<p>Thank You,<br>
Rahul</p>
RuntimeError: DataLoader worker (pid 963720) is killed by signal: Segmentation fault

username: Patrick_G
cooked: <p>Hey guys! When using my cutom dataset in dataloader, i get the following error:</p>
<p>RuntimeError: DataLoader worker (pid 963720) is killed by signal: Segmentation fault.</p>
<p>It was already discussed a few times and many people could solve it by increasing the shared memory. I’m absolutely not familiar with shared memory, but if i do “sudo sysctl -a | grep shm” in terminal, i get:<br>
“kernel.shm_next_id = -1<br>
kernel.shm_rmid_forced = 0<br>
kernel.shmall = 18446744073692774399<br>
kernel.shmmax = 18446744073692774399<br>
kernel.shmmni = 4096<br>
vm.hugetlb_shm_group = 0”<br>
which i would say means i already have a maximum value for share memory (i never cahnged it, is this default for linux?)…</p>
<p>Generally the shared memory problem would make sense in my case since the error only occurs if i use a specific transform. This transform is called vahadane normalization and uses a reference image to make all images have the same color distribution. The reference image is part of the normalizer object and could be to big for the shared memory. But how does this make sense if my shared memory is already that big? I’m confused, every help will be much appreciated &lt;3</p>
Unchanged behaviour of using a pretrained Model for QAT

username: fbernhar
cooked: <p>I´m experiencing the same behaviour using QAT from scratch or loading a pretrained model.</p>
<hr>
<p>qat_model = MLP(Ni=1, Nh=256, No=1,)</p>
<p>The MLP consist of:</p>
<p>QuantStub()<br>
Linear()<br>
DeQuantstub()<br>
A non implemented activation function i try.</p>
<h2><a name="repeated-5-times-1" class="anchor" href="#repeated-5-times-1"></a>– repeated 5 times.</h2>
<h1><a name="load-model-2" class="anchor" href="#load-model-2"></a>Load Model</h1>
<p>qat_model.load_state_dict(torch.load(‘dict/model_state_dict.pt’, map_location=‘cpu’))<br>
qat_model.train()</p>
<p>activation_bitwidth = 8 <span class="hashtag-raw">#whatever</span> bit you want<br>
bitwidth = 8 <span class="hashtag-raw">#whatever</span> bit you want</p>
<p>fq_activation = torch.quantization.FakeQuantize.with_args(observer=torch.quantization.MinMaxObserver.with_args(<br>
quant_min=0,<br>
quant_max=2**activation_bitwidth-1,<br>
dtype=torch.quint8,<br>
qscheme=torch.per_tensor_affine,<br>
reduce_range=False,))</p>
<p>fq_weights = torch.quantization.FakeQuantize.with_args(<br>
observer = torch.quantization.MinMaxObserver.with_args(<br>
quant_min=-(2 ** bitwidth) // 2,<br>
quant_max=(2 ** bitwidth) // 2 - 1,<br>
dtype=torch.qint8,<br>
qscheme=torch.per_tensor_symmetric,<br>
reduce_range=False,))</p>
<p>intB_qat_qconfig = torch.quantization.QConfig(activation= fq_activation,weight = fq_weights)<br>
qat_model.qconfig = intB_qat_qconfig<br>
torch.ao.quantization.prepare_qat(qat_model, inplace=True)</p>
<p>The loss function is similar with and without the quat_model.load_state_dict, it feels like the weighs are overwritten.</p>

username: jerryzh168
cooked: <p>how did you get the initial state_dict? <code>‘dict/model_state_dict.pt’</code>?</p>
<p>and for the case that you don’t load from state_dict, what is the weight for original fp32 model before QAT prepare? is it random weights?</p>
Does quantization in eager mode require inserting multiple different FFs?

username: seamoonlight-YBY
cooked: <p>Does quantization in eager mode require inserting multiple different FFs (torch.nn.quantized.FloatFunctional) at different computation positions, or is a single FF sufficient to perform different computational functions? For example, in the code below, do I need to define both ff1 and ff2 as separate FFs?</p>
<pre><code class="lang-auto">class Block35(nn.Module):

    def __init__(self, scale=1.0):
        super().__init__()

        self.scale = scale

        self.branch0 = BasicConv2d(256, 32, kernel_size=1, stride=1)

        self.branch1 = nn.Sequential(
            BasicConv2d(256, 32, kernel_size=1, stride=1),
            BasicConv2d(32, 32, kernel_size=3, stride=1, padding=1)
        )

        self.branch2 = nn.Sequential(
            BasicConv2d(256, 32, kernel_size=1, stride=1),
            BasicConv2d(32, 32, kernel_size=3, stride=1, padding=1),
            BasicConv2d(32, 32, kernel_size=3, stride=1, padding=1)
        )

        self.conv2d = nn.Conv2d(96, 256, kernel_size=1, stride=1)
        self.relu = nn.ReLU(inplace=False)
        self.ff1 = nn.quantized.FloatFunctional()
        self.ff2 = nn.quantized.FloatFunctional()

    def forward(self, x):
        x0 = self.branch0(x)
        x1 = self.branch1(x)
        x2 = self.branch2(x)
        out = torch.cat((x0, x1, x2), 1)
        out = self.conv2d(out)
        # out = out * self.scale + x
        out = self.ff2.add(self.ff1.mul(out, self.scale), x)
        out = self.relu(out)
        return out
</code></pre>

username: jerryzh168
cooked: <p>yes, you’ll need multiple different FFs, since each one records the state of Tensor at a single point in the program</p>
In flatten the output scale is different from the input scale

username: matthew331
cooked: <p>I use pytorch 2.1 with QAT training in FX mode and save model in the ONNX format. When I check model by the Netron, I find that in flatten the output scale is different from the input scale, which I don’t think is normal, because the flatten should only change the dimensions and not the values, what went wrong here?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a58281662bdd16579a2004a6966da43afa074af2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a58281662bdd16579a2004a6966da43afa074af2" title="擷取"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a58281662bdd16579a2004a6966da43afa074af2_2_398x500.jpeg" alt="擷取" data-base62-sha1="nCao8rDdHxMbXCmPkZ5d3Da6TGq" width="398" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a58281662bdd16579a2004a6966da43afa074af2_2_398x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a58281662bdd16579a2004a6966da43afa074af2_2_597x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a58281662bdd16579a2004a6966da43afa074af2.jpeg 2x" data-dominant-color="403F3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">擷取</span><span class="informations">708×889 48.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: jerryzh168
cooked: <p>could you print the model after prepare_qat_fx and training? and check the input and output fake quant for the flatten op?</p>
Installation of Ultralytics and Yolov5 over Jetson Nano board (Jetpack 4.6.4).)

username: Aakash_J
cooked: <p>Device name - Jetson Nano<br>
OS - 4.6.4<br>
Python - 3.9</p>
<p>Objective - Install Ultralytics and run Yolo scripts - <a href="https://docs.ultralytics.com/yolov5/tutorials/running_on_jetson_nano/#install-pytorch-and-torchvision" class="inline-onebox" rel="noopener nofollow ugc">NVIDIA Jetson Nano Deployment - Ultralytics YOLOv8 Docs</a></p>
<p>Question - I have installed below dependencies as per the above mentioned article for ultralytics, now only pytorch and torchvision is remaining, that i have to install as per my hardware compatibility mentioned here - <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048" rel="noopener nofollow ugc">https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048</a>, i haven’t find anything that can relate case. As per this reference link we only have PyTorch v1.10.0 for jetpack 4.6.4 and will only be supported for python 3.6, please help me to setup this.</p>
<p><strong>pip list</strong> - pandas=1.1.4, pyyaml=5.3.1, pillow=10.1.0, pip=23.3.1, tqdm=4.64.0, matplotlib=3.3.0, numpy=1.26.2, psutil=5.9.6, opencv-python=4.6.0.66, requests=2.23.0, scipy=1.11.4, thop=0.1.1.post2207130030, seaborn=0.11.0.</p>
When exporting an arbitrary model, how can one know the model has been fully decomposed into Core ATen ops?

username: kgbounce
cooked: <pre><code class="lang-auto"># for example, model = transformers.MixtralForCausalLM
  curr_model = model(config=model.config_class())
  curr_model.eval()
  if show_intermediate_result:
      print(curr_model)
  
  # generate prompt tokens
  tokenizer = AutoTokenizer.from_pretrained(model_config_dict[model][1])
  inputs = tokenizer(input_str, return_tensors="pt")
  
  # configure the export setup.
  example_args = ()
  example_kwargs = {
      'input_ids': inputs['input_ids'],
      'attention_mask': inputs['attention_mask']}
  
  # export the model
  exported_program: torch.export.ExportedProgram = export(
      curr_model, args=example_args, kwargs=example_kwargs)
</code></pre>
<p>I can see the core ATen ops in <code>exported_program</code>, but am not sure if the <code>model</code> has been <strong>fully decomposed</strong>. What should I be looking for? Thanks.</p>

username: Martin_Yuan
cooked: <p>It should be fully decomposed when you call to_edge().</p>

username: kgbounce
cooked: <p>Thank you <a class="mention" href="/u/martin_yuan">@Martin_Yuan</a>. I’ll give it a try. So if a given model cannot be fully decomposed when someone calls <code>to_edge()</code> on it, will some errors or decomposition coverage report be provided to the users? It’s about the general feedback mechanism of <code>export()</code> and <code>to_edge()</code> on its decomposition coverage. Thanks!</p>
Converting code for training on multiple gpu to single gpu based training

username: sparshgarg23
cooked: <p>I recently came across a code that was  trained on multiple gpus. However, since i don’t have access to multiple gpu(gcp or aws),my only option is to train on colab.<br>
as such, is it possible to convert a code that was written for training on multiple gpus to single gpu based code.</p>
<p>I know that the standard process for training on multiple gpu people usually change the sampler from random or sequential to distributed data sampler.<br>
But after looking at the code for distributed training, seems there are other nuances involved in it.<br>
For reference here is the link to the repo for distributed training as well as the code section which i believe might need to be changed.</p>
<p><strong>Repository that uses multiple gpu or dsitributed training</strong></p>
<pre><code class="lang-auto">https://github.com/DerrickWang005/CRIS.pytorch
</code></pre>
<p><strong>Code block which i think might need to be changed,so as to ensure i can train on single gpu</strong></p>
<pre><code class="lang-auto">def main_worker(gpu, args):
    args.output_dir = os.path.join(args.output_folder, args.exp_name)

    # local rank &amp; global rank
    args.gpu = gpu
    args.rank = args.rank * args.ngpus_per_node + gpu
    torch.cuda.set_device(args.gpu)

    # logger
    setup_logger(args.output_dir,
                 distributed_rank=args.gpu,
                 filename="train.log",
                 mode="a")

    # dist init
    dist.init_process_group(backend=args.dist_backend,
                            init_method=args.dist_url,
                            world_size=args.world_size,
                            rank=args.rank)

    # wandb
    if args.rank == 0:
        wandb.init(job_type="training",
                   mode="online",
                   config=args,
                   project="CRIS",
                   name=args.exp_name,
                   tags=[args.dataset, args.clip_pretrain])
    dist.barrier()

    # build model
    model, param_list = build_segmenter(args)
    if args.sync_bn:
        model = nn.SyncBatchNorm.convert_sync_batchnorm(model)
    logger.info(model)
    model = nn.parallel.DistributedDataParallel(model.cuda(),
                                                device_ids=[args.gpu],
                                                find_unused_parameters=True)
</code></pre>
Modifying intermediate values during forward

username: Crizal
cooked: <p>If I have a series layers in my implementation of forward, is it possible for me to inspect an intermediate value between those layers, modify it, and send it through the rest of the network? How would I go about doing something like this?</p>

username: ptrblck
cooked: <p>You could use forward hooks as seen in this code snippet:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        self.fc2 = nn.Linear(1, 1)
        
    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        return x

def hook(m, input, output):
    output = output * 100.
    return output

model = MyModel()

x = torch.randn(1, 1)
out = model(x)
print(out)
out.mean().backward()
print(model.fc1.weight.grad)
model.zero_grad()

model.fc1.register_forward_hook(hook)
out = model(x)
print(out)
out.mean().backward()
print(model.fc1.weight.grad)
</code></pre>

username: Pervaiz_Khan
cooked: <p>Hi,</p>
<p>The documentation of forward_hook says " The hook will be called every time after <code>forward()</code> has computed an output". Just wanted to clarify that if I modify the value of a certain layer of a model, that modification will the part of forward call, and hence the output of the forward call would be different from without modified forward call. Right ?</p>
Weird bug: how does unsqueeze and flatten affect gradients?

username: spacegoing
cooked: <p><strong>Problem Description:</strong><br>
I have a tensor named <code>returns</code> with shape <code>[bsz]</code> and another tensor <code>v_new</code> with shape <code>[bsz, 1]</code>. I am calculating the value function loss (<code>vf_loss</code>) in two different ways, and I’m observing different convergence behaviors for each method.</p>
<p>Here’s the relevant part of my code:</p>
<pre data-code-wrap="python"><code class="lang-python"># Tensor shapes: returns [bsz], v_new [bsz,1]

# Method 1
aa = (returns.unsqueeze(-1) - v_new)  # After unsqueeze, aa has shape [bsz, 1]
vf_loss_aa = aa.pow(2).mean()

# Method 2
bb = (returns - v_new.flatten())  # v_new flattened to [bsz], bb has shape [bsz]
vf_loss_bb = bb.pow(2).mean()
</code></pre>
<p><strong>Observations:</strong></p>
<ul>
<li>When using <code>vf_loss_aa</code> (computed from <code>aa</code>), the model does not converge.</li>
<li>However, with <code>vf_loss_bb</code> (computed from <code>bb</code>), the model converges normally.</li>
</ul>
<p><strong>Points to Consider:</strong></p>
<ul>
<li>Logically, <code>aa</code> and <code>bb</code> should represent the same values, albeit with different shapes. <code>aa</code> is 2D ([bsz, 1]), and <code>bb</code> is 1D ([bsz]).</li>
<li>The <code>.pow(2).mean()</code> operation should yield the same result for both <code>aa</code> and <code>bb</code> if the values are the same.</li>
<li>I’ve checked that <code>aa.flatten()</code> and <code>bb</code> are equivalent.</li>
</ul>
<p><strong>Questions:</strong></p>
<ol>
<li>Why would there be a difference in convergence behavior between these two methods, given that the operations and the resulting values should theoretically be the same?</li>
<li>Could this discrepancy be related to how PyTorch handles gradients for tensors of different shapes?</li>
<li>Are there any known issues or subtleties in PyTorch related to this kind of situation?</li>
</ol>
<p>Any insights, explanations, or suggestions for further debugging would be greatly appreciated. I’m curious to understand the underlying reason for this difference in behavior.</p>
<p>Thank you in advance for your help!</p>

username: spacegoing
cooked: <p>This is my bad, I accidentally changed the shape of a var which assumed to be a different shape in subsequent functions and lead to this bug.</p>
<p>For anyone hits similar issues: this must be a bug in your code. pls look closely: There shouldn’t be any difference using pytorch’s reshape view flatten squeeze etc funcs w.r.t. gradients.</p>
No gradient in simple function

username: BluePlant
cooked: <p>The following function does not behave as expected when called from my codebase:</p>
<pre data-code-wrap="python"><code class="lang-python">def foo():
    torch.autograd.set_grad_enabled(True)
    torch.set_grad_enabled(True)
    with torch.enable_grad():
        y = torch.randn(10, device='cuda')
        y = torch.nn.Parameter(y.detach(), requires_grad=True)
        print(y.requires_grad, y.mean())
</code></pre>
<p>As you might notice, I tried my best to enable gradients in this code. If I call this function by itself, the printed output is as expected, i.e. <code>True tensor(-0.8699, device='cuda:0', grad_fn=&lt;MeanBackward0&gt;)</code>.</p>
<p>However, if I run this during an evaluation pass of a lightning-model, the output is <code>True tensor(-0.0105, device='cuda:0')</code>, so the <code>grad_fn</code> is missing, even though <code>requires_grad</code> is <code>True</code>. As a result I cannot to a backward pass if I process <code>y</code> further.</p>
<p>What else could prevent gradients from being computed?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="BluePlant" data-post="1" data-topic="193792">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/51bf81/48.png" class="avatar"> BluePlant:</div>
<blockquote>
<p>However, if I run this during an evaluation pass of a lightning-model</p>
</blockquote>
</aside>
<p>Do you see the same behavior without using lightning? PyTorch itself won’t disable gradient calculation behind your back, so you might want to check what the evaluation loop in lightning does.</p>

username: BluePlant
cooked: <p>Thanks for the hint towards lightning. I found <a href="https://github.com/Lightning-AI/pytorch-lightning/issues/13948#issuecomment-1221148065" rel="noopener nofollow ugc">this issue</a> on their github, apparently lightning sets <code>torch.inference_mode(True)</code> when evaluating. This now works as expected:</p>
<pre data-code-wrap="python"><code class="lang-python">def foo():
    with torch.inference_mode(False):
        y = torch.randn(10, device='cuda')
        y = torch.nn.Parameter(y.detach(), requires_grad=True)
        print(y.requires_grad, y.mean())
</code></pre>
Torch is not using gpu full performance

username: Venkatesh_Balakrishn
cooked: <p>Hi when i try to run two CNN algorithms with separate torch weights the execution is slow. Also i checked the GPU utilization it is not fully utilized it is lying in 30% only .  I need to use full GPU potential when parallely running two algorithms.  kindly help me to overcome this issue.</p>
<p>Algorithms links<br>
1.<a href="https://github.com/ultralytics/ultralytics.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ultralytics/ultralytics: NEW - YOLOv8 🚀 in PyTorch &gt; ONNX &gt; OpenVINO &gt; CoreML &gt; TFLite</a><br>
2.<a href="https://github.com/bigmb/Unet-Segmentation-Pytorch-Nest-of-Unets.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - bigmb/Unet-Segmentation-Pytorch-Nest-of-Unets: Implementation of different kinds of Unet Models for Image Segmentation - Unet , RCNN-Unet, Attention Unet, RCNN-Attention Unet, Nested Unet</a></p>

username: ptrblck
cooked: <p>Profile your code and check where the bottleneck is using the native PyTorch profiler or e.g. Nsight Systems.</p>
Dropout across multiple GPU

username: vanshil_shah
cooked: <p>If I seed each process differently, then dropout would behave differently for each process. So conceptually the tensor that is dropped in one process is picked in another then when we are doing all reduce on gradients there would be no zeroed-out gradient which is different from single GPU training if we use dropout. So do we have to necessarily set the same manual seed if we use a dropout layer across all the processes to make it conceptually work?</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/ddp-and-dropout-behave-the-same-across-gpus/125052/7">here</a>.</p>
Workaround to change learning rate for specific parameters in LBFGS amid lack of support for per-parameter option

username: Adel_Redjimi
cooked: <p>Hello,</p>
<p>I’m working on a shape deformation library (IMODAL on github). At some point, we optimize a differentiable function. LBFGS works best compared to SGD-variants (up to our knowledge).</p>
<p>However, we wish to prioritize some variables over some others. First idea was to change the learning rate for the desired priority variables.</p>
<p>At the time of writing this question, LBFGS from <code>torch.optim</code> does not support per-parameter options.</p>
<p>Is there a workaround to achieve this? What are other optimization options I can reach out for given that LBFGS worked better than SGD and Adam. I’d appreciate any links to resources providing insight on LBFGS vs. SGD-variants and whether we can ‘‘emulate’’ LBFGS with a first-order method.</p>
<p>Thanks in advance,</p>
Can PyTorch transforms be added to a Torchscript model?

username: ocharles
cooked: <p>I have a basic image classifier, and I want to package this up as a Torchscript model. I plan to deploy this in a Haskell application, using the C++ API as a foreign function interface. This works fine, but I would also like to add the necessary transforms to this model. My model is a fine-tuned Resnet model, and when performing training I use the provided transform to resize/center crop etc. It seems unfortunate to have to duplicate this into my inference application manually, and I was hoping I could just add it to my Torchscript model.</p>
<p>My naive attempt is:</p>
<pre><code class="lang-auto">class E2EClumpDetector(pl.LightningModule):
    def __init__(self, clump_detector):
        super().__init__()
        self.clump_detector = clump_detector

    def forward(self, batch):
        return torch.nn.functional.sigmoid(self.clump_detector(self.clump_detector.transform(batch)))
</code></pre>
<p>But this gives me</p>
<pre><code class="lang-auto">RuntimeError: 
Module 'ClumpDetection' has no attribute 'transform' (This attribute exists on the Python module, but we failed to convert Python type: 'torchvision.transforms.transforms.Compose' to a TorchScript type. Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type Compose.. Its type was inferred; try adding a type annotation for the attribute.):
  File "/home/ollie/work/clump-detection/clump_detection.py", line 76
    def forward(self, batch):
        return torch.nn.functional.sigmoid(self.clump_detector(self.clump_detector.transform(batch)))
</code></pre>
<p>So I don’t think this works. Does anyone have any advice that doesn’t require me to duplicate the series of transforms?</p>

username: ocharles
cooked: <p>Ah, I’ve just seen <a href="https://pytorch.org/vision/stable/transforms.html#torchscript-support" class="inline-onebox" rel="noopener nofollow ugc">Transforming and augmenting images — Torchvision 0.16 documentation</a>! Maybe it’s really as simple as swapping out <code>Compose</code> for <code>Sequential</code>.</p>
<p>Edit: yes, it was that simple! Just should have RTFM first <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
ONNX export with optional arguments

username: Peter_Featherstone
cooked: <p>Is there a way to do an ONNX export of a model where one of its arguments is optional?<br>
For example:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Linear(256, 256)

    def forward(self, x, bias: Optional[torch.Tensor] = None):
        x = self.net(x)
        if bias is not None:
            x = x + bias
        return x
</code></pre>
<p>Then I would like the following to work:</p>
<pre><code class="lang-auto">net = Model().eval()
x = torch.randn(4, 256)
b = torch.randn(4, 256)

torch.onnx.export(net, (x,None), '/tmp/model.onnx', opset_version=19, 
                  input_names=['x', 'bias'],
                  output_names=['y'],
                  dynamic_axes={'x'     : {0: 'B'},
                                'y'     : {0: 'B'},
                                'bias'  : {0: 'B'}})

ort = onnxruntime.InferenceSession('/tmp/model.onnx', providers=["CPUExecutionProvider"])

x = torch.randn(2, 256)
b = torch.randn(2, 256)

y1  = net(x, b)
y2, = ort.run(None, {'x': x.numpy(), 'bias': b.numpy()})
np.testing.assert_allclose(y1.numpy(), y2)
</code></pre>
<p>thank you</p>

username: Peter_Featherstone
cooked: <p>I tried first scripting the model using:</p>
<pre><code class="lang-auto">net = torch.jit.script(net)
</code></pre>
<p>But that didn’t help.</p>
Ignore Boxes for Object Detection

username: SeucheAchat9115
cooked: <p>Hi,<br>
I would like to ask if there is a functionality to add Ignore boxes in object detection like it is typically performed for some datasets.<br>
Example: Classes Person and Bicycle. If there is a Bicycle object, I would not like to detect the rider as a person, because the Bicycle is defined to be joint rider + bicycle. Therefore I would like to have ignore boxes during training, where we can ignore losses of such predictions / ground-truth in Fasterrcnn_resnet50 for example.<br>
Does anyone know how can this functionality be added in torchvision object detection?</p>
How to avoid sampling invalid negative edges in PyTorch-Geometric

username: ahn1340
cooked: <p>I’m fairly new to PyTorch Geometric, and I’m trying to build my first graph variational autoencoder. I am minibatching the dataset, which concatenates the edge indices along the second dimension, such that its shape looks something like this: [2, n_edges1 + n_edges2 + …].</p>
<p>If I use torch_geometric.utils.negative_sampling, it seems to sample negative edges between nodes that belong to different graphs. But since each graph has its own set of positive and negative edges to train the autoencoder, sampling those invalid edges between nodes of different graphs will produce wrong gradient signals in training. How can I restrict the function’s behavior to avoid this?</p>

username: salvdr
cooked: <p>Agreed. Do you manually recompute the metrics while dropping those negative edges?</p>
RNN LSTM that uses different timeseries as input and output

username: ckask
cooked: <p>I’m currently working on simulating an optimization algorithm that involves using an hourly timeseries (input_df) to calculate 36 different hourly timeseries (output_df). My approach involves creating an RNN LSTM with input=1 and output=36. However, for simplicity, I’ve initially set the output to 1.</p>
<p>I’ve observed significant improvements in results during model training when incorporating the previous 24 observations of the input_df and the previous one observation of the output_df to predict the next input_df value (Y(t) = f(Y(t-1), X(t),…,X(t-24)).</p>
<p>My challenge arises when I attempt to make predictions for other timeseries. Specifically, I’m unsure of how to configure the RNN to utilize the previous observation as input. Additionally, I’m uncertain about how to initiate the entire process, especially considering that for X(1), I don’t have Y(1).</p>
<p>I would greatly appreciate any insights, advice, or suggestions on how to address these issues and optimize the RNN for predictions across various timeseries.</p>
Reading a word problem stored in a image?

username: ThePie
cooked: <p>I wanted to see what models would be best suited for taking in a image and having the model perform a image to text (OCR of some kind) and then outputting the contents of the image in a “question” “answers” format.</p>
<p>The set of images have a question in the center and then a single column table that represent one answer choice. I would imagine a model would have to be retrained to know that the text in the center of the image is the question, and each column representants a answer choice. In this case there are two answer choices.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc2db9184e67c3fcf218e4584ff8a10314d8c913.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc2db9184e67c3fcf218e4584ff8a10314d8c913" title="Demo_Q"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc2db9184e67c3fcf218e4584ff8a10314d8c913_2_345x231.png" alt="Demo_Q" data-base62-sha1="t8fmDsQNUVui8Io01s0qYJsY0Ar" width="345" height="231" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc2db9184e67c3fcf218e4584ff8a10314d8c913_2_345x231.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc2db9184e67c3fcf218e4584ff8a10314d8c913_2_517x346.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc2db9184e67c3fcf218e4584ff8a10314d8c913_2_690x462.png 2x" data-dominant-color="E5E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Demo_Q</span><span class="informations">1437×966 9.28 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The model does not need to answer the question, it just needs to output the text.</p>
<p>Any models that would make a good start here for research’s (has this already been done?) or if no models are good for the task then what is the best approach for training a model to complete these tasks.</p>
<p>The result of the model would output the JSON like below or something similar.</p>
<pre><code class="lang-auto">{

  {

    Question: What is the sound that the cereal rice crispes make? Snap, (i)___, (ii) ___.

  },

  {

    Answer 1: Crackle, Pop, Crunch

    Answer 2: Burp, Pop, Crackle

  }

}
</code></pre>
Rewards decreasing in DQN (multi-actions)

username: FinCdy
cooked: <p>Hi,<br>
I’m trying to use a DQN to do reinforcement learning, but I think I’m doing wrong in the optimization part of the model… I’ve tried modifying the state and the function to give rewards, but the problem remains and it’s the following.<br>
The rewards always drop more or less linearly (and noisy) during the episodes, so I think I’ve done an error in this code snippet:</p>
<pre><code class="lang-auto">    def optimize_model(self, state, actions, rewards, next_state, gamma):
        # Convert lists to tensors
        actions = torch.tensor(actions, dtype=torch.long)
        rewards = torch.tensor(rewards, dtype=torch.float)
        next_state = next_state.float()
        state = state.float()

        # Get current Q values -&gt; get only the actions done 
        # (the output has 3 possible actions for each person: take only the one with max value for each person)
        state_action_values = self.policy_net(state)[actions]

        # Get next Q values
        with torch.no_grad():
            next_state_values = self.target_net(next_state)
        next_state_values = next_state_values.view(-1, 3)  # Reshape to (num_people, 3)
        next_actions = next_state_values.max(1)[1]  # Get the action done for each person
        next_state_values = next_state_values.gather(1, next_actions.unsqueeze(-1)).squeeze(-1).detach()

        # Compute the expected Q values
        expected_state_action_values = (next_state_values * gamma) + rewards
        # Compute loss
        loss = nn.functional.smooth_l1_loss(state_action_values, expected_state_action_values)

        # Optimize the model
        self.optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(self.policy_net.parameters(), max_norm=1)
        self.optimizer.step()

        
</code></pre>
<p>As added in the comments, the policy_net should take as input a state and return as output actions (3 for each person, whose number is given in a variable).<br>
Are there conceptual mistakes in there?<br>
Thank you very much</p>
Checkpoint didn’t support list output

username: hhxx
cooked: <p>I attempted to utilize <strong>torch.utils.checkpoint</strong> in my code, but I discovered that it does not support <strong>List</strong> inputs.</p>
<p>Below is the standard usage of <strong>torch.utils.checkpoint</strong>:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.utils.checkpoint as cp

class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.fc1 = nn.Linear(784, 1024)
        self.fc2 = nn.Linear(1024, 1024)
        self.fc3 = nn.Linear(1024, 512)
        self.fc4 = nn.Linear(512, 10)

    def forward(self, x):
        # Apply the first 2 layers
        x = nn.functional.relu(self.fc1(x))
        x = nn.functional.relu(self.fc2(x))
        # Apply the checkpointed layers
        x = cp.checkpoint(self._checkpointed_forward, x)

        # Apply the last 1 layer
        x = self.fc4(x)
        return x

    def _checkpointed_forward(self, x):
        x = nn.functional.relu(self.fc3(x))
        return x

# Create a random input tensor
x = torch.randn(2, 784)

# Create the neural network
model = MyNet()

# Compute the output
output = model(x)

# Compute the gradients
output.sum().backward()
# import pdb; pdb.set_trace()

# Print the gradients of the first layer
print(model.fc1.weight.grad)
</code></pre>
<p>If I modify the above code as follows (the main change is using a List in <strong>_checkpointed_forward</strong>):</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def forward(self, x):
        ...
        x = cp.checkpoint(self._checkpointed_forward, [x])
        x = self.fc4(x)
        return x

    def _checkpointed_forward(self, x):
        x = nn.functional.relu(self.fc3(x[0]))
        return x
...
output.sum().backward()
print(model.fc1.weight.grad)
</code></pre>
<p>I observe that <strong>model.fc1.weight.grad = None</strong> in this case.<br>
Additionally, I set <strong>pdb.set_trace()</strong> in the <strong>class CheckpointFunction</strong>:</p>
<pre><code class="lang-auto">class CheckpointFunction(torch.autograd.Function):
    def backward(ctx, *args):
        import pdb; pdb.set_trace()
</code></pre>
<p>I notice that CheckpointFunction.backward is not even executed.</p>
<p>I am curious about why the code behaves in this manner.</p>

username: ptrblck
cooked: <p><code>torch.utils.checkpoint</code> is used for activation checkpointing trading compute for memory. I’m unsure if you want to save a <code>list</code> object but if so use <code>torch.save</code>.</p>

username: hhxx
cooked: <p>Hi, I updated the code example, could you help to check it? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Model deployment process to FPGA

username: dalseeroh
cooked: <p>Hello folks,</p>
<p>I wrote a UNet-based model that inpaints the corrupted pixels in an input image. It is written in Python using PyTorch frameworks. It is relatively huge network, so the inference time is 200ms/image on CPU and 80ms/image on GPU. Now I want to deploy this model on Intel FPGA in the embedded products run by ARM core. The reason to do this is:</p>
<ol>
<li>To improve this inference time</li>
<li>To save computing power at the end user</li>
</ol>
<p>I am still investigating how to get this done. I want to avoid re-writing the model from the ground using HDL on FPGA. Has anyone done things like this before?</p>

username: marksaroufim
cooked: <p>I haven’t tried it but you could convert your model to ONNX before deploying to FPGA <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-fpga-web-service" class="inline-onebox" rel="noopener nofollow ugc">Deploy ML models to FPGAs - Azure Machine Learning | Microsoft Docs</a></p>
<p>But in general there’s easier ways of increasing inference time than using FPGA <a href="https://discuss.pytorch.org/t/inference-extremely-slow-on-aws-cpu/111310/2" class="inline-onebox">Inference extremely slow on AWS (CPU)? - #2 by marksaroufim</a></p>

username: Jon_runs
cooked: <p>I second this quite relevant question. Also, there is a limit to improving inference speeds on CPUs and GPUs, while the speed limit for FPGAs seems to be orders of magnitude lower.</p>
Looping Issue in PyInstaller Packaged FastAPI App with Torchvision Dependency

username: Haruethai_Han
cooked: <p>I’m working on a FastAPI application that uses Torchvision for image processing. The application is packaged using PyInstaller</p>
<p>When I run the compiled .exe, I receive a warning from Torchvision about failing to load the image Python extension, and then the application seems to enter an infinite loop</p>
<p>‘’’ torchvision\io\image.py:13: UserWarning: Failed to load image Python extension: '‘If you don’t plan on using image functionality from <code>torchvision.io</code>, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have <code>libjpeg</code> or <code>libpng</code> installed before building <code>torchvision</code> from source? ‘’’</p>
<p>How can I ensure that all necessary binaries for Torchvision are correctly included in the PyInstaller bundle? Are there any known issues with Torchvision and PyInstaller compatibility</p>
Significant Slowdown in Training Speed After Using compile() in PyTorch 2.1.0

username: smartbarbarian
cooked: <p>Hello everyone,</p>
<p>I’m currently experiencing a substantial decrease in training speed after using the <code>compile()</code> function in my PyTorch model. Before the inclusion of <code>compile()</code>, my training iterations were completed in approximately 1 second per iteration. However, after integrating <code>compile()</code>, the speed has drastically reduced to about 70 seconds per iteration even after the 1st iteration.</p>
<p>This slowdown seems to be predominantly occurring during the model’s forward pass and the backward pass of the loss function. I haven’t integrated any third-party libraries.</p>
<p>I’m looking for advice on how to profile and pinpoint the source of this slowdown. I’d greatly appreciate your input.</p>
<p><strong>Environment Details</strong>:</p>
<ul>
<li>GPU: NVIDIA RTX 3090</li>
<li>PyTorch Version: 2.1.0</li>
<li>CUDA Version: 11.8</li>
<li>cuDNN Version: 8</li>
<li>Docker Image: pytorch/pytorch:2.1.0-cuda11.8-cudnn8-devel</li>
</ul>
<p>Thank you in advance for any assistance or insights you can provide!</p>
AttributeError: 'ImageClassifierOutput' object has no attribute 'numpy'

username: Stavan_Rupareliya
cooked: <p>Hello,<br>
I am trying to model 'ImageClassifierOutput, output into numpy.<br>
i got this error.<br>
AttributeError: ‘ImageClassifierOutput’ object has no attribute ‘numpy’<br>
please for the same.</p>

username: ptrblck
cooked: <p><code>.numpy()</code> is a function defined for <code>tensor</code>s, so you might need to access the actual internal <code>tensor</code> in <code>ImageClassifierOutput</code> before converting it to a numpy array.</p>

username: Stavan_Rupareliya
cooked: <p>Thanks for your reply</p>
Create multiple torch.normal tensors

username: alexgagnon
cooked: <p>I’m trying to create a tensor of shape (n, m, o) from a normal distribution, but using a mean in the shape (m, o). It appears torch.normal only allows setting <code>size</code> when mean and std are numbers. For example in the simple case, I’m trying to create shape 5, 1, 1.</p>
<pre data-code-wrap="py"><code class="lang-py">mean = torch.tensor([[0.0]])
std = torch.tensor([[1.0]])

print(torch.normal(mean, std)) # works, only creates 1
print(torch.normal(mean, std, size=(5, 1, 1)) # doesn't work, no matching signature
</code></pre>
<p>What’s a good way to handle the above? (Should ideally be created right on the GPU)</p>

username: KFrank
cooked: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="alexgagnon" data-post="1" data-topic="193678" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexgagnon/48/65664_2.png" class="avatar"> alexgagnon:</div>
<blockquote>
<p>What’s a good way to handle the above? (Should ideally be created right on the GPU)</p>
</blockquote>
</aside>
<p>You can either <code>expand()</code> <code>mean</code> and <code>std</code> to add your “batch size” of <code>n = 5</code><br>
or use <code>torch.distributions.Normal</code> to whose <code>.sample()</code> method you<br>
can pass <code>n</code> (packaged as a tuple):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.1
&gt;&gt;&gt;
&gt;&gt;&gt; device = 'cuda'
&gt;&gt;&gt;
&gt;&gt;&gt; mean = torch.tensor ([[0.0]]).to (device)
&gt;&gt;&gt; std = torch.tensor ([[1.0]]).to (device)
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt; torch.normal (mean.unsqueeze (0).expand (5, 1, 1), std.unsqueeze (0).expand (5, 1, 1))
tensor([[[-0.3201]],

        [[-0.0794]],

        [[ 0.2601]],

        [[ 0.7900]],

        [[-0.3996]]], device='cuda:0')
&gt;&gt;&gt;
&gt;&gt;&gt; dist = torch.distributions.Normal (mean, std)
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt; dist.sample ((5,))
tensor([[[-0.3201]],

        [[-0.0794]],

        [[ 0.2601]],

        [[ 0.7900]],

        [[-0.3996]]], device='cuda:0')
</code></pre>
<p>Best.</p>
<p>K. Frank</p>