akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
.view function in JIT scripts

dshah3: <p>I am trying to do bitwise operations on a bfloat16 tensor of a random size. I want to do this in a JIT script. Normally, if the input tensor is a, we can use b.view(torch.int16) and then do bitwise masking. However, in a JIT script, this doesn’t work.</p>
<p>Below is the code:</p>
<pre><code class="lang-auto">@torch.jit.script
def get_bits(t: torch.Tensor) -&gt; torch.Tensor:
  t = torch.tensor([1.0, -1.0, 0.5, -0.5, 2.0], dtype=torch.bfloat16)
  b = t.view(torch.int16)
  print(b)
  return b
</code></pre>
<p>And here is the error:</p>
<pre><code class="lang-auto">tensor([ 1.0000, -1.0000,  0.5000, -0.5000,  2.0000], dtype=torch.bfloat16)
Traceback (most recent call last):
  File "/Users/lib/test.py", line 7, in &lt;module&gt;
    t = round_to_fp8_represented_as_int8(x, 2)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/Users/lib/bits.py", line 13, in round_to_fp8_represented_as_int8
) -&gt; torch.Tensor:

    b = t.view(torch.int16)
        ~~~~~~ &lt;--- HERE
    print(b)
RuntimeError: shape '[2]' is invalid for input of size 5
</code></pre>
<p>What’s the issue here? What is an alternative? This is an issue no matter what tensor size I use.</p>

BrandyN11: <aside class="quote no-group" data-username="dshah3" data-post="1" data-topic="203793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/c2a13f/48.png" class="avatar"> dshah3:</div>
<blockquote>
<p>What’s the issue here? What is an alternative? This is an issue no matter what tensor size I use.</p>
</blockquote>
</aside>
<p>The issue seems to be with the <code>.view()</code> method in JIT script. As an alternative, try using <code>.to(torch.int16)</code> for type casting, and then perform the bitwise operations. If that doesn’t work, consider using a different approach to manipulate the tensor’s bits without changing its view. Also, ensure that the size of the tensor matches the expected shape when converting.</p>
AMP during inference

yiftach: <p>Hi,<br>
Does AMP speed up inference, or is it just for training? I used to think the reduced precision was only for gradients (and therefore irrelevant for inference) but I’ve seen mentions of choosing other implementations for the forward too, which sounds like it might be designed for inference-only usecases too.</p>
<p>Secondly, just to make sure - AMP might cause a slight decrease in accuracy compared to fp32, but considerably less than moving both the model and the data to fp16?</p>

Jing_Ping_Wong: <p>Hi <a class="mention" href="/u/yiftach">@yiftach</a> , my guess is that AMP isn’t really needed for inference, as we don’t need to keep track of model activations as we pass through the layers for 1 forward pass (we can discard them as we go higher up the layers), whereas we would need to keep all of the activations in memory for the backward pass during training. As such the memory savings for inference using mixed precision (in the sense that AMP implements) isn’t very substantial compared to what it would be during training.</p>
<p>That said, during inference, my understanding is that the bottleneck is the memory requirements for the kv-cache (just a subset of the activations), for which I believe some engines have quantized options available for reducing overall memory usage, like the FP8 kv-cache in vllm:<br>
<a href="https://docs.vllm.ai/en/latest/quantization/fp8_e5m2_kvcache.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.vllm.ai/en/latest/quantization/fp8_e5m2_kvcache.html</a></p>
<p>I’m still quite new to all these, so please let me know if this doesn’t make sense!</p>
The conv weight grad from loss.backward has nans, but it only has infs when compute manually

3yuan: <p>After finding the nans in the conv weight grad, I tried to compute the grad manually. And I find only some infs in the manually computed grad, but not any nans; and other values (which are not infs or nans) are very close.<br>
I want to know why nans is procuded by pytorch. Thanks</p>

ptrblck: <p>NaN creation depends on the used operations and backend. While e.g. cuDNN will propagate NaNs, it’s currently unclear which operation creates them in the first place. I’m unsure what your use case is to differentiate between Infs and NaNs, but feel free to post a minimal and executable code snippet reproducing the difference.</p>

3yuan: <p>The operation is Conv2d in stem stage in resnet.<br>
The test steps was posted <a href="https://discuss.pytorch.org/t/the-gradient-from-loss-backward-is-different-from-the-gradient-computed-manually/202931">here</a></p>

ptrblck: <p>I don’t see any executable code snippet in the linked post, unfortunately, so don’t know what’s causing the difference.</p>

3yuan: <p>It may be difficult to reproduce, I just debug it in my project, and found the diff between the gradient from backward() and the the value compute using chain rule in the conv layer.<br>
Thanks</p>
NCCL for Windows

sarthak_verma: <p>Hello everyone i am new to the community, Is there any way to build NCCL in windows to get the power of NCCL for multi-GPU training??</p>

ptrblck: <p>No, since NCCL does not support Windows. Use Gloo for distributed operations on Windows.</p>

sarthak_verma: <aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/MyCaffe/NCCL">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/MyCaffe/NCCL" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/18c29a92550576b7e1d29431a100a51aa7f6745d5e4e10995a435b6991790043/MyCaffe/NCCL" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/MyCaffe/NCCL" target="_blank" rel="noopener nofollow ugc">GitHub - MyCaffe/NCCL: Windows version of NVIDIA's NCCL ('Nickel') for...</a></h3>

  <p>Windows version of NVIDIA's NCCL ('Nickel') for multi-GPU training - please use https://github.com/NVIDIA/nccl for changes. - MyCaffe/NCCL</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
in the above link there is build of nccl for windows given by MyCaffe, will it work if i build nccl from this and use it my code for PyTorch.</p>

sarthak_verma: <p>Please Help if you can</p>

ptrblck: <p>As already explained, NCCL on Windows is not supported and I don’t know if this fork will work. You could of course try it out or just use Gloo.</p>

sarthak_verma: <p>Okay Thanks for the help</p>
Fit a custom function parameters

Ghostv: <p>Hi,</p>
<p>I’m trying to do something very basic but i’m not quite sure if it’s possible with pytorch (and how to do it).</p>
<p>I wanna fit a custom function to a series of data in order to get the fitting parameters from it (ofc, if it converges).</p>
<p>I wanna do something like this… let’s say i have a series of data x = [x1,x2,x3,… …,xn], y = [y1,y2,y3,… …,yn] and my hypothesis is that it can be interpolated by a function f(x) with coefs. let’s say… A, B and C, let’s say a polynomial → y = Ax^2 + Bx + C</p>
<p>How would a general model for this be? My guess is something like this, but not sure “how to tell pytorch” which vars to vary in order to fit it:</p>
<pre><code class="lang-auto">class NN():
    def __init__(self):
        super().__init__()
        self.fx = A*x**2 + B*x +C

    def forward(self,x):
        y0 = self.fx(x)
</code></pre>
<p>Any advice if that’s the right way to do it?</p>

ptrblck: <p>You would define trainable parameters in the <code>__init__</code> method and use them in the <code>forward</code>. <a href="https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html">This tutorial</a> might be helpful.<br>
For your example, this should work:</p>
<pre data-code-wrap="python"><code class="lang-python">class NN(nn.Module):
    def __init__(self):
        super().__init__()
        self.A = nn.Parameter(torch.randn(1)) 
        self.B = nn.Parameter(torch.randn(1))
        self.C = nn.Parameter(torch.randn(1))

    def forward(self,x):
        out = self.A * x**2 + self.B * x + self.C
        return out

# create data    
x = torch.linspace(0, 10, 1000)
A, B, C = 2, 3, 4
target = A * x**2 + B * x + C

plt.plot(x.numpy(), target.numpy())

# model training
model = NN()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()

nb_epochs = 25000
for epoch in range(nb_epochs):
    optimizer.zero_grad()
    out = model(x)
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()
    print("epoch: {}, loss: {:.3f}\nA: {}, B: {}, C: {}".format(
        epoch, loss.item(), model.A.item(), model.B.item(), model.C.item()))

# ...
# epoch: 24999, loss: 0.000
# A: 2.0, B: 3.0, C: 4.0
</code></pre>

Ghostv: <p>Perfect!!! That’s what i was missing, thanks a lot!!</p>
<p>How do i get the values of the parameters after training?</p>

ptrblck: <p>The <code>print</code> statement directly access the trained parameters and shows their values during the training.</p>
How to inspect the memory pool usage for cudagraph capture?

youkaichao1: <p>Hi, team, I find pytorch cudagraph capture uses pytorch internal memory pool rather than cuda async memory pool. Then, are there any methods that I can use, to query the status of the memory pool? e.g. current usage and reserved memory.<br>
In cuda, I can use <code>cudaDeviceGetGraphMemAttribute</code> .</p>
Issue with max-autotune mode in torch.compile

Prakhar: <p>Hi,</p>
<p>I am getting the following warning when trying to use the max-autotune mode using torch.compile.</p>
<p>“”"<br>
torch._inductor.utils: [WARNING] not enough SMs to use max_autotune_gemm mode<br>
skipping cudagraphs due to [‘non-cuda device in graph’]</p>
<p>“”"</p>
<p>Does this mean that I am using max-autotune no cudagraph mode automatically, if so, can someone explain how much performance could I possibly be using.</p>
<p>Also, I am using NVIDIA A100 GPU, I don’t think this warning should have come at all.</p>

ptrblck: <aside class="quote no-group" data-username="Prakhar" data-post="1" data-topic="203758">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prakhar/48/61211_2.png" class="avatar"> Prakhar:</div>
<blockquote>
<p>Also, I am using NVIDIA A100 GPU, I don’t think this warning should have come at all.</p>
</blockquote>
</aside>
<p>That’s right since your A100 should have 108 SMs while the warning uses a threshold of <code>68</code> as seen <a href="https://github.com/pytorch/pytorch/blob/d2df0f56a3046d9f8bf8eb34fd787bf506647d3c/torch/_inductor/utils.py#L944-L949">here</a>.<br>
Did you enable MIG and reduced the SM count?</p>
How do I fix the error RuntimeError: mat1 and mat2 shapes cannot be multiplied (37x12288 and 24576x128)?

lovejn03: <pre><code>    # Correct input features for fc1
    self.fc1_input_features = self._get_fc1_input_features()
    self.fc1 = nn.Linear(self.fc1_input_features, 128)
    self.fc2 = nn.Linear(128, 6)

def _get_fc1_input_features(self):
    with torch.no_grad():
        x = torch.zeros(1, 1, 3, 1024)  # Simulate input
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = x.view(x.size(0), -1)
        return x.size(1)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = x.view(x.size(0), -1)  # Correct shape
    x = F.relu(self.fc1(x))
    x = self.fc2(x)  # Removed extra F.relu
    return x
</code></pre>

ptrblck: <p>Your code is incomplete so could you post the missing code snippets including the shape of the input to create an executable code snippet reproducing the issue?</p>
Can't force class weights

ekrx: <p>I have a class imbalance where class are split like this:<br>
0: 5.2%<br>
1: 6.51%<br>
2: 76.7%<br>
3: 6.38%<br>
4: 5.13%</p>
<p>I’m computing class weights with</p>
<pre><code class="lang-auto">from sklearn.utils.class_weight import compute_class_weight
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(ys), y=ys)
# produces : class_weights = [3.8412954, 3.06959094, 0.26054039, 3.13274541, 3.8984915]
...

loss_function = nn.CrossEntropyLoss(weight=class_weights)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

</code></pre>
<p>However I still see that the model have bias towards predicting class 2 and not really learning anything other class.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebdc3829515f6e5990daad776afb1e423374bd48.png" data-download-href="https://discuss.pytorch.org/uploads/default/ebdc3829515f6e5990daad776afb1e423374bd48" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebdc3829515f6e5990daad776afb1e423374bd48_2_690x162.png" alt="image" data-base62-sha1="xEw1rXoeoXSDIh0KntrSsxAEiSc" width="690" height="162" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebdc3829515f6e5990daad776afb1e423374bd48_2_690x162.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebdc3829515f6e5990daad776afb1e423374bd48_2_1035x243.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/ebdc3829515f6e5990daad776afb1e423374bd48_2_1380x324.png 2x" data-dominant-color="FAFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1885×444 32.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any advice how to address this?</p>

ptrblck: <p>Did you increase the weights of the minority classes to “force” the model to learn them or did you only use the <code>sklearn</code> computed weights?</p>

ekrx: <p>I just used the sklearn computed weights.</p>
<blockquote>
<p>Did you increase the weights of the minority classes to “force” the model to learn them</p>
</blockquote>
<p>I thought that by using the weights computed by sklearn I was doing just that.</p>

ekrx: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I was wondering if you had some other suggestions here.<br>
Thanks in advance.</p>

ptrblck: <p>Increase the weights of the minority classes until your model is overfitting these to make sure your code does not have any other issues.</p>
Support for Snapdragon X Elite

Anders_V: <p>I’m wondering what support there is for ARM-based laptops, especially the upcoming Snapdragon X Elite chip, which seems to have a massive focus on AI, but nobody is talking about what this means in terms of development and training machine learning models?</p>
<p>Although not the exact same, a friend of mine bought the original M1 powered Mac laptops when they launched and they struggled with certain development tasks because compilers were still catching up with official support (as opposed to emulation).</p>
Disk Quota Exceeded error

Prakhar: <p>Hi, I’m not even sure if this is aboslutely related to Pytorch. Anyways, it is my first time writing a query so please ask any questions that will help you better understand the problem.</p>
<p>This is the final error that I am getting:<br>
…<br>
torch._dynamo.exc.BackendCompilerFailed: backend=‘inductor’ raised:<br>
OSError: [Errno 122] Disk quota exceeded: &lt;directory/.triton/dump/long_alphanumeric&gt;<br>
…</p>
<p>I have run the same code previously and did not encounter this error. This is the first time. Please provide any guidance if this has happened with you too</p>

ptrblck: <p>You might need to clear the default cache folder on your system and/or set the Triton cache to another dir via <code>TRITON_CACHE_DIR</code>.</p>

Prakhar: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, it is a shared system and several other users are also using the same GPU machine. Won’t clearing the default cache cause issue for them?</p>

ptrblck: <p>That’s a valid concern unless each user is working in e.g. an isolated docker container and is using local folders for their cache.</p>

Prakhar: <p>So, for anyone facing this issue later:<br>
This is the code to find out the directory in which the Triton cache is currently getting stored<br>
“”"<br>
import os<br>
import triton</p>
<p>current_cache_dir = os.getenv(‘TRITON_CACHE_DIR’, os.path.expanduser(‘~/.triton/’))<br>
print(f"Current Triton cache directory: {current_cache_dir}“)<br>
“””</p>
<p>and then this is the code to change the directory<br>
“”"<br>
os.environ[‘TRITON_CACHE_DIR’] = ‘/path/to/new/cache/dir’<br>
“”"</p>
<p>Now, moving onto my issue. It is somehow automatically resolved, but if it happens again, I guess I can use this directory change method.<br>
I am keeping this open so that if the problem arises next time and I’m unable to fix, I’ll continue here.</p>
TypeError: Caught TypeError in DataLoader worker process 0.

Arpita_Halder: <p>TypeError: Caught TypeError in DataLoader worker process 0.</p>
<p>Original Traceback (most recent call last):</p>
<p>File “/home/hpc/rzku/mlvl109h/.conda/envs/CineVN/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py”, line 308, in _worker_loop</p>
<pre><code>data = fetcher.fetch(index)
</code></pre>
<p>File “/home/hpc/rzku/mlvl109h/.conda/envs/CineVN/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py”, line 54, in fetch</p>
<pre><code>return self.collate_fn(data)
</code></pre>
<p>File “/home/hpc/rzku/mlvl109h/.conda/envs/CineVN/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 265, in default_co&gt;</p>
<pre><code>return collate(batch, collate_fn_map=default_collate_fn_map)
</code></pre>
<p>File “/home/hpc/rzku/mlvl109h/.conda/envs/CineVN/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 132, in collate</p>
<pre><code>return elem_type(*(collate(samples, collate_fn_map=collate_fn_map) for samples in zip(*batch)))
</code></pre>
<p>File “/home/hpc/rzku/mlvl109h/.conda/envs/CineVN/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 132, in </p>
<pre><code>return elem_type(*(collate(samples, collate_fn_map=collate_fn_map) for samples in zip(*batch)))
</code></pre>
<p>File “/home/hpc/rzku/mlvl109h/.conda/envs/CineVN/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”, line 150, in collate</p>
<pre><code>raise TypeError(default_collate_err_msg_format.format(elem_type))
</code></pre>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘NoneType’&gt;</p>

ptrblck: <p>Based on the stacktrace it seems your <code>Dataset.__getitem__</code> method returns <code>None</code> instead of valid tensors or arrays, so could you check why this might be the case?<br>
Are you using a custom <code>Dataset</code> and forgot to <code>return</code> the samples?</p>
Why the number of FLOPs won't change in a model because of activation function change

palguna_gopireddy: <p>Hi.<br>
I am using the following code to count the number of flops in my model.</p>
<pre><code class="lang-auto">def measure(model):
        model.eval()
        model.cuda()

        dummy_input = torch.randn(1, 2, 128, 128).cuda()

        macs, params = profile(model, inputs=(dummy_input,), verbose=0)
        macs, params = clever_format([macs, params], "%.3f")
        print("&lt;" * 10)
        print("Flops:", macs)
        print("Parameters:", params)
</code></pre>
<p>I used different activation functions in my model like ReLU, Swish, Mish, TanhExp. But all have given same number of flops.</p>
<p>So my questions are</p>
<ol>
<li>Why the change in the flops is not happening because of change in activation function</li>
<li>Am I using the correct approach for calculating the flops?</li>
</ol>
1Torch was not compiled with flash attention. ComfyUI user

kubi_kubicol: <p>Im a ComfyUI user and i have this errors when I try to generate images:</p>
<p>C:\Python312\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\attention.py:345: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at …\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)<br>
out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False)</p>
<p>and</p>
<p>:\Python312\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\conv.py:605: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at …\aten\src\ATen\native\cudnn\Conv_v8.cpp:919.)<br>
return F.conv3d(</p>
<p>the program stop to working.</p>
<p>Can somebody help me please?</p>
PyTorch install temp files directory

Esih_Sukaesih: <p>Hello,</p>
<p>so i installed PyTorch using the command below</p>
<pre><code class="lang-auto">pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu121
</code></pre>
<p>and because my C:// is full, i couldnt install it and after i restarted my pc, the disk space is still full because of the files downloaded during installation.</p>
<p>now i want to clean up my disk, but i dont know where are those files located</p>

ptrblck: <p>The question seems to be specific to <code>pip</code> and <a href="https://stackoverflow.com/a/41111916">this answer</a> shows where the pip cache folders are located in case you want to clean them.</p>

Esih_Sukaesih: <p>i actually checked pip folder in app data before, but no luck<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92ad755dc72d55d379888164d97123add3b44eb4.png" data-download-href="https://discuss.pytorch.org/uploads/default/92ad755dc72d55d379888164d97123add3b44eb4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92ad755dc72d55d379888164d97123add3b44eb4.png" alt="image" data-base62-sha1="kVzljhJpCSv4v8qWd64YxU5jDla" width="690" height="223" data-dominant-color="202020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">969×314 2.66 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>In that case the cache might not cause the issue, but actually downloaded and installed packages?<br>
Did you check the installed packages in your pip envs?</p>
Is `torch.ao.quantization` being migrated to `torchao.quantization`?

johncf: <p>I see that these two modules are very different code bases:</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/tree/main/torch/ao/quantization" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/ao/quantization at main · pytorch/pytorch · GitHub</a></li>
<li><a href="https://github.com/pytorch/ao/tree/main/torchao/quantization" class="inline-onebox" rel="noopener nofollow ugc">ao/torchao/quantization at main · pytorch/ao · GitHub</a></li>
</ul>
<p>But they share basically the same name. <code>torchao</code> seems to be updated more frequently, so is it aiming to replace the other?</p>

johncf: <p>Looks like this question was asked in an issue under <code>torchao</code> repo, <a href="https://github.com/pytorch/ao/issues/47#issuecomment-2011154766" rel="noopener nofollow ugc">here</a>. And <a href="https://github.com/pytorch/ao/issues/47#issuecomment-2011227741" rel="noopener nofollow ugc">msaroufim responded</a>:</p>
<blockquote>
<p>eventually once things get proven out in <code>torchao</code> they would get upstreamed to <code>torch.ao</code> so the goal is to have this be a standalone repo to have a higher development velocity.</p>
</blockquote>
<p>So looks like I got it backwards in my question.</p>

HDCharles: <p>Yeah, in the past, quantization has been a largely cpu focused technique, gpu models tended to focus on CNNs which are harder to speed up with quantization. As a result, the suite of tools in torch/ao reflects that. Its only in the last 1.5 years where transformers and perf optimized gpu use cases have become a much bigger area that we started focusing on gpu quantization. Core Pytorch has a higher release standard and its own release cadence that doesn’t work well for us at the moment.  Its easier to move fast in that repo even if we eventually want to consolidate back into core down the line.</p>
Is it possible to have trainable and non-trainable parameters in the same tensor?

Miguel_Camacho: <p>Hi PyTorch community,</p>
<p>I have a question regarding tensor optimization in PyTorch. I would like to know if it’s possible to have a tensor where some parts are trainable (i.e., <code>requires_grad=True</code>) and other parts are fixed (i.e., <code>requires_grad=False</code>).</p>
<h3><a name="context-1" class="anchor" href="#context-1"></a>Context:</h3>
<p>I’m working with a regression model and I want to freeze the weights of the model. Then, I want to train only part of the input vector to minimize the model’s output. However, I want some parts of the input vector to remain fixed during training.</p>
<h3><a name="example-2" class="anchor" href="#example-2"></a>Example:</h3>
<p>Let’s say I have an input vector of 9 parameters, where the first 3 should be fixed and the remaining 6 should be trainable. Here is the code I have tried:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

# Suppose this is your regression model

class RegressionModel(nn.Module):
    def __init__(self):
        super(RegressionModel, self).__init__()
        self.fc1 = nn.Linear(9, 1)  # A model with a single linear layer

    def forward(self, x):
        return self.fc1(x)


# Instantiate the model
model = RegressionModel()

# Freeze the model weights
for param in model.parameters():
    param.requires_grad = False

# Create the input
input_vector = torch.tensor([0.19463597, 1.436385601, 0.028681654,
                             0.5, 0.5, 0.5, 0.5, 0.5, 0.5], requires_grad=True)
input_vector = input_vector.unsqueeze(0)

fixed_params = [0, 1, 2]
for idx in fixed_params:
    input_vector[0, idx] = torch.tensor(input_vector[0, idx].item(), requires_grad=False)

# Here I have this error: RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
</code></pre>
<h3><a name="question-3" class="anchor" href="#question-3"></a>Question:</h3>
<p>Is there a way to directly have parts of a tensor with <code>requires_grad=True</code> and other parts with <code>requires_grad=False</code>?</p>

ptrblck: <p>No, the <code>requires_grad</code> attribute is set for the tensor itself and you might need to create different subtensors and recreate the final tensor as seen in <a href="https://discuss.pytorch.org/t/custumize-the-backpropagation-phase-of-a-neural-network-to-ignore-the-update-of-some-parameters/167557/4">this example</a>.</p>
How to use ParallelEnv?

Vefery: <p>I finished <a href="https://pytorch.org/rl/tutorials/coding_ppo.html" rel="noopener nofollow ugc">this</a> guide and in the end there’s a note at the end that I can use ParallelEnv env to speed up data collection. I couldn’t find any examples or information about this besides documentation page. So I tried to just change TransformedEnv to ParallelEnv in cod. But when I try to use data collector it throws me errors</p>

vmoens: <p>The syntax is</p>
<pre><code class="lang-auto">ParallelEnv(num_of_p_envs: int, env_constructor: Callable[[], EnvBase], mp_start_method: str="spawn")
</code></pre>
<p>if you want to spawn one single env (alternatively you can pass a list of constructors as second argument)<br>
Example</p>
<pre data-code-wrap="python"><code class="lang-python">from torchrl.envs import GymEnv, ParallelEnv
import functools
make_env = functools.partial(GymEnv, "CartPole-v1")
num_envs = 2

env = ParallelEnv(num_envs, make_env)
assert env.batch_size = (num_envs, )
print(env.rollout(10, break_when_any_done=False))

</code></pre>
<p>The doc is here<br>
<a href="https://pytorch.org/rl/reference/generated/torchrl.envs.ParallelEnv.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/rl/reference/generated/torchrl.envs.ParallelEnv.html</a></p>

Vefery: <p>Sorry for late reply, I had a busy month</p>
<p>Thanks to you and docs I figured how to create ParallelEnv and do random rollouts. But how do I use it with an actual policy? In my case the policy is naturally made to work with one environment (in: number of features, out: number of actions). But when I try to execute something like this:</p>
<pre><code class="lang-auto">env.rollout(10, policy=policy_module)
</code></pre>
<p>I get dimensions mismatch error because rollout has different shape. I’m not sure what to do, I can’t access rollout code, and designing policy module specifically for ParallelEnvs seems wrong</p>

vmoens: <p>Can you give an example or a snippet of how you’re building the policy? Or the error you’re getting?<br>
In general if your policy can look at your observation regardless of the batch size you should be fine.</p>
DWC to Torch Linear or MatMul

hibama: <p>Hello,</p>
<p>Recently, I read “Flatten Transformer: Vision Transformer using Focused Linear Attention”(<a href="https://arxiv.org/abs/2308.00442" class="inline-onebox" rel="noopener nofollow ugc">[2308.00442] FLatten Transformer: Vision Transformer using Focused Linear Attention</a>) and I became curious about the method mentioned in the paper to transform Depthwise Convolution into a simple Matmul operation. According to the paper, a DWC operation with a (192, 14, 14) input and a (5, 5) kernel can be replaced with a (3, 196, 196) @ (192, 14, 14) form. However, I am having difficulty implementing this method.</p>
<pre><code class="lang-auto">def depthwise_conv2d_matmul(input, weight, bias=None, stride=1, padding=2, dilation=1):
    bsz, channels, h, w = input.shape
    k_channels, _, k_h, k_w = weight.shape

    assert h == w, 'Input tensor must be square'
    assert channels == k_channels, 'Number of input channels and kernel channels must match'
    assert k_h == k_w, 'Kernel must be square'
    input_size = h
    kernel_size = k_h

    # Split the input tensor and weight tensor along the channel dimension
    input_splits = input.split(1, dim=1)
    weight_splits = weight.split(1, dim=0)
    #print(input_splits[0].shape)
    output_splits = []
    for i in range(channels):
        # Unfold the input, input_unf.shape:                                torch.Size([bsz, kernel_size*kernel_size, window_size])
        input_unf = F.unfold(input_splits[i], weight_splits[i].shape[-2:], dilation=dilation, padding=padding,
                             stride=stride)
        #print(input_unf.shape)
        # Perform depth-wise convolution
        # input_unf.transpose(1, 2) shape:                                  torch.Size([bsz, window_size, kernel_size*kernel_size])
        # weight_splits[i].view(weight_splits[i].shape[0], -1).t() shape:   torch.Size([kernel_size*kernel_size, 1])
        # out_unf.shape:                                                    torch.Size([bsz, 1, window_size])

        out_unf = input_unf.transpose(1, 2).matmul(
            weight_splits[i].view(weight_splits[i].shape[0], -1).t()
        ).transpose(1, 2)
        # If bias is not None, add bias
        if bias is not None:
            out_unf += bias[i].view(1, -1, 1)

        # Fold the output tensor and add it to the list of output splits
        combined_out = F.fold(out_unf, (input_size + (2 * padding) - (dilation * (kernel_size - 1)) - 1) // stride + 1,
                              (1, 1))

        output_splits.append(combined_out)

    # Concatenate the output splits along the channel dimension to get the final output
    return torch.cat(output_splits, dim=1)
</code></pre>
<p>I have found examples that use unfold to partially modify DWC into Matmul, but how can I use this to modify it into a simple two-matrix multiplication as described above?</p>
Next(iter()) keeps returning the same data

miguel_angel: <p>Hello! I’m trying to access images from a testloader I made with dataloader so I can pass it onto a network. However, everytime I run the code cell with next(iter()) I keep getting the same images back to back even though there are so many to choose from! I thought every time I ran that cell block I’d get a new batch of images for test… Can someone help me with this?</p>
<pre><code class="lang-auto"># testing data
testset = datasets.ImageFolder('./data/test_data', transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=32, shuffle=True)
</code></pre>
<pre><code class="lang-auto"># Showing the images
test_images, test_labels = next(iter(testloader))
fig = plt.figure(figsize=(15,4))

for i in range(5):
    
    ax = fig.add_subplot(1, 5, i + 1)
    plt.imshow(images[i].permute(1, 2, 0))
    plt.axis("off")
</code></pre>
<p>This is the output every single time ):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f7dbf68858c0d3dccf27ef13a6b775f78350ba8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2f7dbf68858c0d3dccf27ef13a6b775f78350ba8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2f7dbf68858c0d3dccf27ef13a6b775f78350ba8_2_690x128.jpeg" alt="image" data-base62-sha1="6M7QXTzGKmnK7cIMQHCMTc6URks" width="690" height="128" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/f/2f7dbf68858c0d3dccf27ef13a6b775f78350ba8_2_690x128.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f7dbf68858c0d3dccf27ef13a6b775f78350ba8.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/f/2f7dbf68858c0d3dccf27ef13a6b775f78350ba8.jpeg 2x" data-dominant-color="A2A0A0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">851×158 49.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Unity05: <p>Are you doing <code>images[i]</code> instead of <code>test_images[i]</code> on purpose?</p>

sepiatone: <p>You need to create the iterable (using iter()) and then call next() on it in the loop. Something like -</p>
<pre><code class="lang-auto"># Showing the images
it_images = iter(testloader)
# test_images, test_labels = next(iter(testloader))
fig = plt.figure(figsize=(15,4))

for i in range(5):
    
    ax = fig.add_subplot(1, 5, i + 1)
    images, _ = next(it_images)
    plt.imshow(images.permute(1, 2, 0))
    plt.axis("off")
</code></pre>
<p>Edit: Yes I’m aware it’s 3 years later, but it may help someone coming along…</p>
How to avoid unused parameters for DDP

Johannes: <p>Hi all,</p>
<p>I recently came across the unused parameters error (<code>RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one.</code>) when training with <code>DDP</code>. In my setup I need to train one network A for some amount of iterations <code>N</code>, after which I will be training network B for some amount of iterations <code>M</code>. Network A will be used to generate some data used in training network B. The process of training networks A and B is then iterated.<br>
So, when training network B I am not using network A of my model (therefore the <code>RuntimeError</code>). In this case, what is the advised setup for training network A and B (without needing to specify <code>find_unused_parameters=True</code>, to avoid additional computational overheads)?<br>
Thank you very much in advance!</p>
<p>Best,<br>
Johannes</p>

H-Huang: <p>Are network A and network B in the same model? It seems like they are and then you do DDP(model), but why not do DDP(modelA) and DDP(modelB)? Does the data generated by A need to be back propagated after calculating the loss from B?</p>

Johannes: <p>Yes, network A and network B are in the same model. So what you’re saying is that I should have 2 (separate) models, one only with network A and one only with network B. Is it right that then DDP should not complain about the parameter of network B (in model_B) if I only call model_A(data)?</p>
<p>Regarding your second point there is currently no need to backdrop the gradients (if this happens though, I would assume that I wouldn’t run into this problem as long as I use network A to generate data for network B in every iteration).</p>
Autograd with adjacency matrices

alexj: <p>I am trying to write Graph NN algorithms without using pytorch-geometric. I will have no sparse learnable parameters, however I will have a lot of embeddings and MLPs.<br>
When documentation states ‘doesn’t support computing derivaties’ (torch.sparse.mm) or 'may not have autograd support" ( torch.nn.functional.linear), do you mean I cannot use sparse tensors at all, or only cannot learn sparse tensors?<br>
What would you recommend to use in my case? My adjacency matrices are quite large, they cannot be dence.</p>
Using feature extractor with module list

Mathew_Sam: <p>Hey guys, I’m trying to get an intermediate feature extractor with the depth anything model. My code looks like this:</p>
<p>self.image_processor = AutoImageProcessor.from_pretrained(“LiheYoung/depth-anything-small-hf”)<br>
model = AutoModelForDepthEstimation.from_pretrained(“LiheYoung/depth-anything-small-hf”)<br>
model.eval()<br>
self.feature_extractor = create_feature_extractor(model.neck,{‘fusion_stage’:‘layers’})</p>
<p>I got this error :<br>
self.feature_extractor = create_feature_extractor(model.neck,{‘fusion_stage’:‘layers’})<br>
*** ValueError: hidden_states should be a tuple or list of tensors<br>
Model architecture looks like this:</p>
<p>DepthAnythingNeck(<br>
(reassemble_stage): DepthAnythingReassembleStage(<br>
(layers): ModuleList(<br>
(0): DepthAnythingReassembleLayer(<br>
(projection): Conv2d(384, 48, kernel_size=(1, 1), stride=(1, 1))<br>
(resize): ConvTranspose2d(48, 48, kernel_size=(4, 4), stride=(4, 4))<br>
)<br>
(1): DepthAnythingReassembleLayer(<br>
(projection): Conv2d(384, 96, kernel_size=(1, 1), stride=(1, 1))<br>
(resize): ConvTranspose2d(96, 96, kernel_size=(2, 2), stride=(2, 2))<br>
)<br>
(2): DepthAnythingReassembleLayer(<br>
(projection): Conv2d(384, 192, kernel_size=(1, 1), stride=(1, 1))<br>
(resize): Identity()<br>
)<br>
(3): DepthAnythingReassembleLayer(<br>
(projection): Conv2d(384, 384, kernel_size=(1, 1), stride=(1, 1))<br>
(resize): Conv2d(384, 384, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))<br>
)<br>
)<br>
)<br>
(convs): ModuleList(<br>
(0): Conv2d(48, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(1): Conv2d(96, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(2): Conv2d(192, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(3): Conv2d(384, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
)<br>
(fusion_stage): DepthAnythingFeatureFusionStage(<br>
(layers): ModuleList(<br>
(0-3): 4 x DepthAnythingFeatureFusionLayer(<br>
(projection): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1))<br>
(residual_layer1): DepthAnythingPreActResidualLayer(<br>
(activation1): ReLU()<br>
(convolution1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(activation2): ReLU()<br>
(convolution2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
)<br>
(residual_layer2): DepthAnythingPreActResidualLayer(<br>
(activation1): ReLU()<br>
(convolution1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(activation2): ReLU()<br>
(convolution2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
)<br>
)<br>
)<br>
)<br>
)</p>
<p>Any feedback is appreciated</p>

Flowersfr: <p>What you did to fix this?</p>

Mathew_Sam: <p>I’ll let you know once I find out <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=12" title=":laughing:" class="emoji" alt=":laughing:" loading="lazy" width="20" height="20"></p>

Flowersfr: <p>hahhaha<a href="https://dgcustomerfirst.fun/" rel="noopener nofollow ugc">.</a>!!<br>
I’ll wait…</p>
How to see details behind CPU-only Libtorch Matrix-Matrix multiplication routines?

velenos14: <p>I have downloaded the libtorch CPU-only version from the website and unzipped it.</p>
<p>Inside my .cpp application, I write (I am using <code>intel-mkl</code>):</p>
<pre><code class="lang-auto">    omp_set_num_threads(64);
    mkl_set_num_threads(64);
</code></pre>
<p>I then check:</p>
<pre><code class="lang-auto">    std::cout &lt;&lt; "torch::get_num_threads() returns: " &lt;&lt; torch::get_num_threads() &lt;&lt; std::endl;

    std::cout &lt;&lt; "omp_get_max_threads() returns: " &lt;&lt; omp_get_max_threads() &lt;&lt; std::endl;
    std::cout &lt;&lt; "mkl_get_max_threads() returns: " &lt;&lt; mkl_get_max_threads() &lt;&lt; std::endl;
</code></pre>
<p>These all return 64. (yes, I do have so many cores, I am on a HPC machine with 128 cores per node and I am launching 2 MPI processes per node).</p>
<p>I then perform <code>std::complex&lt;double&gt;</code> matrix-matrix multiplications via <code>torch:matmul()</code>.<br>
These multiplications, for me, seem to be slow.</p>
<p>How can I check that:</p>
<ol>
<li>Libtorch uses MKL behind the scenes</li>
<li>Libtorch uses threads for its MM multiplications? Is my check from above guaranteeing that Libtorch uses more than 1 thread behind the scenes?</li>
</ol>
<p>Thank you!</p>

akss: <p>The linked libraries can be found with the “dependency walker” tool or similar on Windows and ldd on linux. A running process can be killed and the stack calls in coredump should reveal mkl calls.</p>
<p>The number of thread can be observed with the OS tools, Sysinternals Process Explorer on Windows or top on linux</p>
Pytorch Distributed RPC connection using nvidia Nanos IP Addresses

Nishant_Madan: <p>I have 2 nvidia Nanos. I want to run the pretrained Pytorch Resnet sharded model on those devices. My main.py code is on GPU and I want to connect those devices using the code on the GPU for distributed inferencing. I am using the Distributed RPC. Please let me know how can I do the distributed inferencing using Pytorch Distributed module or any other suggestion is also welcoming.</p>
Pytorch FCN segmentation model architecture question

mikae: <p>Hi,</p>
<p>I am trying to do semantic segmentation and thought of trying out torchvision.models.segmentation.fcn_resnet50. I just see down convolution in this model and there is no upsampling happening through conv transpose layers. Then, how does the segmentation output be of same size as of the input?</p>
<p>Thanks</p>

ptrblck: <p><code>_SimpleSegmenationModel</code> uses <code>F.interpolate(x, size=input_shape)</code> <a href="https://github.com/pytorch/vision/blob/c585a515fba2c34d5fbef221fccc0902db588131/torchvision/models/segmentation/_utils.py#L28">here</a> and will thus interpolate the output to match the input shape.</p>
How can I remove the 0 in the 2-dimensional tensor?

songsong0425: <p>Greetings, I want to extract the nonzero elements in the 2-dimensional tensor.</p>
<pre data-code-wrap="python"><code class="lang-python">example
# tensor([[    0,  4535,     0,  ...,  9446,  5344, 0],
#         [    0,  1037,     0,  ...,  2391, 33243, 40207]])

result
# tensor([[4535,   ...,  9446,  5344],
#         [1037,  ...,  2391, 33243]])
</code></pre>
<p>I thought it was a simple problem, but I countered the error because of the mismatch of shape.<br>
I tried as below. Does anyone have a good idea?</p>
<pre data-code-wrap="python"><code class="lang-python">result[:, torch.cat((result[0].nonzero()[:, 0], result[1].nonzero()[:, 0])).unique()]

#tensor([[ 4535, 34602, 31080,  2097, 30280, 30764,     0, 30471,  5728, 32816, #Unexpected 0 
#              2012,     0,  6726, 30444,     0, 34984, 32871, 32039,     0, 32528, ...], #Unexpected 0 
#            [ 1037, 35914,  9275,  7511,  9624,  9925,  7217,  9039,  1386,   362,
#              2022,  2719,     0,  6996, 31594, 34346, 33777, 34213, 37084, 32728,...]]) #Unexpected 0 
</code></pre>
<p>Thank you for reading this question.</p>