akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What does `torch/distributed/fsdp/flat_param.py", line 1999, in _check_storage_allocated p_assert(storage_size > 0, "Expects storage to be allocated")` assertion do?

yxchng1: <p>What does <code>torch/distributed/fsdp/flat_param.py", line 1999, in _check_storage_allocated     p_assert(storage_size &gt; 0, "Expects storage to be allocated")</code> assertion do?</p>
<p>What can be the possible causes for storage size being 0?</p>

sujayskumar: <p>Getting the same error. Any ideas?</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the error, please?</p>

shikhar-srivastava: <p>Hard to produce a minimal executable for this. Faced the same error.<br>
Removed the assert, and FSDP training seems to execute just fine.</p>
<p>Would be helpful to understand what scenarios this assert solves for?</p>
Questions about Independent distribution

zichunxx: <p>Hi!</p>
<p>I read the introduction to Independent distribution but was confused about the code snippet example:</p>
<pre><code class="lang-auto">from torch.distributions.multivariate_normal import MultivariateNormal
from torch.distributions.normal import Normal
loc = torch.zeros(3)
scale = torch.ones(3)
mvn = MultivariateNormal(loc, scale_tril=torch.diag(scale))
[mvn.batch_shape, mvn.event_shape]
normal = Normal(loc, scale)
[normal.batch_shape, normal.event_shape]
diagn = Independent(normal, 1)
[diagn.batch_shape, diagn.event_shape]
</code></pre>
<ol>
<li>
<p>What are <code>batch_shape</code> and <code>event_shape</code>?</p>
</li>
<li>
<p>Why does <code>Independent</code> reinterpret the batch dims of distribution as event dims?</p>
</li>
</ol>
<p>It would be nice to get some easy-to-understand answers. Thanks!</p>
Can I use my gpu Nvidia GeForce 920M with Pytorch GPU?

MonojitSarkar: <p>My gpu is pretty old. And the latest pytorch gpu has stopped support for it.</p>
<p>However I am still willing to use older versions of pytorch if that can make my gpu work?</p>
<p>Can someone offer me some advice on it? Or can I use the latest pytorch gpu version along with gpu?</p>
<p>Note: My gpu already supports cuda, but latest pytorch gpu considers my gpu obsolete.</p>
<p>Thanks.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/no-kernel-image-is-available-for-execution-on-the-device/175011/3">This topic</a> is related and based on the response it seems <code>torch==1.2.0</code> works for your Kepler GPU.</p>

MonojitSarkar: <p>thanks for the response <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Implementing Quantized Linear Layer in Numpy

costelljoe: <p>I’m trying to implement a Torch quantized linear layer in Numpy so that I can verify the implementation before translating to my own embedded C code. I have a basic implementation below which correctly quantizes the inputs but the output of the linear layer doesn’t match the torch outputs.</p>
<p>My current code is based on <a href="https://discuss.pytorch.org/t/is-bias-quantized-while-doing-pytorch-static-quantization/146416/6">this post</a>, and I’m using the qnnpack backend.</p>
<p>Any tips are appreciated!</p>
<pre><code class="lang-auto"># I have float input 'x', linear layer 'fc', and quant stub 'quantx'

# quantize x
x_q = np.round((x / quantx_scale) + quantx_zero_point)
x_q = np.clip(x_q, 0, 255).astype(np.uint8)

# linear layer
matmul_out = np.matmul(x_q, fc_weight.T)
bias_q = np.round(fc_bias / (quantx_scale * fc_weight_scale)).astype(np.int32)
scale_factor = quantx_scale * fc_weight_scale / fc_scale
outq = np.round((matmul_out + bias_q) * scale_factor + fc_zero_point)
outq = np.clip(outq, 0, 255).astype(np.uint8)
</code></pre>
<p>I’m extracting the weights like:</p>
<pre><code class="lang-auto">fc_weight = fc.weight().int_repr().numpy()
fc_weight_scale = fc.weight().q_scale()
fc_bias = fc.bias().detach().numpy()
fc_scale = fc.scale
fc_zero_point = fc.zero_point
quantx_scale = quantx.scale.numpy()[0]
quantx_zero_point = quantx.zero_point.numpy()[0]
</code></pre>

chenster_liu: <p>Your process looks fine. I refer to <a href="https://discuss.pytorch.org/t/understanding-quantized-linear-layer/154000/3">this</a> for back engineering the QuantizedLinear OP.<br>
Only one point you may take care is that overflow can happen in</p>
<blockquote>
<p>matmul_out = np.matmul(x_q, fc_weight.T)</p>
</blockquote>
<p>Better to try the below again:</p>
<blockquote>
<p>matmul_out = np.matmul(x_q.astype(np.int32), fc_weight.T.astype(np.int32))</p>
</blockquote>

Francisco_Romaldo_M: <p>Check out this blog post, it explains it in detail</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://franciscormendes.github.io/2024/05/16/quantization-1/">
  <header class="source">
      <img src="https://franciscormendes.github.io/gallery/logo.svg" class="site-icon" width="150" height="150">

      <a href="https://franciscormendes.github.io/2024/05/16/quantization-1/" target="_blank" rel="noopener nofollow ugc" title="05:00AM - 16 May 2024">Hexo – 16 May 24</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="http://example.com/gallery/thumbnails/quantization.png" class="thumbnail">

<h3><a href="https://franciscormendes.github.io/2024/05/16/quantization-1/" target="_blank" rel="noopener nofollow ugc">A Manual Implementation of Quantization in PyTorch - Single Layer</a></h3>

  <p>IntroductionThe packaging of extremely complex techniques inside convenient wrappers in PyTorch often makes quick implementations fairly easy, it also removes the need to understand the inner workings</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Why should `at::cuda::philox::unpack(philox_args)` be called from kernels

sobha: <p>Looking at the random number generation API of pytorch, it seems that <code>at::cuda::philox::unpack(philox_args)</code> is only called from kernels (global or device functions), yet it is also a <code>__host__</code> function. Would it matter if it’s only called once on the host? If not, why is that? The function seems to be a pure function.</p>
<p>I’m not interested in potential performance gains by freeing a few bytes; rather, I’m trying to understand why.</p>
<p>is it because <code>*arg.seed_.ptr</code> can only be in the GPU? If so why is the function marked as <code>__host__</code>?</p>
Loss increasing when optim.Adam has maximize set to False

pranavm: <p>Hi all,<br>
I am currently trying to do a reconstruction task where I have a target structure and a randomly initialized structure, and these structures can be represented in many ways. For all but one representation, the reconstruction shows a decreasing loss as the initialized structure’s positions get closer to the target’s. However, for one representation the loss seems to be increasing, and the only way I got the loss to decrease was by setting <code>maximize=True</code> for optim.Adam. I am not sure how to go about debugging this as I have already tried other optimizers as well as manually doing gradient descent. Does anyone know what I could do to solve this problem?</p>
<p>Thanks</p>
TensorDataset with lazy loading?

Geremia: <p>I have hundreds of gigs of data in the form of hundreds of thousands of files to use for training. I cannot load it all at once with <code>torch.utils.data.TensorDataset()</code> because I don’t have enough RAM to hold it all at once. How do I “lazily load” data into a <code>TensorDataset</code>, such that it reads in the training files on an as-needed basis?</p>

JuanFMontesinos: <p>Use numpy memory map.</p>

Geremia: <p>That would work if I had one large file, but I have hundreds of thousands of small binary files that add up to 100s of gigabytes.</p>

JuanFMontesinos: <p>Yes, but you can construct this huge file or split it in several big files for convenience. Alternatively, you also have hfd5 format that allows lazy loading and carrying metadata.</p>
<p>Anyway, it seem your use case would be better solved with a custom dataset that loads each file on-the-fly.Have a look at:<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></p>
<p>So the short answer is:<br>
If huge dataset / array, use hdf5 or memory map.<br>
If hundreds of small files, use a custom dataset.</p>

Geremia: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="4" data-topic="204191">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>If hundreds of small files, use a custom dataset.</p>
</blockquote>
</aside>
<p>So basically overload <code>__getitem__</code>.<br>
I tried that, but I/O becomes a big bottleneck, unless I do some fancy caching or something.<br>
Perhaps HDF5 is the best way to go.</p>
CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect

Anirudh_Alameluvari: <p>I get this error when I try to run multiple models in parallel</p>
<pre><code class="lang-auto">net = get_model(model_path) # This is a basic resnet 18 with the final fc changed.
configs = get_configs(model_path) # Gets the hyperparams for the model.
prepped = before(image, perspective, configs["data"]["mean"], configs["data"]["std"]) # Preprocessing the input
with lock: # Having a lock here fixes the error
    with torch.no_grad():
        pred = (net(prepped) &gt; 0).type(torch.int).squeeze(0).cpu().tolist()
</code></pre>
<p>Without synchronization, I get the cuda illegal access error after some time. Never with the synchronization. I’m unable to make any sense of this.</p>

ptrblck: <p>Could you describe your use case in more detail and how you are parallelizing the run?</p>

Anirudh_Alameluvari: <pre><code class="lang-auto">with concurrent.futures.ThreadPoolExecutor() as executor:
        futures = {
            executor.submit(predict, image, perspective, model_path): key
            for key, model_path in models_map.items()
        }

        return {index for future, index in futures.items() if future.result()}
</code></pre>
<p>For more context, I am passing frames of a real time video through an ensemble of models, each model trained to predict one aspect of the frame. All the models are resnets. It doesn’t necessarily happen on the first frame. But after running this for a while on the video stream, I face this error which fails all subsequent GPU calls.</p>
Optim.Adam, LR in tensor

EGanji: <p>Which implementation support tensor for LR</p>
<ul>
<li><strong>lr</strong> (<a href="https://docs.python.org/3/library/functions.html#float" rel="noopener nofollow ugc"><em>float</em></a><em>,</em> <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>,</em> <em>optional</em>) – learning rate (default: 1e-3). A tensor LR is not yet supported for all our implementations. Please use a float LR if you are not also specifying fused=True or capturable=True.</li>
</ul>

EGanji: <p>No, really; I wanted to use tensor LR, but the documentation says</p>
<blockquote>
<p>A tensor LR is not yet supported for all our implementations.</p>
</blockquote>
<p>Then, I wondered if any implementation supports tensor LR?! If so, which one?</p>
Layer not present in model.parameters()

bdzyubak: <p>I am trying to freeze all layers except the lm_head of the t5-base model:</p>
<pre><code class="lang-auto">model = AutoModelForSeq2SeqLM.from_pretrained('t5-base').to('cuda')
model.lm_head
</code></pre>
<blockquote>
<p>Linear(in_features=768, out_features=32128, bias=False)</p>
</blockquote>
<p>But when running the standard freezing loop, I don’t see lm_head in model parameters:</p>
<pre><code class="lang-auto">for param_name, param in model.named_parameters():
        if startswith_list(param_name, layers_keep_training):
            param.requires_grad = True
            params_to_train.append(param_name)
        else:
            param.requires_grad = False
</code></pre>
<p>Is it named differently, or nested? How would I freeze all layers except that one? If I freeze all named_parameters, might I miss anything like I’m missing lm_head here?</p>

ptrblck: <p>Indeed the <code>lm_head</code> attribute is missing so it seems the parameter might be initialized as another attribute.<br>
Checking for the <code>id</code> shows this is the case:</p>
<pre data-code-wrap="python"><code class="lang-python">w_id = id(model.lm_head.weight)
for name, param in model.named_parameters():
    if w_id == id(param):
        print(name)
# shared.weight

print((model.lm_head.weight == model.shared.weight).all())
# tensor(True)
</code></pre>
How to run training on 3 GPUs with 2^n batch_size

anass_017: <p>I encounter errors when I attempt to use a batch size that is not divisible by 3. Is there an option that allows the last GPU to handle the remaining part of the batch?</p>
<p>Example</p>
<p>batch_size=32<br>
can be divided as</p>
<p>GPU1 : 10  (batch_size//3)<br>
GPU2 : 10 (batch_size//3)<br>
GPU3 : 12 (batch_size//3+r)</p>

ptrblck: <p>The <code>DistributedSampler</code> should take care of “padding” the dataset creating equal splits by repeating previous samples.</p>

anass_017: <p>thanks for your feedback</p>
Https://pytorch.org/docs/stable/distributed.html#which-backend-to-use

GuWei007: <p><a href="https://pytorch.org/docs/stable/distributed.html#which-backend-to-use" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/distributed.html#which-backend-to-use</a></p>
<p><strong>If you encounter any problem with NCCL, use Gloo as the fallback option</strong><br>
What mechanism is there in PyTorch to implement nccl fallback to gloo</p>

ptrblck: <p>You would need to specify it during the distributed initialization.</p>
How to properly lock a dataset when using multiple workers in dataloading?

jeremyfix: <p>Hello,</p>
<p>I’m facing an issue when writting my custom dataset object that I think is linked to the way multiprocessing is implemented with dataloading.</p>
<p>My use : my dataset object opens a binary file handler on a very large single file and every time I next to return the i-th sample, I move the file handler with seek, and read the corresponding number of bytes.</p>
<p>As the file handler is built in the constructor, my initial belief was that I needed to protect the “seek/read” part of my code with a multiprocessing Lock but I have the impression that I’m not doing it properly or that this is not the right way to do so ;</p>
<p>I could provide the code but it is kind of long and dependent on the structure of my binary data format but in pratice, the layout is the following :</p>
<pre><code class="lang-auto">import os
import struct
from multiprocessing import Lock

import torch.utils.data as data


class Dataset(data.Dataset):

    def __init__(self, filepath):
        super().__init__()
        self.fp = open(filepath, 'rb')
        self.fplock = Lock()
        self.row_format = '&lt;?ifff'
        self.row_size = struct.calcsize(self.row_format)

    def __getitem__(self, idx):
        file_offset = ..... # computed as a function of idx
        with self.fplock:
             self.fp.seek(file_offset, os.SEEK_SET)
             row = self.fp.read(self.row_size)
             values = struct.unpack(self.row_format, row)
       return values

    def __len__(self):
        return xxxxx 

dataset = Dataset()
loader = DataLoader(dataset, batch_size=xxxx, num_workers=7) 

for X in loader:
      ...

</code></pre>
<p>Isn’t the correct to proceed ?   When I set num_workers=1, the data are read correctly but If I set num_workers &gt; 1, I can see that my data are not always correctly decoded; My feeling is that the multiprocesses are interfering with the seek/read  although I protect the critical section with a multiprocessing Lock ;</p>
<p>Thank you for  your help ;</p>
<p>Jeremy./</p>

Pranit_Panda: <p>did you figure out a solution? I am having the same problem and am starting to suspect that Locks simply don’t work with torch</p>

jeremyfix: <p>Sorry for the late reply , I did not see your message.</p>
<p>Actually no. I have not found any solution.</p>
<p>The thing I had in mind and that I wanted to test was to open the file in the init_worker_fn .</p>
<p>Actually, it is while now but my idea was that every worker could have its own file handler rather than having one global that is locked; But I did not try that yet ;</p>
<p>Jeremy.</p>

Alvaro_Rodriguez: <p>I believe the issue might be that your start_method is spawn, and that you are passing a Dataset to each process. As the Lock is initialized within the process, each Dataset instance will have a different Lock.</p>
<p>The solution would be to define the Lock in main() and pass it to every process as an argument.<br>
Passing many Locks and shared values is a hassle, so I end up creating a mp.dict() that contains every shared structure.</p>
<p>Hope it helps future readers.</p>
Inconsistent behavior of .detach() on memory usage

Mandias: <p>While training a model I had assumed that I could use <code>.detach().numpy()</code> as a functionally equivalent call for <code>.item()</code> (for a single-value tensor), but found its behavior to be inconsistent.</p>
<p>Specifically, I was using it to store the mean loss during training iterations and found that when certain loss functions were used (BCELoss and MSELoss) it would result in constantly increasing memory usage. However, if other loss functions were used (such as a custom Dice loss function) there were no memory issues.</p>
<p>Changing the optimizer had no effect on the memory usage, nor did re-creating the loss function class each epoch (in case something was being stored there).</p>
<p>Naturally, the solution was to just use <code>.item()</code>, but it seems strange to me that there was an issue at all.</p>
<p>Am I wrong in expecting <code>.detach().numpy()</code> to work without issue for a single-value tensor?</p>
PyTorch and GPU compatibility: "Other" GPUs

vsant: <p>Hello,</p>
<p>I know that similar questions have already been asked here about this topic and “true” NVIDIA GPU’s compatibility with PyTorch is known (<a href="https://developer.nvidia.com/cuda-gpus" rel="noopener nofollow ugc">https://developer.nvidia.com/cuda-gpus</a>). The point is that it seems (I am not sure about this) that some manufacturers rely on NVIDIA architecture to create their “own” GPUs. See for instance:</p>
<ul>
<li>Galax GeForce RTX 4060 8GB DDR6 128 Bits (<a href="https://www.galax.com/pt/graphics-card/1clickoc-series/geforce-rtx-4060-oc-2x.html" rel="noopener nofollow ugc">https://www.galax.com/pt/graphics-card/1clickoc-series/geforce-rtx-4060-oc-2x.html</a>);</li>
<li>ASUS Dual GeForce RTX™ 4060 OC Edition 8GB GDDR6 (<a href="https://www.asus.com/motherboards-components/graphics-cards/dual/dual-rtx4060-o8g/" rel="noopener nofollow ugc">https://www.asus.com/motherboards-components/graphics-cards/dual/dual-rtx4060-o8g/</a>).</li>
</ul>
<p>Can we expect the same compatibility between PyTorch and these “other” GPUs? If yes, do we have the same compute capability considering these “other” GPUs and the “true” NVIDIA GPUs (in the case of the “true” NVIDIA RTX 4060 the compute capability is 8.9)?</p>
<p>Thank you.</p>

ptrblck: <p>The compute capability won’t change, i.e. a 4060 will have a compute capability of 8.9 as it won’t depend on the actual manufacturer.</p>

vsant: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. And to be sure, PyTorch is compatible with these other GPUs as well. Right?</p>

ptrblck: <p>Yes, all current PyTorch binaries support sm_89.</p>

vsant: <p>Thank you again <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
A trivial but annoying "bug" in random_split

fabin1790: <p>I’m researching about federated learning, and I’m splitting the dataset to simulate the scenario in which a few nodes form a federated learning network with their respective training data possessed.</p>
<p>So first I simulate 5 nodes with:<br>
trainsets = random_split(whole_trainset, [0.2, 0.2, 0.2, 0.2, 0.2], generator=generator)<br>
And 10 nodes with:<br>
trainsets = random_split(whole_trainset, [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1], generator=generator)</p>
<p>They all worked fined, but then I went to 20 nodes:<br>
ValueError: Sum of input lengths does not equal the length of the input dataset!<br>
print(torch.<strong>version</strong>) gives “2.2.1+cu118” so it’s not like I’m using too-old version.<br>
Then I realized that print(0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05+0.05) gives “1.0000000000000002” by plain Python in the first place.</p>
<p>I believed that it’s easy-to-fix (only this part in random_split, not why Python works like this I guess), so I decided to create an account and report it here.<br>
I also made an issue on Github.</p>

ptrblck: <p>Thanks for reporting the issue! Would you also be interested in fixing it? If so, let the code owners know in the GitHub issue you’ve created.</p>

fabin1790: <p>Sorry, I’m currently busy on the researches, so I can’t just jump into these open source, code review, pull requests, contribute, etc.<br>
But I believe something like this will work:</p>
<p>If the sum is very close to 1 (instead of equal to 1), then assume that it’s the “proportion case” and do some workaround.<br>
For example, calculate dataset_length*each_proportion and see if they sum up to dataset_length.<br>
If not, then as long as the sum of all the proportion numbers is very close to 1, I think there should still be feasible workaround. It’s probably even already implemented.<br>
For example:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import random_split
trainsets = random_split(range(11), [0.5, 0.5])
print(len(trainsets[0]))
print(len(trainsets[1]))
</code></pre>
<p>Gives 6 and 5.<br>
I’m not sure how the complete mechanism had actually been, but I think the same mechanism can be applied as long as sum([0.05, … , 0.05]) is very close to 1, or whatever <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.random_split" rel="noopener nofollow ugc">lengths list</a> that sums very closely to 1. (Probably need a few additional checking and workaround when their sum is not equal to 1 though.)</p>
<p>And that GitHub issue haven’t get a bug label or any reply. I guess the devs just won’t notice it that soon.</p>
Autograd of if else branching returns None instead of 0.0

edamondo: <p>I am new to pytorch and I wanted to understand a bit how torch.autograd.grad works. For this I wrote a small code but I don’t understand why it is not working.</p>
<p>import torch<br>
from torch.autograd import grad</p>
<pre><code class="lang-auto">import torch
from torch.autograd import grad
x = torch.tensor(-5.0, requires_grad=True)

def my_relu(z):
    if z &gt; 0.0:
        return z
    else:
        z = torch.tensor(0.0, required_grad=True)
        return z

y = my_relu(x)
h = grad(y, x, retain_graph=True)
</code></pre>
<p>I get</p>
<p><code>One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</code></p>
<p>but if I set this parameter <code>allow_unused=True</code> then I get <code>h=None</code> instead of <code>h=0.0</code> as the output. What is going on here?</p>

ptrblck: <p>You are never using the <code>z</code> input to <code>my_relu</code> in the <code>else</code> branch since you are creating a new leaf variable:</p>
<pre data-code-wrap="python"><code class="lang-python">else:
    z = torch.tensor(0.0, required_grad=True)
</code></pre>
<p>Here, <code>z</code> is a new tensor without any knowledge about the input <code>z</code> and is also not attached to it in any way.</p>

edamondo: <p>Oh I see. And is it possible to compute the gradient with respect to this new leaf?</p>

ptrblck: <p>Yes, but it would be <code>1.</code> since you are computing <code>h = grad(y, y, retain_graph=True)</code> as no operation was done on <code>z</code> (or <code>y</code> which is assigned to <code>z</code>).</p>
How to update the trainable group for a DDP wrapped model

Ledzy: <p>I am training a model with multiple modules (e.g. transformer layers), and I aim to switch the trainable layer during the training. For centralized training this is straightforward: I only need to change the attribute <code>required_grad</code> to determine which layer should be backward. However, when using DDP, this seems to be nontrivial, as the gradient needs to synchronized, and the synchronization is handled by the reduce hook registered when initializing the DDP model.</p>
<p>I found related github issues <a href="https://github.com/pytorch/pytorch/issues/21344" rel="noopener nofollow ugc">#21344</a> and <a href="https://github.com/pytorch/pytorch/pull/21591" rel="noopener nofollow ugc">#21591</a>. It suggests directly destroying and reconstructing a DDP model may be a viable approach, yet it seems not very elegant and a bit time consuming for my use case.</p>
<p>Could you provide me some information about how to switch trainable parameter for a DDP model? I guess removing reduce old hooks and re-add new ones may be a clean and feasible solution, but I’m not sure how to do that exactly. cc <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>Any help would be appreciated!</p>

agu: <p>Does setting <code>find_unused_parameters=True</code> work for you?</p>

Ledzy: <p>Thanks for your information.</p>
<p>Directly set <code>find_unused_parameters=True</code> and update the <code>requires_grad</code> field for each parameter switch doesn’t seem to work. From my understanding, the flag <code>find_unused_parameters=True</code> allows model to not synchonize some parts that are not involved in loss calculation. However, in my case, all the params are involved in loss calculation, and i only need to synchonize some of them (with <code>requires_grad</code>=True`).</p>
<p>One possible solution may be wrapping the whole model by DDP to assign grad reducer hook to every parameter, and only fire the hook when the parameter’s <code>requires_grad</code>=True. However, I don’t know how to implement this exactly. Is there any API that I can refer to?</p>
<p>Thanks!</p>
PyTorch CPP Installation

Karthi_Prasad_S: <p>I am facing issue while installing PyTorch for CPP, if possible can anyone help me to figure it out.</p>

ptrblck: <p>I’m unsure what your use case is, but you might want to download <code>LibTorch</code> from <a href="https://pytorch.org/get-started/locally/">here</a>, which does not ship with the Python API.</p>

Karthi_Prasad_S: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/188f5096532239d509a56c27946c44e2fcf84fb9.png" data-download-href="https://discuss.pytorch.org/uploads/default/188f5096532239d509a56c27946c44e2fcf84fb9" title="error"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/188f5096532239d509a56c27946c44e2fcf84fb9_2_690x461.png" alt="error" data-base62-sha1="3vgvBKsztyolCpFSvBsMAEcbzrX" width="690" height="461" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/188f5096532239d509a56c27946c44e2fcf84fb9_2_690x461.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/188f5096532239d509a56c27946c44e2fcf84fb9_2_1035x691.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/188f5096532239d509a56c27946c44e2fcf84fb9_2_1380x922.png 2x" data-dominant-color="3A3B35"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">1486×994 148 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have issue while executing the cmake file to run libtorch in cpp</p>
How to map GradBucket parameters and gradients to model parameters inside the hook?

ankit_b: <p>I want to manually update the gradients of a model from the comm_hook.</p>
<p>Inside the hook, we can access parameters and gradients. However, I don’t know how to map GradBucket parameters and gradients to the model version.</p>
<pre><code class="lang-auto">def comm_hook(state: ModuleNet, bucket: torch.distributed.GradBucket): -&gt; torch.futures.Future[torch.Tensor]:
    gradient_tensors = bucket.gradients()
    model_params = bucket.parameters()

   # Update ModuleNet model gradients from the GradBucket.

# Main Logic
model = ModuleNet()
ddp_model = DDP(model)
ddp_model.register_comm_hook(model, comm_hook)
output = ddp_model(input)
loss = ..
loss.backward()
optimizer.step()
</code></pre>

agu: <p>It looks like there is a <code>GradBucket.parameters</code> API that may be useful: <a href="https://pytorch.org/docs/stable/ddp_comm_hooks.html#torch.distributed.GradBucket.parameters" class="inline-onebox" rel="noopener nofollow ugc">DDP Communication Hooks — PyTorch 2.3 documentation</a></p>

caesar025: <p>Is there any update on this? I would also be interested in how to associate parameters in GradBuckets with layer types. The <code>GradBucket.parameters</code> unfortunately only provides the values of the parameters.</p>
Autocast keep cache across multiple forward pass

Thomas_Sajot: <p>The current implementation of autocast will drop the cache between consecutive forward passes while training (for example using pytorch lightning).<br>
This is because autocast should not be used during the backward pass (<a href="https://pytorch.org/docs/stable/amp.html#autocasting" rel="noopener nofollow ugc">autocasting doc</a></p>
<p>When all layers are unique in the model, there will be a call to <code>aten:to</code> / <code>aten:copy</code> on all weights at every foward pass.<br>
I was wondering if it would be possible to keep one cache instance during training.<br>
As the cache is a copy of the weights, I would expect the backward pass to not update these values directly, but would it be theoretically possible to support such use case?</p>
<p>The image is an example where the linear layer calls <code>aten:to</code> 3 times for its input, weight and bias.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5dc4969d8627e0b85a9d32daf83b0ab0100a9b0c.png" data-download-href="https://discuss.pytorch.org/uploads/default/5dc4969d8627e0b85a9d32daf83b0ab0100a9b0c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dc4969d8627e0b85a9d32daf83b0ab0100a9b0c_2_690x100.png" alt="image" data-base62-sha1="dnvAMLbgIFmTNEbxF92DyRMh4K0" width="690" height="100" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dc4969d8627e0b85a9d32daf83b0ab0100a9b0c_2_690x100.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dc4969d8627e0b85a9d32daf83b0ab0100a9b0c_2_1035x150.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5dc4969d8627e0b85a9d32daf83b0ab0100a9b0c_2_1380x200.png 2x" data-dominant-color="BBD3C5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1508×220 22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Query on PyTorch op decompositions?

teraba: <p>I see multiple op decompositions defined in PyTorch:</p>
<ul>
<li>torch/_decomp/decompositions.py</li>
<li>torch/_inductor/decomposition.py</li>
<li>torch/jit/_decompositions.py</li>
</ul>
<p>Few questions over this:</p>
<ul>
<li>What is the scope of these decomposition types?</li>
<li>Does these decomposition take place in all 3 modes in PyTorch (eager, compile and lazy)?</li>
<li>Are there any limitation in decomposing an op? like is the schema of the op updated if I use a custom decomposition to decompose it to a variant of the same op? (the variant may be a custom op or a non-core Aten op)</li>
</ul>
Does using a 'lite' model and 'optimized for mobile' change anything if used in native python code?

prashp: <p>After training a model, I have decided to serialize it using <code>torch.jit.trace</code> and then <code>optimize_for_mobile</code> and finally <code>_save_for_lite_interpreter</code>.</p>
<p>However my end goal is to utilize this model in another python program (using <code>torch.load()</code>) running on a much weaker PC, <strong>NOT</strong> on a mobile device or in a C++ program.</p>
<p>Is there any point in doing the serialization, optimization, and saving it to be compatible with the lite interpreter? All examples I’ve seen online only address targeting Android/iOS or C++ programs.</p>

prashp: <p>I’ve noticed I 3x inference speedup on a desktop intel cpu when using <code>optimize_for_mobile</code> and <code>_save_for_lite_interpreter</code> compared to using just the torchscript model. Is there any explanation what’s happening behind the scenes?</p>
<p>The docs also mention that torch.compile() is an option (with the openvino backend for example).</p>
Function to break tensor along dim

nlgranger: <p>There is a function in pytorch to break down a tensor into individual variables along a given dim. I can’t remember its name!<br>
Something equivalent to <code>x, y, z = a.moveaxis(d, 0)</code>.</p>

ptrblck: <p>You might be looking for <code>torch.chunk</code> or <code>torch.split</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(3, 2)
a, b, c = x.chunk(chunks=3, dim=-0)
print(a, b, c)
# tensor([[-1.4919,  2.4972]]) tensor([[ 0.4858, -1.2359]]) tensor([[0.4699, 0.3722]])
</code></pre>
Adam optimizer with softmax gives grad_fn=<SoftmaxBackward0>

undefined: <p>I’m using the Adam optimizer for backpropagation. In my last layer of my network I have the SoftMaxfunction. I’m using CrossEntropyLoss. I know that SoftMaxwith CrossEntropyLoss is not the best choise as CrossEntropyLoss already performs SoftMax, but in this case the model output gives values between 0 and 1.<br>
When feeding the network one sample, I get an output like this:</p>
<pre><code class="lang-auto">Model output: tensor([[1.0000e+00, 1.4013e-45]], device='cuda:0', grad_fn=&lt;SoftmaxBackward0&gt;)
</code></pre>
<p>But when I remove SoftMax from the last layer, I get:</p>
<pre><code class="lang-auto">Model output: tensor([[-5.9609,  4.6506]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>What I have seen is that without SoftMax training works faster and loss decreases to almost 0.</p>
<p>I would like to know if there is a difference between these two cases? Why is grad_fn different when I’m using Adam optimizer in both cases?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/undefined">@undefined</a>,</p>
<aside class="quote no-group" data-username="undefined" data-post="1" data-topic="204083">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/undefined/48/70420_2.png" class="avatar"> undefined:</div>
<blockquote>
<p>I would like to know if there is a difference between these two cases?</p>
</blockquote>
</aside>
<p>When using the SoftMax function, you’re predicting a class given an input (which for your model predicts the first class, out of 2 available classes).</p>
<aside class="quote no-group" data-username="undefined" data-post="1" data-topic="204083">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/undefined/48/70420_2.png" class="avatar"> undefined:</div>
<blockquote>
<pre><code class="lang-auto">Model output: tensor([[1.0000e+00, 1.4013e-45]], device='cuda:0', grad_fn=&lt;SoftmaxBackward0&gt;)
</code></pre>
</blockquote>
</aside>
<p>In the second case, you’re just outputting the logits and if you fit that to your model, you’re no longer predicting classes based on a probability, but just fitting the output (which isn’t the same).</p>
<p>If you want to read more info, there’s a nice thread with more information here: <a href="https://discuss.pytorch.org/t/logits-vs-log-softmax/95979/2" class="inline-onebox">Logits vs. log-softmax - #2 by KFrank</a></p>

undefined: <p>As I understood it correct using SoftMax with CrossEntropyLoss is bad practice. Thats why I wont use SoftMax. Another thing I’m curious about is which loss function to use as I have 2 classes? CrossEntropyLoss or BCEWithLogitsLoss?</p>
Multithreading num threads priority in Pytorch

Shreya: <p>Hi,<br>
I’m testing optimisations offered by PyTorch (cpu x86 only). I have set environmental variable OMP_NUM_THREADS = 1 and  MKL_NUM_THREADS = 1.</p>
<ol>
<li>Are there any chances of multi-threading (inter ops or/and intra ops)?</li>
<li>Can pytorch override OMP_NUM_THREADS, MKL_NUM_THREADS values based on cpu info to set optimal threads for better performance?</li>
<li>When MKL_NUM_THREADS = 1 then, can MKL or MKL_DNN routines use more than 1 thread based on cpu info?</li>
</ol>
"Model \"XYZ\" has no worker to serve inference request. Please use scale workers API to add workers."

lsmith77: <p>In my config.properties I have “default_workers_per_model=1”.<br>
There is no traffic on this server except for me manually making a single call.</p>
<p>Why is no worker launched/available?</p>

lsmith77: <p>Ah figured out the issue</p>
<p><code>WARNING: The directory '/app/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device: '/tmp/pip-target-ipboar3k/lib/python/cmake/data/share/cmake-3.26/Help/prop_tgt/VS_DOTNET_REFERENCES.rst'</code></p>
<p>Obviously a bit misleading error message.</p>

Yoann_Buzenet: <p>Where did you find this error?</p>
<p>update : For me it was in the docker logs ${container} that I saw it</p>
Skipping some data points in convolution

Vaibhav_Chahar: <p>Hi,<br>
I have a bit tricky question. I am training a PixelCNN network that I am working on. In my data, there are some points that I don’t want my CNN to take in account. First I applied a Mask on my data and it works for some points but for others it is not working. Then I found the problem, at those points, when the kernel (3 X 3) convoluted, it is taking the account of the data points which are unmasked or necessary surrounding points. For example in the data below, for the masked point “A” the kernel will take account for two 1s in the last row of first matrix.</p>
<pre><code>      [[[[ 0.,  0.,  0.,  0.],
      [ A.,  0.,  0.,  0.],
      [ 0.,  0.,  0.,  0.],
      [ 0.,  1.,  1.,  1.]],

     [[ 0.,  0.,  0.,  0.],
      [-1.,  1., -1., -1.],
      [-1.,  1.,  1.,  1.],
      [ 1., -1.,  1.,  0.]]]], device='cuda:1')
</code></pre>
<p>My question is, is there anything that I can do, so that my kernel will skip these points (for example “A”, or the 0s in above dataset) or I can do something else? Masking them won’t help and I can’t remove these 0s from my dataset.</p>
<p>Thank you</p>
Node Meta Data not propagated to replaced nodes while using fx subgraph rewrite?

yshnsh: <p>I am trying to write custom subgraph rewrite using torch fx subgraph_rewriter. I see that after the replacement the added nodes do not contain “meta” information. Is there any way to capture these during the pattern matching and replace?</p>

cokespace2: <p>you mean “meta” info by modules’s attributes? Like Conv2d’s kernel size, strides, padding etc.</p>

trusira: <p>Have you looked at <a href="https://github.com/pytorch/pytorch/blob/30788739f4cec672cdabc21c155bbe30ac5d66e6/torch/fx/traceback.py#L16" rel="noopener nofollow ugc"><code>torch.fx.traceback.preserve_node_meta()</code></a>?</p>
Load models directly into CUDA?

liwan: <p>Hello PyTorch Community,</p>
<p>I’m looking to optimize memory usage when loading PyTorch models onto CUDA. Typically, we load the model onto CPU memory first and then move it to CUDA, which can consume significant memory during the loading process.</p>
<p>Is there a method to load the model directly onto CUDA, bypassing CPU memory? Alternatively, what are the best practices to minimize memory usage when loading a model onto CUDA?</p>
<p>Thank you for your insights and guidance.</p>

ptrblck: <p>We are working on an RFC to use GPUDirectStorage allowing tensors to bypass the host and to be loaded directly into the device. I’m unsure where this RFC is stuck, but the last time I checked <a class="mention" href="/u/mikaylagawarecki">@mikaylagawarecki</a> was working on the serialization part.</p>

liwan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a lot for the information. Glad to see someone is actively working on this!</p>
Using custom data type, instead of double, with a PyTorch model in C++

aevs: <p>I’m trying to understand if it’s possible to use a custom data type with a PyTorch model that has been trained in Python then exported to C++. I’ve got my model all up and running in C++ when the input data comes from a standard double array, like so:</p>
<pre><code class="lang-auto">double sim_state[8]; // this gets populated with some data

std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(torch::from_blob(sim_state, {1, 8}, at::dtype(at::kDouble)));
at::Tensor prediction = module.forward(inputs).toTensor();
</code></pre>
<p>Now, I want to use a different data type instead of kDouble. My data type is called DACE::DA from the dace.h header, and essentially it’s a polynomial instead of a number. The DACE::DA type comes with all the required functionality for addition, subtraction, multiplication, etc., and the inputs can be given as</p>
<pre><code class="lang-auto">DACE::AlgebraicVector&lt;DACE::DA&gt; sim_state(8); // this gets populated with some data
</code></pre>
<p>Of course, the <code>torch::from_blob</code> fails even before getting to the <code>model.forward</code> call, as expected. Is there a way to use custom data types like this with my PyTorch model? I should also note that I do not want to convert my DACE::DA to double, input it into the model, get the double outputs and then convert back to DACE::DA. I need to be able to propagate my DACE::DA type through my model.</p>
<p>Thanks in advance for any feedback on this!</p>

aevs: <p>bumping as I still am unable to find a solution to this</p>
Best way to customize torchvision faster rcnn loss

mg.group: <p>Hi all,<br>
I would be grateful for any suggestion on this issue.</p>
<p>I need to add some (possibly complex) component to the loss for my object detection. I am using torchvision fasterrcnn_resnet50_fpn.</p>
<p>From some old post it was suggested to assign the new loss to some member variable of the RoIHeads but looking into the code I can see that the loss is defined here:<br>
<a href="https://github.com/pytorch/vision/blob/3fb88b3ef1ee8107df74ca776cb57931fe3e9e1e/torchvision/models/detection/roi_heads.py#L12" rel="noopener nofollow ugc">Loss</a><br>
at and then directly invoked here:<br>
<a href="https://github.com/pytorch/vision/blob/3fb88b3ef1ee8107df74ca776cb57931fe3e9e1e/torchvision/models/detection/roi_heads.py#L772" rel="noopener nofollow ugc">Invoking Loss</a></p>
<p>Is there any way to replace the original loss with mine without changing the source code? For example, if subclassing of RoIHeads can do the job, how to make it minimal?</p>
<p>Thanks in advance for your usual responsiveness</p>

morepenguins: <p>I have the same question, but no answer. Let me know if you figured it out!</p>
RuntimeError: expected stride to be a single integer value or a list of 4 values to match the convolution dimensions, but got stride=[2, 2, 2]

HamidGadirov: <p>I am getting this error after I implemented hypernetwork to learn weights for the CNN. This is e.g. how I define the weights:<br>
<code>'conv0_0_weight': torch.Size([c//2, in_planes, 4, 4, 4])</code></p>
<p>This is how it is used:<br>
<code>x = functional_conv3d(x, weights['conv0_0_weight'], weights['conv0_0_bias'], self.c//2, stride=2, padding=1)</code></p>
<p>and this is the functional conv3d:</p>
<pre><code class="lang-auto">def functional_conv3d(x, weight, bias, out_planes, stride=1, padding=1, dilation=1):
    x = F.conv3d(x, weight, bias, stride=stride, padding=padding, dilation=dilation)
    x = nn.PReLU(num_parameters=out_planes)(x)
    return x
</code></pre>
<p>I printed the shapes of the input tensor and weights:</p>
<pre><code class="lang-auto">  # input x: torch.Size([2, 3, 128, 128, 128])  
  # conv0_0_weight : torch.Size([2, 128, 3, 4, 4, 4])
</code></pre>
<p>The first value (of 2) is the batch size. I checked the documentation and c//2, in_planes, 4, 4, 4 in the weight shape initialization are according to out_channels, in_channels, kT, kH, kW. Is there something I am missing?</p>

ptrblck: <aside class="quote no-group" data-username="HamidGadirov" data-post="1" data-topic="203262">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamidgadirov/48/70216_2.png" class="avatar"> HamidGadirov:</div>
<blockquote>
<p>I checked the documentation and c//2, in_planes, 4, 4, 4 in the weight shape initialization are according to out_channels, in_channels, kT, kH, kW.</p>
</blockquote>
</aside>
<p>The <code>in_channels</code> (128 in your example) do not match the channel dimension of <code>x</code> (3 in your example).<br>
Also, you are using an additional dimension with the size of 3 in your conv weigh, so remove it or drop the first dimension in case it’s supposed to represent the batch dimension, since parameters do not use a batch dim.</p>

HamidGadirov: <p>Thanks for your reply. The problem was related to the fact that the batch size dimension was adding another dimension to the data, since I was using a functional model, causing inconsistency of the dimensions for the conv layers. I added iteration over each element in the batch in the <code>forward()</code>, and after that the issue was resolved.</p>
Using torch rpc with a function defined remotely

priyathamkat: <p>I would like to use rpc to run a function remotely. All the examples in the documentation have a method that is defined with in the same script or is from torch. I was wondering if its possible to use <code>rpc.rpc_sync</code> with some sort of a proxy <code>func</code>. For example, with <code>xmlrpc</code> it is possible to do the following:</p>
<pre data-code-wrap="python"><code class="lang-python">import xmlrpc.client

with xmlrpc.client.ServerProxy("http://localhost:8000/") as proxy:
    print("3 is even: %s" % str(proxy.is_even(3)))
    print("100 is even: %s" % str(proxy.is_even(100)))
</code></pre>
<p>Note that <code>is_even</code> is not defined in this client script.</p>

H-Huang: <p>The function needs to be defined in both scripts. The function name and arguments are pickled and then communicated across ranks, so the each rank needs to have the same function definition.</p>
Time for data going through model

Albertma: <p>Hi, I found an interesting thing, and I am not sure whether it is my mistake or not.<br>
When training a model, in one epoch for many mini-batches, the time for images going through model of the first mini-batch is 2.54s, and then 0s for the next 10 mini-batches(i did not see the time for all 851 mini-batches, only 10). Please see the code:</p>
<pre><code>    for batch_idx, (imgs, ...) in enumerate(train_loader):
        optimizer.zero_grad()
        image = imgs.to(device)
        time1_start = time.time()
        x, y= model(image)
        time1_end = time.time()
        print('time1：%.2f' % (time1_end - time1_start))
        # the time for printing is 2.54s, 0.00, 0.00, 0.00, ......
</code></pre>
<p>So it means after the first mini-batch, the time for images going through model is nearly 0.<br>
As a student, I am curious and want to learn the reason.<br>
Thanks.</p>

ptrblck: <p>Your profiling is most likely wrong. E.g. if you are using a GPU, you would need to synchronize the code before stopping and starting host timers since CUDA operations are executed asynchronously.</p>