akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the difference between the docker images of the same tag in ghcr.io and docker.io

cdzhan: <p>such as <a href="https://github.com/orgs/pytorch/packages/container/pytorch/159656730?tag=2.1.2-cuda12.1-cudnn8-devel" class="inline-onebox" rel="noopener nofollow ugc">Package pytorch · GitHub</a> and <a href="https://hub.docker.com/layers/pytorch/pytorch/2.1.2-cuda12.1-cudnn8-devel/images/sha256-a5de097b482f5927baf2322f4419f11044bfe4f08c7b7593dbaff8e41d03a204?context=explore" class="inline-onebox" rel="noopener nofollow ugc">Docker</a></p>

cdzhan: <p><a class="mention" href="/u/malfet">@malfet</a> Do you know their difference?</p>

marksaroufim: <p>They’re the same and mirror each other, it’s quite common to mirror binaries since it’s not ideal to rely on a single vendor</p>

cdzhan: <p>Thank you for your reply! It make sense. But why are the Digest sha256 different?</p>

marksaroufim: <p>IIUC there are no guarantees that SHA256 will be consistent across vendors but <a class="mention" href="/u/malfet">@malfet</a> can correct me if I’m wrong</p>
Failed to load image Python extension: libc10_cuda.so:

Raughl1: <p>I don’t even know what to think anymore, I have an RX 6600, six months ago there was a torch 1.13.1, I didn’t know the troubles. Now I’ve been tormented with all possible combinations of pytorch, and so far the closest I’ve come to launching stable diffusion is rocm 5.6. But when it starts, it turns out that it doesn’t see any library libc10_cuda.so</p>
<pre><code class="lang-auto">/home/jorj/stable/stable-diffusion-webui/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libc10_cuda.so: cannot open shared object file: No such file or directory
  warn(f"Failed to load image Python extension: {e}")
/home/jorj/stable/stable-diffusion-webui/torchvision/__init__.py:23: UserWarning: You are importing torchvision within its own root folder (/home/jorj/stable/stable-diffusion-webui). This is not expected to work and may give errors. Please exit the torchvision project source and relaunch your python interpreter.
</code></pre>
<p>How do I resolve this error?</p>

marksaroufim: <p>Those look like just warnings though? I assume the program is still running just fine</p>
Can the torchserve be regarded as a general MLServe platform?

Wen_Flybywind: <p>From the document of torchserve, I think it can support any kind of ML architecture, not limited to only pytorch, am I right ?<br>
So has any one tried to deploy, say a tensorflow model, on torchserve ? Any experience to share ?<br>
Will be very glad if u can give me some suggestions!</p>

Nanlalalalalal: <p>Try use onnx and Triton Inference Server， for high performan serving. Also You could try <a href="https://github.com/torchpipe/torchpipe" rel="noopener nofollow ugc">torchpipe</a> for pytorch interface and onnx/tensorrt backends</p>

marksaroufim: <p>Indeed torchserve is a general framework it works by launching python processes and those processes can run anything</p>
<p>There’s first class support for ONNX <a href="https://github.com/pytorch/serve/blob/master/test/pytest/test_onnx.py" class="inline-onebox">serve/test/pytest/test_onnx.py at master · pytorch/serve · GitHub</a> and Tensorflow can be supported easily</p>
Cmake picking the wrong compiler version

Avishka_Perera: <p>I am building PyTorch from source.</p>
<p>I am using Ubuntu 20.04. I have multiple gcc versions installed. I have set the symlinks for “/usr/bin/gcc” and “/usr/bin/g++” so that they will point to “/usr/bin/x86_64-linux-gnu-gcc-9” and “/usr/bin/x86_64-linux-gnu-g+±9”. Earlier it was 7.</p>
<p>So now I get the following outputs for the following commands</p>
<ul>
<li><code>gcc --version</code>:</li>
</ul>
<pre><code class="lang-auto">gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
</code></pre>
<ul>
<li><code>g++ --version</code>:</li>
</ul>
<pre><code class="lang-auto">g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
</code></pre>
<ul>
<li><code>x86_64-linux-gnu-g++ --version</code>:</li>
</ul>
<pre><code class="lang-auto">x86_64-linux-gnu-g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
</code></pre>
<ul>
<li><code>x86_64-linux-gnu-gcc --version</code>:</li>
</ul>
<pre><code class="lang-auto">x86_64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
</code></pre>
<p>But when I build pytorch with the command <code>python setup.py develop</code>, Cmake picks the old gcc version</p>
<pre><code class="lang-auto">Building wheel torch-2.3.0a0+gitc378001
-- Building version 2.3.0a0+gitc378001
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/avishka/temp/pytorch/torch -DCMAKE_PREFIX_PATH=/home/avishka/anaconda3/envs/ie2/lib/python3.10/site-packages;/home/avishka/anaconda3/envs/ie2 -DNUMPY_INCLUDE_DIR=/home/avishka/anaconda3/envs/ie2/lib/python3.10/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/home/avishka/anaconda3/envs/ie2/bin/python -DPYTHON_INCLUDE_DIR=/home/avishka/anaconda3/envs/ie2/include/python3.10 -DPYTHON_LIBRARY=/home/avishka/anaconda3/envs/ie2/lib/libpython3.10.a -DTORCH_BUILD_VERSION=2.3.0a0+gitc378001 -DUSE_NUMPY=True /home/avishka/temp/pytorch
FATALGCC-9.4 or newer is required to compile PyTorch, but found 7.5.0
-- /usr/bin/c++ /home/avishka/temp/pytorch/torch/abi-check.cpp -o /home/avishka/temp/pytorch/build/abi-check
...
</code></pre>
<p>Any help is highly appreciated.</p>

ptrblck: <p>You could try to export <code>CC</code> and <code>CXX</code> pointing to <code>gcc</code> and <code>g++</code>, respectively, to make sure <code>cmake</code> picks the right compilers.</p>
Android libTorch

drus13: <p>Hello everyone! I have some questions accordingly pytorch repository, I trying to build libtorch for Android. I would like to know if you have recently been able to build the library and your source code with the model using the provided scripts in the repository? I want to learn how to train a model using a simple MNIST training problem using libtorch on Android, but I encounter numerous errors when trying to build a library or compile my source code. Maybe some of you guys will have time and could help me, I would be very grateful.</p>

marksaroufim: <p>I don’t believe libtorch for android is maintained at this point, the new recommendation for mobile builds is to use executorch <a href="https://github.com/pytorch/executorch" class="inline-onebox">GitHub - pytorch/executorch: End-to-end solution for enabling on-device AI across mobile and edge devices for PyTorch models</a></p>

drus13: <p>Is it possible to train model with that new framework?</p>

marksaroufim: <p>No executorch is primarily intended for inference</p>

drus13: <p>Looks like it. So regarding libtorch, can it be used for simple learning on android? I think I saw several threads on this forum and someone was trying to put together a library for Android? Maybe i need to use older versions for this if the current version is not built?</p>
Issues with model behaviour when using .eval() mode

oppiliF: <p>I need some help understanding a strange issue I’ve encountered with the ‘.eval()’ mode for PyTorch models. When I try some pretrained models, they do not perform as intended when in ‘.eval()’ mode. I have tested pretrained models from different repositories: MaLP (Github: vishal3477/pro_loc), Stargan (Github: yunjey/stargan) and GDWCT (Github: WonwoongCho/GDWCT).</p>
<p>Unfortunately, all models exhibit unexpected behavior when switched to evaluation mode, and I’m struggling to identify the root cause.</p>
<p>Other users have reported a similar problem in <a href="https://github.com/yunjey/stargan/issues/102" rel="noopener nofollow ugc">this GitHub issue</a>, but no one has been able to find a fix.</p>
<p>I would greatly appreciate any insights or suggestions.</p>
<p>-–</p>
<p>Python version: 3.7<br>
Pytorch version: 1.13.1</p>
How are pytorch tensors stored internally?

raunaks: <p>Hi,<br>
I was trying to find out more about how the pytorch tensors are encoded in the source code, but I can’t seem to find a concrete answer.<br>
Could someone please point me to the relevant source code in C++ or Python? i.e. what happens when we run torch.tensor([1,2,3]), at a low level? Which classes are used to represent the object(s)?</p>
<p>Additionally, is there a way to get the bit level representation of the tensor as it is stored in memory?</p>

ptrblck: <p><a href="http://blog.ezyang.com/2019/05/pytorch-internals/">This blog post</a> explains the internals as well as the tensor representation and might be helpful.</p>
Source code structure of pytorch

Andy_P: <p>Dear community,<br>
I want to systematically learn the source code structure of pytorch.<br>
How should I learn? Is there any official document where I can learn from it?<br>
Thanks.</p>

marksaroufim: <p>I’d suggest picking a specific subset you’re interested in and going from there pytorch is massive codebase</p>
<p>That said if you want a high level overview of all the different main components this talk is great <a href="https://x.com/tarantulae/status/1734192354716725401?s=20">https://x.com/tarantulae/status/1734192354716725401?s=20</a></p>

Andy_P: <p>Thank you for your reply, these documents are very helpful to me.<br>
Besides this, I mainly want to learn about the subset of PyTorch related to CUDA, especially the aspects related to GPU memory allocation and calls.<br>
Do you have any suggestions?<br>
Thanks.</p>

marksaroufim: <p>Zach De Vito’s blog is a fantastic resource <a href="https://zdevito.github.io/">https://zdevito.github.io/</a></p>

Andy_P: <p>Thank you very much.This helped me a lot.</p>
The CUDA Graph is empty

dkz: <pre><code class="lang-auto">temp = x[b // 2:].clone()
mask = mask.permute(0, 3, 1, 2)
x[b // 2:][mask[b // 2:]] = x[:b // 2][mask[:b // 2]]
x[:b // 2][mask[:b // 2]] = temp[mask[b // 2:]]
</code></pre>
<p>type of x is float and type of mask is bool<br>
When I add the code to the forward function of the model, it will display</p>
<blockquote>
<p>‘UserWarning: The CUDA Graph is empty.’ This usually means that the graph was attacked to be captured on wrong device or stream The error ‘Triggered internally at…/ten/src/Aten/cuda/CUDAGraph. cpp: 224.’</p>
</blockquote>
<p>After removing this code, it will not appear, and this error will not occur without using ‘torch. compile’.<br>
Why did this situation occur? Is it because boolean indexes cannot appear in the forward when using compile?</p>

marksaroufim: <p>Are you using nightlies by any chance? Also cc <a class="mention" href="/u/eellison">@eellison</a></p>

dkz: <p>The version is stable(2.1.2). I solved this by adding ‘torch.compiler.disable’. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
RuntimeError: torch.linalg.lu_factor: (Batch element 1): U[4,4] is zero and using it on lu_solve would result in a division by zero. If you still want to perform the factorization, consider calling linalg.lu(A, pivot) or linalg.lu_factor_ex(A, pivot)

shaoyu_young: <pre><code class="lang-auto">RuntimeError: torch.linalg.lu_factor: (Batch element 1): U[4,4] is zero and using it on lu_solve would result in a division by zero. If you still want to perform the factorization, consider calling linalg.lu(A, pivot) or linalg.lu_factor_ex(A, pivot)
</code></pre>
<p>I’m sorry to bother you, but I feel confused about this error. I want to confirm that this means not a bug about <code>torch.linalg.lu_factor</code> but I need to use another API (i.e., linalg.lu)</p>
bin/python: relocation error:  /lib/python3.11/site-packages/torch/lib/libcudnn_cnn_infer.so.8: symbol _ZN15TracebackLoggerC1EPKc version libcudnn_ops_infer.so.8 not defined in file libcudnn_ops_infer.so.8 with link time reference

Asebe_Teka: <p>/media/server/ce553280-8b4f-4d10-aa2b-1a5d09580e6a/Asebe/anaconda3/envs/AMT/bin/python: relocation error: /media/server/ce553280-8b4f-4d10-aa2b-1a5d09580e6a/Asebe/anaconda3/envs/AMT/lib/python3.11/site-packages/torch/lib/libcudnn_cnn_infer.so.8: symbol _ZN15TracebackLoggerC1EPKc version libcudnn_ops_infer.so.8 not defined in file libcudnn_ops_infer.so.8 with link time reference</p>

xgbj: <p>It looks like a symbol issue during compilation. You can try clearing the compilation cache and recompiling it.</p>
How to only make a change to a submodule and then update Pytorch

chanaka: <p>I’m making modifications to third party submodule (Gloo <code>third_party/gloo</code>) to accommodate my use cases and I want to test them through PyTorch without building it from scratch. Is there a way to rebuild PyTorch and incorporate these changes without going through the whole build process again as I only changed the Gloo source? Cutting down time is the main target here. Step by step guide to just build a submodule and incorporate it into the already built PyTorch would be most appreciated.</p>

ptrblck: <p>Only the initial source build would need to rebuild the entire framework. Changes to local files would then trigger incremental builds for the changed files. However, I’m not sure how third party dependencies are handled and if changes there would be detected properly, so you should try it out and report here how changes in <code>gloo</code> are treated.</p>

chanaka: <p>Thank you! I’ll double check and report back.</p>

chanaka: <p>Confirm that even for the third-party repo (<code>Gloo</code> in this case) only the changes were compiled when building.</p>
When using torch.distributed Point-to-point communication,is there any way to handle error by myself

Jimmy_Yu: <p>Can I implement such a function ,</p>
<p>In send side :<br>
call dist.isend() and do not care about either it is success or failed . and do following jobs.<br>
In recv side :<br>
call dist.irecv() and wait for some seconds, handling errors or timeout by myself without blocking the program.</p>
<p>I tryed to make a simple example :</p>
<pre><code class="lang-auto">dist.init_process_group("nccl",world_size=ompi_world_size,rank=ompi_world_rank,
                        init_method='file:///my_path/dist_file',timeout=timeout)
if not dist.is_initialized():
   raise RuntimeError("init fail") 
rank = dist.get_rank()
world_size = dist.get_world_size()
if not torch.cuda.is_available():
    raise RuntimeError("CUDA is not available")
torch.set_default_device('cuda:0')

vec = torch.rand(32)
res = None
success = False

if rank == 2:
    time.sleep(100)
else :
    if rank %2 == 0:
        dist.isend(vec,rank+1)
    else:
        res = dist.irecv(vec,rank-1)
    
    if rank % 2 != 0:
        for l in range(5):
            if res.is_success():
                success = True
            time.sleep(1)

print(f"result: {success}, {vec} in {rank} after send/recv")

dist.destroy_process_group()
</code></pre>
<p>asuume that something wrong in rank 2 , rank 3 wait for 5s and give up.<br>
but in result , rank 2 and 3 are both blocking util forever.</p>

yf225: <p>Repro: <a href="https://gist.github.com/yf225/bfb8052a9892b0e914bcd4df615ca000" class="inline-onebox" rel="noopener nofollow ugc">test_ptd_forum_195413.py · GitHub</a></p>
<p>cc. <a class="mention" href="/u/h-huang">@H-Huang</a> <a class="mention" href="/u/kwen2501">@kwen2501</a></p>
Get per GPU gradients for DDP model

Veydan_Wu: <p>Hi, for models wrapped with DDP, I suspect gradients across all GPUs are accumulated automatically after the <code>backward()</code> call and before the <code>optimizer.step()</code> call.</p>
<p>How do I get the per device gradients before gradients are synchronized across devices?</p>
<p>Say there are N GPUs, each of these N copies of model run</p>
<pre><code class="lang-auto">model_out = ddp_model(model_input)
loss = loss_fn(model_out)
loss.backward() 
</code></pre>
<p>before calling</p>
<pre><code class="lang-auto">optimizer.step()
</code></pre>
<p>is there a way I get the per GPU gradients via something like</p>
<pre><code class="lang-auto">ddp_model[rank_id].parameters()[0].grad
</code></pre>
<p>before the gradients are synchronized? Thanks.</p>

yf225: <p>DDP comm hook should be useful for implementing what you need: <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.register_comm_hook" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch 2.1 documentation</a></p>
Accelerate model on Hexagon (Snapdragon accelerator)

bwau: <p>This is a slightly more general question, but I figured this forum is likely where I’d find the developers who could give me a hand.</p>
<p>I’m looking to deploy a slightly modified int8 quantized MobileNetV3 model to Android devices. My goal is to run accelerated on the hexagon accelerators that are present on Snapdragon chips. I’ve made initial tests with pytorch-mobile-lite and NNAPI, which works well for pixel phones (Tensor G2 chip), but doesn’t run accelerated on Samsung Galaxy S21(Snapdragon 888).</p>
<p>I’ve dug around in various tflite, pytorch-lite and onnx tutorials, as well as the documentation for NNAPI and Qualcomm Neural Processing SDK. I’ve failed to find a way to deploy which will run accelerated on at least the last few generations of Snapdragon devices. It seems that each framework only supports a subset of chips, either 855 and earlier(tf-lite delegate for instance), or 888 and later. What are current best practices for running models accelerated on Android? Is there a way to avoid building many separate deploy paths for various accelerators?</p>

manuelcandales: <p>MV3 is supported by ExecuTorch (with XNNPack and HTP). ExecuTorch is our new end-to-end solution for enabling on-device AI across mobile and edge devices. The documentation is here, including tutorials: <a href="https://pytorch.org/executorch" rel="noopener nofollow ugc">https://pytorch.org/executorch </a>.</p>
<p>Please use <a href="https://github.com/pytorch/executorch/issues" rel="noopener nofollow ugc">Issues · pytorch/executorch · GitHub </a> to report bugs, request for comments, enhancements and new features.</p>
Invoking Time of nn.Module _register_state_dict_hook()

Jiajun_Zha: <p>Below is the source code of _register_state_dict_hook I quoted from official Pytorch nn.Module</p>
<pre><code class="lang-auto">    def _register_state_dict_hook(self, hook):
        r"""These hooks will be called with arguments: `self`, `state_dict`,
        `prefix`, `local_metadata`, after the `state_dict` of `self` is set.
        Note that only parameters and buffers of `self` or its children are
        guaranteed to exist in `state_dict`. The hooks may modify `state_dict`
        inplace or return a new one.
        """
        handle = hooks.RemovableHandle(self._state_dict_hooks)
        self._state_dict_hooks[handle.id] = hook
        return handle
</code></pre>
<p>The comment says that “These hooks will be called … after the “state_dict” of self is set”.<br>
While I’m not sure when exactly the “state_dict” of a nn.Module is set. Is it set after initialization? Or every time after the weight get updated?</p>

albanD: <p>Hi,</p>
<p>This is linked to when the <code>load_state_dict()</code> function is called.<br>
Note that this function is internal and might change without notice in the future.</p>

Jiajun_Zha: <p>Aha, I see.<br>
So the same goes for nn.Module _reister_state_dict_pre_hook(), right?</p>

albanD: <p>Yes this is the same idea</p>

ecolss: <p>Hi <a class="mention" href="/u/alband">@albanD</a> , a simple followup on this thread, I want to filter out some parameters before save model checkpoints, and I thought I could use <code>_register_state_dict_hook</code> to modify <code>state_dict</code> (or the <code>destination</code> dict), but now I’m not sure if this is the best way in pytorch, because you said this function is internal and may change.</p>
<p>So what’s the proper way to change state_dict and filter out some params (don’t save them into model ckpt files)?</p>

albanD: <p>Hey!</p>
<p>The full proposal to fix this is at <a href="https://github.com/pytorch/pytorch/issues/75287" class="inline-onebox">[RFC] Consolidated and unified state_dict and load_state_dict hooks · Issue #75287 · pytorch/pytorch · GitHub</a><br>
Contributions adding the missing pieces are welcomed!</p>
Gather-and-reduce in one operation

ajk: <p>Hi,</p>
<p>In my use-cases a very common pattern is to use gather and then reduce one dimensions.</p>
<p>Example:</p>
<pre><code class="lang-auto">B = 4
D = 16
W = 128
N = 1000
sketch = torch.randn(B, D, W)
codes = torch.randint(W, [B, D, N])
scores = sketch.gather(dim=-1, index=codes).min(dim=-2).values
</code></pre>
<p>Unfortunately it means I am creating intermediate value after-gather and before-min that scales memory-wise linearly with D and N dimensions, even though D dimensions is immediately reduced.</p>
<p>I can do loop-over-D-dimension and reduce into accumulator to avoid unnecessary memory explosion:</p>
<pre><code class="lang-auto">scores = torch.zeros(B, N)
for i in range(D):
    scores = torch.minimum(scores, sketch[:, i].gather(dim=-1, index=codes[:, i]))
</code></pre>
<p>but unfortunately it gets extremely slow.</p>
<p>Is it possible to somehow get best-of-two worlds? It seems like it should be doable with some kind of native gather-and-reduce operation.</p>
<p>To best of my knowledge it would require having custom native operation. Is that correct? If so could you share some links/snippets/code references that might be a good starting point if one would want to implement such native operation?</p>

cwindolf: <p>Totally agree. <code>torch.gather_reduce</code> would be awesome to have.</p>
Nevergrad float32 training

TWTom: <p>I am trying to use Nevergrad for derivative-free method to train a neural network. It seems that when I use the optimization, it will auto cast the model to float64, but I want it to stay at float32 (or float16), how would I do that?</p>
<p>Here’s the test snippet:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/11uPYA6RQOk9l2--uHNsz3Mq7CBQ3q2gY?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/ac0e6bae38aa2b5a52a4f6e10b43c43d/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/11uPYA6RQOk9l2--uHNsz3Mq7CBQ3q2gY?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/11uPYA6RQOk9l2--uHNsz3Mq7CBQ3q2gY?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


TWTom: <p>Well, I’m a dumb dumb. Apperantly it’s just because of the <code>.to(dtype=torch.float64)</code> I copied from other places. And after changing all to <code>.to(dtype=torch.float32)</code>, and make sure <code>torch.from_numpy</code> also comes with one, it starts working.</p>
How can change a .pt file?

neverstop: <p>Hello, I am working on a pre-trained model. The model’s weights and biases are loaded from a .pt file and I get the following error. Can you help me? Thanks.</p>
<p>RuntimeError: Error(s) in loading state_dict for Network:<br>
size mismatch for fc.0.weight: copying a param with shape torch.Size([1600, 9]) from checkpoint, the shape in current model is torch.Size([1600, 55]).<br>
size mismatch for fc.6.weight: copying a param with shape torch.Size([9, 400]) from checkpoint, the shape in current model is torch.Size([55, 400]).<br>
size mismatch for fc.6.bias: copying a param with shape torch.Size([9]) from checkpoint, the shape in current model is torch.Size([55]).</p>

ptrblck: <p>The shapes of a few parameters mentioned in the error message does not match which points to a change in the model architecture. Where did you get the checkpoint from and which model are you creating?</p>

neverstop: <p>Thanks for answer.</p>
<aside class="onebox githubfolder" data-onebox-src="https://github.com/deepcombustion/deepcombustion/tree/main/DeePCK">
  <header class="source">

      <a href="https://github.com/deepcombustion/deepcombustion/tree/main/DeePCK" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/deepcombustion/deepcombustion/tree/main/DeePCK" target="_blank" rel="noopener nofollow ugc"></a></h3>

  <p><a href="https://github.com/deepcombustion/deepcombustion/tree/main/DeePCK" target="_blank" rel="noopener nofollow ugc">//github.com/deepcombustion/deepcombustion/tree/main/DeePCK</a></p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This DNN model is  for chemical combusiton.  I want to use it.</p>
<p>Model input size :9  but my input size:55</p>
<p>There is a weights an biases “model.pt” file in project.</p>
<p>I want to change input paramters for my “.yaml” inputs.</p>

ptrblck: <aside class="quote no-group" data-username="neverstop" data-post="3" data-topic="195827">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/a587f6/48.png" class="avatar"> neverstop:</div>
<blockquote>
<p>Model input size :9 but my input size:55</p>
</blockquote>
</aside>
<p>Load the checkpoint before manipulating the model and changing the number of features.</p>
Cannot run pytorch with CUDA 12.1 on a server with 8 x A100

Hasan_Sait_ARSLAN1: <p>Hi!</p>
<p>I have installed CUDA 12.1 on my machine with 8 x A100 GPUs, and installed the latest pytorch with CUDA 12.1 compatibility.</p>
<p>No matter what I have tried, I could not get rid of the following error, which makes me crazy:</p>
<pre><code class="lang-auto">
Python 3.11.7 (main, Dec 15 2023, 18:12:31) [GCC 11.2.0] on linux                                                                                                                      
Type "help", "copyright", "credits" or "license" for more information.                                                                                                                 
&gt;&gt;&gt; import torch                                                                                                                                                                       
&gt;&gt;&gt; torch.cuda.is_available()                                                                                                                                                          
/home/ubuntu/.build/miniconda3/envs/pytorch/lib/python3.11/site-packages/torch/cuda/__init__.py:138: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /opt/conda/conda-bld/pytorch_1702400430266/work/c10/cuda/CUDAFunctions.cpp:108.)                                                                                                                             
  return torch._C._cuda_getDeviceCount() &gt; 0
False
</code></pre>
<p>Appreciate your help, please!</p>

ptrblck: <p>Same as <a href="https://discuss.pytorch.org/t/pytorch-not-recognizing-gpu-cuda-initialization-cuda-driver-initialization-failed-you-might-not-have-a-cuda-gpu/195653/4">here</a> which is not reproducible.</p>
DataLoader tensor expects each tensor to be equal size but got different sizes

Anthony_Colombo: <p>Hello!<br>
i am new to pytorch and an running a set of images using a DataLoader that has an image of dimension (3,1079,1001)  paired with labels to this image of the same dimension.</p>
<p>The exact errors is:<br>
RuntimeError: stack expects each tensor to be equal size, but got [3, 1079, 1001] at entry 0 and [3, 1078, 1150] at entry 1</p>
<p>on each of my images, I went through my data loader and ensured that my data loader is not altering my dimensions of my images, and that my image dimensions match my paired label dimensions.</p>
<p>I saw <a href="https://discuss.pytorch.org/t/dataloader-gives-stack-expects-each-tensor-to-be-equal-size-due-to-different-image-has-different-objects-number/91941">this</a> post but am not sure if this applies</p>
<p>i am pretty certain that my images are not getting resized or getting padded and am pretty sure they are all matching dimesions.</p>
<p>any advice or suggestions to a new student here</p>
<p>thank you!</p>
Param.grad is always None

anum_abdul_salam: <pre><code class="lang-auto">def calculate_fisher_matrix_3(task_model, train_loader, criterion, feature_extractor, num_samples=1000):
    task_model.train()
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    fisher_matrix = [torch.zeros_like(param.data) for param in task_model.parameters()]
    total_samples = 0

    for i, data in enumerate(train_loader, 0):
        inputs = data["pixel_values"].requires_grad_(True)
        labels = data["labels"].float().requires_grad_(True)

        sizei = data["pixel_values"].shape[2], data["pixel_values"].shape[3]

        optimizer.zero_grad()

        # Forward pass
        logits = task_model(inputs)[0]
        upsampled_logits = nn.functional.interpolate(
            logits,
            size=sizei,  # (height, width)
            mode='bilinear',
            align_corners=False
        )
        logits = upsampled_logits.argmax(dim=1)
        logits = logits.float()

        # Compute loss
        loss = criterion(logits, labels)

        # Compute gradients for fisher_matrix
        loss.retain_grad()
        loss.backward()

        for i, param in enumerate(task_model.parameters()):
            if param.grad is not None:
                fisher_matrix[i] += param.grad.detach() ** 2

        # Reset gradients for the next batch
        task_model.zero_grad()

        total_samples += len(inputs)

        if total_samples &gt;= num_samples:
            break

    for i in range(len(fisher_matrix)):
        fisher_matrix[i] /= total_samples

    return fisher_matrix
</code></pre>
<p>I want to compute FIM, However its not updating param.grad, Please help</p>
Training multiple independent moddels at once

Nhan_Nguyen: <p>I have 50 completely independent models I want to train in parallel on 8 gpus. I have the model training run in a script that I run like</p>
<p><code>python training_script.py device_num</code></p>
<p>The simple way to do this is</p>
<pre><code class="lang-auto">for group in groups:
    processes = [subprocess.POpen(f'python training_script.py {device}'.split()) for device in range(8)]
    [p.wait() for p in process]
</code></pre>
<p>where groups, are the 50 processes split into groups of 8.</p>
<p>The downside of this is some models take longer than others to train and all models need to finish before it moves to the next group.</p>
<p>I was hoping to do something like multiprocess.spawn, but I need the last process to return the device number so it is clear which device is open to run on. I tried using Queue and Process from multiprocessing but I can’t get more than 1 process to run at once.</p>
<p>Any help would be very appreciated. Thanks</p>

Unuu_U: <p>Is this issue solved? I’m having a similar problem but I’m using torch.multiprocessing.Pool where I pass the id of the accelerator device similar to yours.<br>
The problem now becomes that I have to use 0 num_workers for dataloader.</p>
Calculating the gradient of one hidden unit wrt to another hidden unit

Sai_Kishore: <p>I want to calculate the gradient of the penultimate layer in a classifier wrt one of the hidden layers. Is there a provision for that</p>

ptrblck: <p>I’m not sure I understand your use case correctly.<br>
Do you want to compute the gradients of the parameters of the penultimate layer w.r.t. the forward activation of a hidden layer?</p>

Sai_Kishore: <p>Yes. In essence what i wanted to do was understand how a particular hidden unit contributes to the value of a hidden unit in the penultimate layer. This could be gauged by the gradient of the penultimate layer parameters wrt to the hidden unit parameters</p>

ptrblck: <p>Something like this might work:</p>
<pre data-code-wrap="python"><code class="lang-python">model = models.vgg16()

activation = {}
def get_activation(name):
    def hook(model, args, output):
        activation[name] = output
    return hook

model.classifier[3].register_forward_hook(get_activation("fc"))

x = torch.randn(1, 3, 224, 224)
out = model(x)

grad = torch.autograd.grad(outputs=activation["fc"], inputs=model.features[0].weight, grad_outputs=torch.ones_like(activation["fc"]))
</code></pre>
<p>which calculates the gradients in <code>model.features[0].weight</code>, which is the kernel of the first conv layer w.r.t. to the activations created in the penultimate linear layer.</p>

Sai_Kishore: <p>Thanks a lot Peter. This is really helpful</p>
Save a huggingface BERT model

Petro_Key: <p>Hi all, I am trying to save a BERT pretrained model from huggingface.<br>
I want to torch.save and load_state_dict a finetuned model.<br>
When I use the tokenizer as is, saved checkpoints can be resumed with loss values that were consistent with those at saved.<br>
However, when I add_tokens and resize_token_embeddings, saved checkpoints cannot be resumed with quite different loss values.</p>
<p>Is there appropriate ways to save the BERT model when I add new tokens?</p>
<p>Thank you.</p>

SravaniSolo: <ul>
<li>Use <code>save_pretrained</code> and <code>from_pretrained</code> to ensure consistency.</li>
<li>Resize token embeddings <strong>after</strong> loading the model and <strong>before</strong> saving.</li>
<li>Save and load both model and tokenizer together.</li>
<li>Consider using a checkpoint manager for managing multiple checkpoints.</li>
</ul>

Petro_Key: <p>Thank you Sravani for your kind reply.<br>
I have resized token embeddings after and before saving, and save and load model and tokenizer.<br>
So, the problem seems that I have used torch.save; I now understand I should have used save_pretrained.</p>
<p>I would appreciate any help for my another question.<br>
When I wrap some models as a class and want to save all the models together, should I separately use save_pretrained for BERT model and torch.save for the others?<br>
Is there any smart way to save the wrapped model altogether?</p>
<p>Thank you.</p>
Inference with own scaling factors

bok3948: <p>Hi, first of all, thank you for your great work.</p>
<p>I’m trying to perform INT8 inference with a quantized model using specific post-training quantization methods like PTQ4ViT and EasyQuant. I’ve reviewed the GitHub code from the quantization papers. Even though they use PyTorch, they obtain scaling factors and other quantization parameters, but they only restrict the range (e.g., INT8 [-128 to 127]). However, the actual calculation seems to be performed in FP32, which will not accelerate inference speed. (Check here: (1) <a href="https://github.com/hahnyuan/PTQ4ViT/tree/main" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hahnyuan/PTQ4ViT: Post-Training Quantization for Vision transformers.</a>, (2) <a href="https://github.com/hustvl/PD-Quant" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hustvl/PD-Quant: [CVPR 2023] PD-Quant: Post-Training Quantization Based on Prediction Difference Metric</a>).</p>
<p>So, I wonder if there is a solution for performing inference with low precision after obtaining my quantization parameters.</p>
<p>Thanks.</p>
Gradient propagation through MaskedTensor creation

heth27: <p>Is it possible to propagate gradients though the creation of a Masked Tensor, something like</p>
<p>import torch<br>
from torch.masked import masked_tensor</p>
<p>x = torch.tensor([-10., -5, 0, 5, 10, 50, 60, 70, 80, 90, 100], requires_grad=True)<br>
mask = x &lt; 0<br>
mx = masked_tensor(x, mask)<br>
mx.sum().backward()</p>
<p>I want to mask the outputs of a neural network, my current workaround is to use NANs, but MaskedTensor seem to be the nicer solution if backpropagation is possible.</p>
Having troubles to vmap a custom function with permute

Remilia-sky: <p>Hello, I want to vectorize the function <code>foo</code></p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

def foo(logprobs, zero_mask, action_space_size):
    ndim = zero_mask.shape[0]
    zero_mask = zero_mask.flip(0)
    reshape_sizes = torch.tensor([action_space_size], dtype=torch.int64).repeat_interleave(ndim)
    logprobs = logprobs.reshape(*reshape_sizes.tolist())

    sorted_indices = torch.argsort(zero_mask)
    logprobs = logprobs.permute(*sorted_indices)

    logprobs = logprobs.reshape(-1, ndim ** zero_mask.sum())
    logprobs = torch.logsumexp(logprobs, dim=-1, keepdim=True)
    logprobs = F.pad(logprobs, [0, ndim ** zero_mask.sum() - 1], value=-1e7)

    logprobs = logprobs.reshape(*reshape_sizes.tolist())
    invert_indices = torch.argsort(sorted_indices)
    logprobs = logprobs.permute(*invert_indices)
    logprobs = logprobs.reshape(-1)

    return logprobs

def bar(logprobs, zero_mask):
    # use vmap
    
    batched_foo = torch.func.vmap(foo, in_dims=(0, 0, None))
    return batched_foo(logprobs, zero_mask, 3)

logits = torch.ones((4, 1, 27), dtype=torch.float32)
logprob = F.log_softmax(logits, dim=-1)
zero_mask = torch.tensor([[False, True, True], [True, False, True], [False, False, False], [True, True, True]])


logprob = bar(logprob, zero_mask)
</code></pre>
<p>The <code>permute</code> in this function is really tricky. After I use vmap, it throws</p>
<pre><code class="lang-auto">TypeError: permute(): argument 'dims' (position 1) must be tuple of ints, but found element of type Tensor at pos 0.
</code></pre>
<p>If I change the code into</p>
<pre><code class="lang-auto">logprobs = logprobs.permute(*sorted_indices.tolist())
</code></pre>
<p>The error msg became</p>
<pre><code class="lang-auto">RuntimeError: Cannot access data pointer of Tensor that doesn't have storage
</code></pre>
<p>I have no idea how to solve it.</p>
I called `scheduler.step()` right after `optimizer.step()`, but why there was a warning said I might call them in an inverse order?

jack_S: <p>Here’s the code snippet:</p>
<pre><code class="lang-auto">from torch.optim.lr_scheduler import CosineAnnealingLR
from torch.optim import SGD
from torchvision.models import resnet18
from copy import deepcopy
from torch import randn

model = resnet18()
optimizer = SGD(model.parameters(), 0.01)
lr_scheduler = CosineAnnealingLR(optimizer, 10)
lr_schedulers = [deepcopy(lr_scheduler) for _ in range(0, 5)]
current_scheduler = None

for i in range(0, 60):
    if i % 10 == 0:
        current_scheduler = lr_schedulers[i // 10]
    x = randn(10, 3, 32, 32)
    loss = model(x).sum()
    loss.backward()
    optimizer.step()
    current_scheduler.step()
</code></pre>
<p>As can be seen in the snippet, <code>scheduler.step()</code> is called right after <code>optimizer.step()</code>, but I received a warning said</p>
<pre><code class="lang-auto">/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:129:
 UserWarning: Seems like `optimizer.step()` has been overridden after learning rate scheduler
 initialization. Please, make sure to call `optimizer.step()` before `lr_scheduler.step()`. 
See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  warnings.warn("Seems like `optimizer.step()` has been overridden after learning rate scheduler "
</code></pre>
Apply a function on each element of tensor array

fatemeh.sepah: <p>hello<br>
i want to apply this function on each element of tensor array, can i do it without for loop and apply_ function? i want to run it on GPU.</p>
<p>import torch</p>
<p>length = 8</p>
<p>def WToSN(In):<br>
LenSN = 2 ** length<br>
SN = torch.zeros(LenSN, device=torch.device(‘cuda’))<br>
SN[0:In] = 1<br>
return SN</p>
<p>weightfold2 = torch.tensor([[1, 2, 3], [4, 5, 6]])<br>
weight = weightfold2.flatten()<br>
WeightSN = torch.tensor([WToSN(elem) for elem in weight])<br>
WeightSN = WeightSN.reshape(weightfold2.size(0), weightfold2.size(1), -1)</p>
<p>thank you</p>

marksaroufim: <p>What’s wrong with using <code>apply_</code>? This works without a for loop</p>
<pre><code class="lang-auto">import torch
def custom_function(x):
    return x ** 2

tensor = torch.tensor([1, 2, 3, 4, 5])
result = tensor.apply_(custom_function)
print(result) # tensor([ 1,  4,  9, 16, 25])
</code></pre>

fatemeh.sepah: <p>thank you for your answer<br>
apply_ function can’t run on GPU, i want to run my code on GPU, isn’t there any solution?</p>

marksaroufim: <p>Ah my bad this should work</p>
<pre><code class="lang-auto">import torch
def custom_function(x):
    return x ** 2

tensor = torch.tensor([1, 2, 3, 4, 5], device="cuda")
tensor = custom_function(tensor)
print(tensor) # tensor([ 1,  4,  9, 16, 25])
</code></pre>

fatemeh.sepah: <p>thank you for your answer<br>
it’s doing correct for this function but in my function:<br>
import torch</p>
<p>length = 8</p>
<p>def WToSN(In):<br>
LenSN = 2 ** length<br>
SN = torch.zeros(LenSN, device=torch.device(‘cuda’))<br>
SN[0:In] = 1<br>
return SN</p>
<p>weight = torch.tensor([[1, 2, 3], [4, 5, 6]], device=torch.device(‘cuda’))<br>
print(weight)<br>
WeightSN = WToSN(weight)<br>
print(WeightSN)</p>
<p>i get this error:TypeError: only integer tensors of a single element can be converted to an index<br>
for this line:    SN[0:In] = 1<br>
can you correct this?thank you</p>
Num_workers on single GPU

furkat: <p>I am experimenting with Xception network on my PC which has single Geforce GTX 1660 GPU. Batch size is 8. The dataset directory contains train and validation directories which in turn contain seven classes in each.<br>
I first create a function that returns  list of images_path and labels_dict as follows:</p>
<pre><code class="lang-auto">def imagee(data_path):
    classes = os.listdir(data_path)

    images_path = []
    labels_dict = {}
    for i, class_name in enumerate(classes):
        class_path = os.path.join(data_path, class_name)
        for image_path in os.listdir(class_path):
            image_path = os.path.join(class_path, image_path)
            images_path.append(image_path)
            labels_dict[image_path] = i
    
    return images_path, labels_dict
</code></pre>
<p>Then, for both validation and train dataset, I get the returned images_path and labels_dict:</p>
<pre><code class="lang-auto">train_images_path, train_labels_dict = imagee(train_data_path)
test_images_path, test_labels_dict = imagee(test_data_path)
</code></pre>
<p>My custom Dataset class is as follows:</p>
<pre><code class="lang-auto">class FaceRecognationDataset(Dataset):
    def __init__(self, images_path, labels_dict, transforms=None):
        super(FaceRecognationDataset, self).__init__()
        self.images_path = images_path
        self.labels_dict = labels_dict
        self.transforms = transforms

    def __len__(self):
        return len(self.images_path)

    def __getitem__(self,index):
        image_path = self.images_path[index]
        label = self.labels_dict[image_path]
        image = Image.open(image_path).convert("RGB")

        if self.transforms != None:
            image = self.transforms(image)

        return image, label
</code></pre>
<p>Transforms, datasets and dataloaders:</p>
<pre><code class="lang-auto">my_transforms = transforms.Compose([
    transforms.Resize(308),
    transforms.RandomCrop(299),
    transforms.ToTensor()])

train_dataset = FaceRecognationDataset(train_images_path, train_labels_dict, transforms=my_transforms)
test_dataset = FaceRecognationDataset(test_images_path, test_labels_dict, transforms=my_transforms)

train_loader = DataLoader(train_dataset, shuffle=True, batch_size=batch_size, num_workers=2)
test_loader = DataLoader(test_dataset, shuffle=True, batch_size=batch_size, num_workers=2)
</code></pre>
<p>I try to get the first batch of test_loaders as follows:</p>
<pre><code class="lang-auto">images, labels = next(iter(test_loader))
</code></pre>
<p>And it takes forever.<br>
After trying for some hours, I removed num_workers in data_loader and it worked out finally.<br>
Why is it taking such long time with num_workers on in my case?</p>

ptrblck: <p>I assume your code hangs and has issues using multiprocessing and is not just slow.</p>

furkat: <p>I guess so too. I just thought it would take too long to finish the process and interrupted it after waiting for 5 mins. What could be the isssue?</p>
Cuda_Launch_Blocking=1 reduces the training speed?

Yangmin: <p>Hi,  I was trying to debug about the</p>
<p>"RuntimeError: cuda runtime error (710) : device-side assert triggered "</p>
<p>by using Cuda_Launch_Blocking=1.</p>
<p>It seems like setting the above environment variable to 1 slows down the training speed of whole code.</p>
<p>Should I use this blocking=1 only in debugging and not in training?<br>
Does it reduce the speed?</p>

ptrblck: <p>Yes, <code>CUDA_LAUNCH_BLOCKING=1</code> is a debug env variable used to block kernel launches and to report the proper stacktrace once an <code>assert</code> is triggered. You should not use it in production, but only during debugging.</p>

Amey_Naik: <p>What does <code>CUDA_LAUNCH_BLOCKING=1</code> do exactly? Can someone please elaborate?</p>

ptrblck: <p>Kernel launches will be blocked, i.e. they won’t be executed asynchronously anymore as already explained. From CUDA’s programming guide:</p>
<blockquote>
<p>Programmers can globally disable asynchronicity of kernel launches for all CUDA applications running on a system by setting the CUDA_LAUNCH_BLOCKING environment variable to 1. This feature is provided for debugging purposes only and should not be used as a way to make production software run reliably.</p>
</blockquote>
Mixed Precision Gradient Checking

Jerome_Ku: <p>I created a <code>torch.autograd.Function</code> that uses custom kernels in <code>fwd</code> and <code>bwd</code> that expect half precision inputs.  The network as a whole is trained in mixed precision using <code>torch.autocast</code> in <code>fp16</code> or <code>bf16</code>, and the <code>forward</code> and <code>backward</code> methods of the function are decorated with <code>torch.cuda.amp.custom.{fwd,bwd}</code>.</p>
<p>Are there any suggestions for running <code>gradcheck</code> for mixed precision functions to verify that my backwards implementation is correct?</p>

ptrblck: <p>Couldn’t you disable <code>autocast</code> for the sake of testing the correctness of the gradients in <code>float64</code>?</p>