akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Different output from the same trained model

nil_tdr: <p>Hello!</p>
<p>I trained my model and saved it. When I load the checkpoints from model.pth file and try to get output for different runs I am getting different outputs for the same input. Should not the weights and biases to be same for each testing of the model?</p>
<p>Thanks for your help!<br>
Nil</p>

ptrblck: <aside class="quote no-group" data-username="nil_tdr" data-post="1" data-topic="191484">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nil_tdr/48/65323_2.png" class="avatar"> nil_tdr:</div>
<blockquote>
<p>Should not the weights and biases to be same for each testing of the model?</p>
</blockquote>
</aside>
<p>Yes and you can manually verify it by accessing and comparing the parameters of both models.</p>
<aside class="quote no-group" data-username="nil_tdr" data-post="1" data-topic="191484">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nil_tdr/48/65323_2.png" class="avatar"> nil_tdr:</div>
<blockquote>
<p>for different runs I am getting different outputs for the same input</p>
</blockquote>
</aside>
<p>Did you disable any randomness via <code>model.eval()</code>?</p>

nil_tdr: <p>No I did not. After doing it I have same output in every run. When I load saved model checkpoints why I have randomness?</p>

ptrblck: <p>You are most likely using some layers, which change their behavior between <code>.train()</code> and <code>.eval()</code>, such as <code>nn.Dropout</code> layers which will be disabled during <code>model.eval()</code>. It’s unrelated to loading a checkpoint and is just the model’s training state.</p>
torchvision.datasets.Flickr8k does not work

Srinjoy_Mukherjee: <p>Hello all,<br>
I am currently working with Flickr8k dataset. I downloaded the dataset from <a href="https://www.kaggle.com/datasets/adityajn105/flickr8k" rel="noopener nofollow ugc">here</a>. Then , I extracted the zip such that all Images are Flickr8k/Images and caption is in Flickr8k/captions.txt.  Then I used the following code</p>
<pre><code class="lang-python">from torch.utils.data import Dataset, DataLoader
image_folder = './Flickr8k/Images'
caption_path = './Flickr8k/captions.txt'

data_transform = transforms.Compose([
    transforms.Resize(128),
    transforms.ToTensor()
])

trainset = datasets.Flickr8k(root=image_folder, ann_file=caption_path, transform=data_transform)

</code></pre>
<p>The trainset does not load anything.</p>
<pre><code class="lang-python">Dataset Flickr8k
    Number of datapoints: 0
    Root location: ./Flickr8k/Images
    StandardTransform
Transform: Compose(
               Resize(size=128, interpolation=bilinear, max_size=None, antialias=warn)
               ToTensor()
           )
</code></pre>
<p>Please tell me where I am going wrong. The Documentation does not specify anything.</p>
<p>Thanks</p>
How to store variables used for training data later for prediction

svk: <p>I  need to calculate standard deviation, mean and max of the training data for normalization. Now I need to use these values calculated on training data also for predicting later. (Which is a separate script that loads the model weights). How should I transfer these values? Is it possible to include them in the saved model somehow?</p>
<p>Also a question regarding the calculation of standard deviation: My data consists of several matrices, each are in separate file. I load the data with a class derived from torch.utils.data.Dataset. The <strong>init</strong> function stores the name of the files and <strong>getitem</strong> opens each file, reads the data and returns the tensor, so not all matrices are loaded into memory at the same time. But this also means I cannot calculate standard deviation without seeing all the data. So I open all files one more time before the above is done to calculate std. Is there a better way?</p>

ptrblck: <aside class="quote no-group" data-username="svk" data-post="1" data-topic="191478">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/9de0a6/48.png" class="avatar"> svk:</div>
<blockquote>
<p>Is it possible to include them in the saved model somehow?</p>
</blockquote>
</aside>
<p>Yes, you can store them as buffers inside your <code>nn.Module</code> via <code>self.register_buffer</code>, which is also done in <code>nn.BatchNormXd</code> layers for the internal running stats.</p>
When i run detection on custom weight file for single class using yolov5 im getting error

Adinath_Kale: <p>Using cache found in C:\Users\adina/.cache\torch\hub\ultralytics_yolov5_master<br>
YOLOv5  2023-11-8 Python-3.8.18 torch-1.10.0+cu113 CUDA:0 (NVIDIA GeForce GTX 1650, 4096MiB)</p>
<p>Fusing layers…<br>
Model summary: 206 layers, 12308200 parameters, 0 gradients, 16.1 GFLOPs<br>
Adding AutoShape…</p>

ptrblck: <p>I don’t see any error messages so could you describe what exactly is failing?</p>
There seems to be a bug in the torch.baddbmm() interface.

scarecrow: <p>There seems to be a bug in the torch.baddbmm() interface. When alpha is equal to 0, if the calculation scale of tensor is relatively large, beta will not participate in the calculation and directly return the value of the torch_tensor3 variable. But if the calculation scale of tensor is small, beta will participate in the calculation at this time. why is that?</p>
<p>This is a bug in pytorch1.9 version, and beta does not participate in the calculation.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecb1d6a3370d2d41d868dc8c48a86927415a8a88.png" data-download-href="https://discuss.pytorch.org/uploads/default/ecb1d6a3370d2d41d868dc8c48a86927415a8a88" title="youdu1699450544000"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ecb1d6a3370d2d41d868dc8c48a86927415a8a88_2_597x499.png" alt="youdu1699450544000" data-base62-sha1="xLTHpbAxSIqdWwMJFzmfBLJfuR2" width="597" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ecb1d6a3370d2d41d868dc8c48a86927415a8a88_2_597x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ecb1d6a3370d2d41d868dc8c48a86927415a8a88_2_895x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecb1d6a3370d2d41d868dc8c48a86927415a8a88.png 2x" data-dominant-color="111211"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">youdu1699450544000</span><span class="informations">1041×871 85.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
device = torch. device(" cuda")</p>
<p>torch_ tensor 1 = torch. randn( 887040, 10,124). to(device)</p>
<p>torch_ tensor_ 2 = torch . randn( 887040, 124, 1). to(device)</p>
<p>torch_ tensor 3 = torch. randn( 887040, 10, 1). to( device)</p>
<p>print(" torch_ tensor_ 3 device:",torch_ tensor_ 3. device)</p>
<p>print(" torch tensor 2 device:",torch tensor 2.device)</p>
<p>print(" torch tensor 1 device:",torch_ tensor 1. device)</p>
<p>torch_ out = torch . baddbmm(torch_ tensor 3,torch_ tensor 1, torch tensor 2, beta= 2, alpha= 0)</p>
<p>print(torch_ tensor _3.cpu().view( -1)[ :10])</p>
<p>print(torch_ out .cpu() .view(-1)[:10])</p>

ptrblck: <p>Could you post code snippets reproducing the issue as well as the outputs instead of images as I cannot see them?</p>

scarecrow: <p>Sorry, I’m a newbie and something went wrong just now, I’ve uploaded the correct image.<br>
If the tensor’s batch_size is reduced to 40, then beta will participate in the calculation. Currently I can only upload one image</p>

scarecrow: <p>Thank you for checking my question</p>

scarecrow: <p>I re-edited and now I paste the code</p>
Train on one set of classes, report metrics on subset of classes

hextopia: <p>Hi,</p>
<p>I’ve been searching everywhere for info on how to best implement this, but I’ve come up short on helpful info.</p>
<p>I’m attempting to build an image classifier model in pytorch that’s trained on one set of image classes (e.g: Dogs vs Cats) but then report test accuracy metrics across subsets of the test dataset to look for trends in how the model performs (e.g: Subclasses like “Brown Dogs, White Dogs, Short Hair Dogs, Long Hair Dogs”)</p>
<p>I know I could build a dataset/dataloader for the test set by concating or stacking all the test sub-datasets into one dataloader, but I’m not sure if there’s a smarter way to generate metrics for a model so I can sort and display them in tensorboard, as I’m about to just iterate through the ~100 different possible subsets in the training dataset and report metrics for each one individually after each epoch of training.  This seems like it’d be far worse performance-wise than tagging and collecting the performance metrics as the dataset is processed in the test step from one dataset formed from all the smaller ones though.</p>
<p>I’m by no means a pytorch expert though, and some of the more obscure functions in pytorch have documentation that I struggle to understand, so the answer could be in plain sight in front of me and I’m just not seeing it.</p>
Torch.cuda.is_available() returns False on Ubuntu

Ajit: <p>Hi,<br>
Facing issue with cuda. Below are the details:</p>
<blockquote>
<blockquote>
<blockquote>
<p>torch.<strong>version</strong><br>
‘1.13.1+cu117’<br>
torchvision.<strong>version</strong><br>
‘0.14.1+cu117’</p>
</blockquote>
</blockquote>
</blockquote>
<p>Python 3.8.0</p>
<p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |<br>
|                               |                      |               MIG M. |<br>
|===============================+======================+======================|<br>
|   0  Quadro P2000        Off  | 00000000:2D:00.0 Off |                  N/A |<br>
| 49%   40C    P8     4W /  75W |     17MiB /  5120MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+<br>
|   1  NVIDIA RTX A2000    Off  | 00000000:8D:00.0 Off |                  Off |<br>
| 30%   37C    P8     5W /  70W |      6MiB /  6138MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+<br>
|   2  NVIDIA GeForce …  Off  | 00000000:99:00.0 Off |                  N/A |<br>
|  0%   40C    P8     2W / 215W |      6MiB /  8192MiB |      0%      Default |<br>
|                               |                      |                  N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes:                                                                  |<br>
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |<br>
|        ID   ID                                                   Usage      |<br>
|=============================================================================|<br>
|    0   N/A  N/A      3279      G   /usr/lib/xorg/Xorg                  9MiB |<br>
|    0   N/A  N/A      3447      G   /usr/bin/gnome-shell                4MiB |<br>
|    1   N/A  N/A      3279      G   /usr/lib/xorg/Xorg                  4MiB |<br>
|    2   N/A  N/A      3279      G   /usr/lib/xorg/Xorg                  4MiB |</p>

ptrblck: <pre><code class="lang-python">| 0 Quadro P2000 Off | 00000000:2D:00.0 Off | N/A |
</code></pre>
<p>One of your GPUs seems to be disabled so make sure the right one is used e.g. via exporting <code>CUDA_VISIBLE_DEVICES</code>.</p>
Pyinstaller & pytorch

Ziri_Ziri: <p>I was able to pack a small example app that includes pytorch and pyQt using Pyinstaller but the dist folder is 4GB .</p>
<p>All I am doing is printing pytorch version on QLabel !</p>
<pre><code class="lang-auto"> from PyQt5.QtWidgets import QMainWindow, QApplication, QLabel, QVBoxLayout, 
 QPushButton, QWidget
 from PyQt5.QtGui import QIcon
 import sys, os
 from torch import __version__
</code></pre>
<p>Am I using Pyinstaller the wrong way ?</p>
<p>Update:</p>
<p>I checked again and 3.99GB is the torch\lib folder that contains all libs and dlls . this is something that i experienced with Libtorch as well.  the dependency files are huge.</p>

marksaroufim: <p>pytorch is quite big, there might be a way these days to do selective builds cc <a class="mention" href="/u/malfet">@malfet</a></p>

psummerford: <p>Have you been able to fix this issue? I’m running into it as well and could not find another forum mentioning this.</p>
How to save and load tensors in the best way when you have very large data

DKaly: <p>I am facing a question that I would need more support from the community.<br>
For example, I want to make a recommendation system. I turn every part of the product into an embed.</p>
<pre><code class="lang-auto">product_url_embedding: Embedding = self.get_product_url_embedding(product_url=product.product_url)
        categories_embedding: Embedding = self.get_categories_embedding(categories=product.categories)
        price_embedding: Embedding = self.get_price_embedding(price=product.price)
</code></pre>
<p>All good so far. But for example, I need products when I make recommendations in this form so that I can calculate the similarities with the user’s actions.<br>
My question is:<br>
How could I keep this data so that I don’t have a big load?<br>
For example, I now save each embed in the database as a <code>BYTEA</code>, but this is expensive in terms of performance…<br>
Any ideas?</p>
Saving optimizer to a file

Jesus: <p>Hello everyone. when i save SGD optimizer using “torch::save(optimizer, “model-optimizer-checkpoint.pt”)”,</p>
<p>i got this error "<br>
[W sgd.cpp:124] Warning: Your serialized SGD optimizer is still using the old serialization format. You should re-save your SGD optimizer to use the new serialization format. (function load)<br>
terminate called after throwing an instance of ‘c10::Error’<br>
what():  No such serialized tensor ‘momentum_buffers/size’<br>
Exception raised from read at …/torch/csrc/api/src/serialize/input-archive.cpp:69 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt;) + 0x6b (0x7f02233dd05b in /home/modjo/dependencies/libtorch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;) + 0xbf (0x7f02233d7f6f in /home/modjo/dependencies/libtorch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#2:</span> torch::serialize::InputArchive::read(std::__cxx11::basic_string&lt;char, std::char_traits, std::allocator &gt; const&amp;, at::Tensor&amp;, bool) + 0xda (0x7f020ab9bcba in /home/modjo/dependencies/libtorch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x514967e (0x7f020ab6767e in /home/modjo/dependencies/libtorch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#4:</span> torch::optim::SGD::load(torch::serialize::InputArchive&amp;) + 0x19d (0x7f020ab94e0d in /home/modjo/dependencies/libtorch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#5:</span> torch::optim::operator&gt;&gt;(torch::serialize::InputArchive&amp;, torch::optim::Optimizer&amp;) + 0x31 (0x7f020ab872c1 in /home/modjo/dependencies/libtorch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag-raw">#6:</span> ./build/train() [0x433a93]<br>
frame <span class="hashtag-raw">#7:</span> ./build/train() [0x4159dc]<br>
frame <span class="hashtag-raw">#8:</span> __libc_start_main + 0xf3 (0x7f02050cf493 in /lib64/libc.so.6)<br>
frame <span class="hashtag-raw">#9:</span> ./build/train() [0x41178e] "</p>
Can an outside scalar be used as a loss value?

SirKrovlood: <p>Here is the situation:<br>
I have an NN, which outputs a sequence of actions (an int in range of [0:3]) for a robot to perform in an environment. I can make the robot perform this sequence and save an array of (x,y,z) positions the robot visits while performing said actions. Then I can use Dynamic Time Warping to compare the resulted array of positions with the GT positions.</p>
<p>I would like to use the DTW value as loss. However, I don’t see any possibility to use it directly. The only solution I came up with, is to firstly perform cross-entropy between the predicted and GT action sequences and then multiply it by DTW using it as a sort of dynamic learning rate.</p>
<p>Is there any other method to use an outside value as a loss?</p>
ExecuTorch & TVM

escorciav: <p>Hi folks!</p>
<p>Kudos to all the people involved in the development &amp; release of ExecuTorch <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=12" title=":tada:" class="emoji" alt=":tada:" loading="lazy" width="20" height="20"><br>
It’s great to see a new revamped and comprehensive pipeline to bring AI to mobile, the edge &amp; “atypical” computing devices <img src="https://discuss.pytorch.org/images/emoji/apple/raised_hands.png?v=12" title=":raised_hands:" class="emoji" alt=":raised_hands:" loading="lazy" width="20" height="20"></p>
<p><strong>Could anyone shed light about</strong> similarity/overlap, pros/cons, diff/synergies between the scopes of  <strong>ExecuTorch &amp; TVM</strong>?</p>
<p>Thanks in advance!</p>

escorciav: <p>The brother thread for this post in the TVM community already grab the attention of <a href="https://tqchen.com/" rel="noopener nofollow ugc">Tianqi Chen</a>  <img src="https://discuss.pytorch.org/images/emoji/apple/heart_eyes.png?v=12" title=":heart_eyes:" class="emoji" alt=":heart_eyes:" loading="lazy" width="20" height="20"></p>
<p><em>Will this post caught the attention of ExecuTorch core dev &amp; maintainers?</em><br>
Let’s up-vote &amp; perhaps <a class="mention" href="/u/smth">@smth</a> will chime in <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>
<aside class="onebox discoursetopic" data-onebox-src="https://discuss.tvm.apache.org/t/executorch-in-the-context-of-tvm/15951/2?u=escorciav">
  <header class="source">
      <img src="https://discuss.tvm.apache.org/uploads/default/optimized/1X/a8769a3a42fbbeb0ea074a52bcf3a790df740574_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://discuss.tvm.apache.org/t/executorch-in-the-context-of-tvm/15951/2?u=escorciav" target="_blank" rel="noopener nofollow ugc" title="10:40PM - 07 November 2023">Apache TVM Discuss – 7 Nov 23</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:70/68;"><img src="https://discuss.tvm.apache.org/uploads/default/original/1X/a8769a3a42fbbeb0ea074a52bcf3a790df740574.png" class="thumbnail" width="70" height="68"></div>

<div class="title-wrapper">
  <h3><a href="https://discuss.tvm.apache.org/t/executorch-in-the-context-of-tvm/15951/2?u=escorciav" target="_blank" rel="noopener nofollow ugc">ExecuTorch in the context of TVM</a></h3>
</div>

  <p>I think there are great synergies. e.g. we could leverage the export from ExecuTorch and connect some of the relax operator offloadings to what ExecuTorch can offer. In the meantime, we can also bring compilation support together</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How can I get the input_shape and output_shape of each node of graph traced by torch.fx

jrt-20: <p>I want to use symbolic_trace from torch.fx to trace the model.<br>
symbolic_trace will return a GraphModule,how can i get the input_shape and output_shape of each node in the graphModule.graph.</p>

soulitzer: <p>the graph module’s graph has a list nodes and you can check for this information on the meta attribute, e.g.<br>
for output_shape, do <code>list(gm.graph.nodes)[i].meta</code><br>
for input_shape, you can grab the args of the node and <code>list(gm.graph.nodes)[i].args[j].meta</code></p>

jrt-20: <p>thanks,bro.I will try it</p>
Neural network weights initialization that compromise each other

lndip: <p>I have tried to investigate how the performance of the model is affected if the weights of the first layer is initialized to be really large and the weights of the second layer is initialized to be really small. It seems that the model cannot learn the right gradients for it to converge. Why is this the case when the first and second layer of weights can compromise with each other?</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.conv1 = torch.nn.Conv1d(141,100,5)
        self.conv2 = torch.nn.Conv1d(100,1,1)
        self.relu = nn.ReLU()
    
    def forward(self,X):
        X = X.transpose(2,1)
        X = self.conv1(X)
        X = self.relu(X)
        X = self.conv2(X)
        return X.transpose(2,1)

model = Net()
model.conv1.weight.data *= 1000
model.conv1.bias.data *= 1000
model.conv2.weight.data /= 1000

# start training here
</code></pre>
How can I link albumentations.compose to transforms.compose

shiekahah: <p>Hello<br>
I’m new to image processing.</p>
<p>I have a project that I need to add more augmentation options to enhance ResNet34 work. They already used transforms.compose() for a few augmentations and I want to use albumentations.compose() How can link them together so they all work on the image dataset?</p>
ConvNext training in Torch 20x slower than TF

Ineed8234: <p><a href="https://colab.research.google.com/drive/1lZ0Ro8OGmSTr-7YnREqw9boXd7Z26PEH?usp=sharing" rel="noopener nofollow ugc">Colab link</a></p>
<p>Hi,<br>
I am trying to build ConvNeXt Tiny models to fine-tune them on CIFAR-100 in both PyTorch and TensorFlow, but I am unable to resolve this problem. PyTorch’s implementation either crashes when I set too big batch size or is a lot slower, and by a lot I mean 10-20 times slower. Please help me find out what might be causing this problem, as I am out of ideas. I’ve tested it on multiple versions and platforms. Nothing helps.</p>
Torch.no_grad and init functions

Vidit_Agarwal: <p>It’s mentioned here <a href="https://pytorch.org/docs/stable/nn.init.html#nn-init-doc" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.init — PyTorch 2.1 documentation</a> that<br>
functions used to initialize neural network parameters run in <a href="https://pytorch.org/docs/stable/generated/torch.no_grad.html#torch.no_grad" rel="noopener nofollow ugc"><code>torch.no_grad()</code></a> mode so that they will not be taken into account by autograd.</p>
<p>I am not able to get a feel of this statement, it seems to be correct that, “why do even need to consider the initialization function for the backward function.” but <strong>what will be the effect if we consider it</strong>?</p>

soulitzer: <p>The init function modifies in-place and that is not allowed on parameters, so it would result in an error.</p>
LibTorch linking in Bazel produces "dyld[22939]: missing symbol called"

vymao: <p>I have a pretty simple setup for libtorch, as I want to use the C++ API in my Bazel project. My WORKSPACE has the following:</p>
<pre><code class="lang-auto">    http_archive(
        name = "libtorch",
        strip_prefix = "libtorch",
        type = "zip",
        urls = ["https://download.pytorch.org/libtorch/cpu/libtorch-macos-2.1.0.zip"],
        build_file = Label("//external:BUILD.libtorch")
    )
</code></pre>
<p>and my libtorch build looks like</p>
<pre><code class="lang-auto">cc_library(
    name = "torch_cpu",
    srcs = glob(["**/*.h", "**/*.hpp", "**/*.a", "**/*.so"]),
    hdrs = glob(["include/**/*.h"]),
    includes = [
        "include",
        "include/torch/csrc/api/include",
    ],
    linkopts = ["-undefined error"],
    visibility = ["//visibility:public"]
)
</code></pre>
<p>Building the main target, which depends on this library, works fine. But when I try running <code>bazel run ...</code>, I get a runtime error like <code>dyld[22939]: missing symbol called</code>. Wondering if someone could help me out?</p>
<p>Currently working on an Intel MacOS 13.5.</p>
Network not learning correctly when a target network is introduced

nl22: <p>I’m pretty new to Pytorch and I’m having issues with getting my agent to learn when I introduce a second network as my target network in a DQN. I originally had a simple DQN with just a policy net and a replay buffer which actually learned pretty well. Per convention and to get better stability I want to introduce a target net. As a first step, I have the target net updating every iteration to verify that I get similar results to the simple one-net DQN.</p>
<p>Okay, now for the issue I’m facing. I have added my second target net by creating a second network object using the same class I used for my policy net. During training, I use my replay memory which consists of (state, action, reward, new_state, done_fllag). I send the state batches forward through the policy net to get my predictions, and then I send the new_state batch forward through my target net to get my target values. From there, I calculate my loss. No matter if I use the policy net or the target net the losses are the same. I call loss.backward(), and the gradient isn’t 0, nor does it explode, but the network only ever learns when I use the policy net wor calculate the losses. The network weights change, so the gradient seems to be affecting it, but it doesn’t affect it in a good way.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84b012ae0ab2d7adb2358c3cfbbd175eaf405a95.png" data-download-href="https://discuss.pytorch.org/uploads/default/84b012ae0ab2d7adb2358c3cfbbd175eaf405a95" title="deubgging_net"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/84b012ae0ab2d7adb2358c3cfbbd175eaf405a95_2_690x485.png" alt="deubgging_net" data-base62-sha1="iVOga6CXXraHSA9uegT1UWTngTb" width="690" height="485" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/84b012ae0ab2d7adb2358c3cfbbd175eaf405a95_2_690x485.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/4/84b012ae0ab2d7adb2358c3cfbbd175eaf405a95_2_1035x727.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/4/84b012ae0ab2d7adb2358c3cfbbd175eaf405a95.png 2x" data-dominant-color="363D46"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">deubgging_net</span><span class="informations">1172×824 99.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m only able to get this to learn when i use the policy net to get the target_reward though.</p>
GPU- clear variable after use

Sumesh_Sankar: <p>How do i clear all the variables, that are stored in GPU via cuda programming,after its use, so that memory can be effectively managed</p>

nstrek: <p>You can try this snippet:</p>
<pre><code class="lang-auto">import gc
gc.collect()

with torch.no_grad():
    torch.cuda.empty_cache()
</code></pre>
<p>and read:</p><aside class="quote" data-post="18" data-topic="14530">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/debvrat/48/23126_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/18">How can we release GPU memory cache?</a> 
  </div>
  <blockquote>
    This works only some of the times. 
Even when I clear out all the variables, restart the kernel, and execute torch.cuda.empty_cache() as the first line in my code, I still get a ‘CUDA out of memory’ error.
  </blockquote>
</aside>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/57858433/how-to-clear-gpu-memory-after-pytorch-model-training-without-restarting-kernel">
  <header class="source">

      <a href="https://stackoverflow.com/questions/57858433/how-to-clear-gpu-memory-after-pytorch-model-training-without-restarting-kernel" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/9178303/glyph" target="_blank" rel="noopener nofollow ugc">
    <img alt="Glyph" src="https://www.gravatar.com/avatar/9f50a44d87ff556d783b165622220608?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/57858433/how-to-clear-gpu-memory-after-pytorch-model-training-without-restarting-kernel" target="_blank" rel="noopener nofollow ugc">How to clear GPU memory after PyTorch model training without restarting kernel</a>
</h4>

<div class="tags">
  <strong>python, pytorch, jupyter</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/9178303/glyph" target="_blank" rel="noopener nofollow ugc">
    Glyph
  </a>
  on <a href="https://stackoverflow.com/questions/57858433/how-to-clear-gpu-memory-after-pytorch-model-training-without-restarting-kernel" target="_blank" rel="noopener nofollow ugc">05:12PM - 09 Sep 19 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to confirm DDP collects all GPU card(rank) results at each epoch

Joeyoung: <p>I ran the example in <a href="https://pytorch.org/tutorials/beginner/ddp_series_theory.html" class="inline-onebox" rel="noopener nofollow ugc">What is Distributed Data Parallel (DDP) — PyTorch Tutorials 2.1.0+cu121 documentation</a> in single node with 4 gpu cards. As shown the video in the example, I expected at each epoch, all ranks must be performed. However, it prints out all epochs are done at each GPU sequentially.<br>
i.e. For 10 epochs, it prints as follows:</p>
<p>[GPU3] Epoch 0<br>
[GPU3] Epoch 1<br>
:<br>
[GPU3] Epoch 8<br>
[GPU3] Epoch 9<br>
[GPU1] Epoch 0<br>
[GPU1] Epoch 1<br>
:<br>
[GPU1] Epoch 8<br>
[GPU1] Epoch 9<br>
[GPU2] Epoch 0<br>
[GPU2] Epoch 1<br>
:<br>
[GPU2] Epoch 8<br>
[GPU2] Epoch 9<br>
[GPU0] Epoch 0<br>
[GPU0] Epoch 1<br>
:<br>
[GPU0] Epoch 8<br>
[GPU0] Epoch 9</p>
<p>Is there anyway to confirm DDP takes into account all GPU card results correctly?<br>
The python env is python 3.11.2, torch 2.0.1, cuda 11.7.<br>
Thank you for your help.</p>

Joeyoung: <p>When printing the model parameter values after opitmizer.step in each batch, it prints out the same values from all ranks. So, DDP ring-allreduce algorithm seems to work. But, it is still not clear how it prints out sequentially all epochs for each rank.</p>
Where to begin developing custom backend for torch.compiler?

anubane: <p>I am looking for references to understand how dynamo works and how exactly can I write a custom backend for <code>torch.compiler</code>. For example, when we invoke the following function:</p>
<pre><code class="lang-python">torch.compiler.list_backends()
</code></pre>
<p>provides the following details:</p>
<pre data-code-wrap="console"><code class="lang-plaintext">['cudagraphs', 'inductor', 'onnxrt', 'openxla', 'openxla_eval', 'tvm']
</code></pre>
<p>How can I add a custom backend to this list? How to begin?</p>

marksaroufim: <p>We are removing most of those backends out of core in early december see this RFC for more details <a href="https://github.com/pytorch/pytorch/issues/109687" class="inline-onebox">[RFC]: Moving most torch.compile backends out of core by 12/1/23 · Issue #109687 · pytorch/pytorch · GitHub</a></p>
<p>To write a custom backend follow the docs here <a href="https://pytorch.org/docs/main/torch.compiler_custom_backends.html" class="inline-onebox">Custom Backends — PyTorch main documentation</a> and here’s a case study of Hidet doing this <a href="https://pytorch.org/blog/introducing-hidet/" class="inline-onebox">Introducing Hidet: A Deep Learning Compiler for Efficient Model Serving | PyTorch</a></p>
IterableDataset on multiple files

Rafi: <p>I have 1000 files, each file is processed with sliding window to generate many training sample.<br>
I write this iterable-style dataset. Everything is working, the only thing is each batch consists of data from 1 file. Is there a way to implement a custom collate function or something similar to merge data from different workers?</p>
<pre><code class="lang-auto">class TextFileDataset(IterableDataset):
              def __init__(self, file_pathes: List,) -&gt; None:
                  self.file_pathes = file_pathes
                 
          
                  self.file_id_map= {file_path: idx for idx,
                                      file_path in enumerate(iterable=file_pathes)}
                 def process_file(self, file_path) -&gt; None:
                       doc_indices = self.load_data(file_path=file_path)
                       idx= self.file_id_map[file_path]
                       for i in range(len(doc_indices)):
                           # run a sliding window
                           ..........
                           yield {"idx": torch.tensor(idx), "text1": torch.tensor(text1), "text2": torch.tensor(text2)}
              def __iter__(self) -&gt; None:
                  worker_info = torch.utils.data.get_worker_info()
                  if worker_info is None:
                      for file_path in self.file_pathes:
                          yield from self.process_file(file_path=file_path)
  
                  per_worker = int(np.ceil(len(self.file_pathes) /
                                   float(worker_info.num_workers)))
                  worker_id = worker_info.id
                  self.iter_file_paths = self.file_pathes[worker_id *
                                                          per_worker:(worker_id + 1) * per_worker]
                  for file_path in self.iter_file_paths:
                      yield from self.process_file(file_path=file_path)
</code></pre>
Onnx-tf -> com.microsoft::FusedMatMul operator not implemented

MG-EuS: <p>Hello,</p>
<p>I need to convert a model that is implemented in pytorch to tensorflow lite. I have exported the model to onnx using <code>torch.onnx.export()</code>. The resulting onnx model works very well with the onnx-runtime. But when I try to use onnx-tf to convert the model to tensorflow, I get an error.</p>
<pre><code class="lang-auto">import onnx
from onnx_tf.backend import prepare

onnx_model = onnx.load("my_model.onnx")
tf_rep = prepare(onnx_model)
tf_rep.export_graph('./tf_export/my_model')
</code></pre>
<p>The error message is</p>
<pre><code class="lang-auto">...
raise BackendIsNotSupposedToImplementIt("{} is not implemented.".format(

    BackendIsNotSupposedToImplementIt: FusedMatMul is not implemented.
</code></pre>
<p>So it tells me that the operation com.microsoft::FusedMatMul is not supported by onnx-tf.</p>
<p>What can I do from here? Can I add operators from com.microsoft to onnx-tf? Can I tell the pytorch onnx exporter not to use operations from a specific domain like com.microsoft? Or can I substitute the unsupported operation?</p>
<p>Any ideas are welcome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="MG-EuS" data-post="1" data-topic="191313">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mg-eus/48/63041_2.png" class="avatar"> MG-EuS:</div>
<blockquote>
<p>So it tells me that the operation com.microsoft::FusedMatMul is not supported by onnx-tf.</p>
</blockquote>
</aside>
<p>You might want to post this question to the TF discussion board as you would find TF experts there.</p>

MG-EuS: <p>yes, you are right. I will do that.</p>
<p>So the pytorch specific part of my questions is:<br>
<em>Can I tell the pytorch onnx exporter not to use operations from a specific domain like com.microsoft?</em></p>

ptrblck: <p>I don’t know if the PyTorch export process decides what exactly will be fused as it seems ONNXRuntime defines it <a href="https://github.com/microsoft/onnxruntime/blob/ceb17f82ffc9f72acb64f28629d7db93d98e5ba9/onnxruntime/core/optimizer/matmul_transpose_fusion.cc#L190">here</a>, so PyTorch might not be in control about the optimizations ONNXRT does.</p>

MG-EuS: <p>but is the onnx runtime used by pytorch for producing the .onnx file? I thought the onnx runtime is only used for inference? I thought pytorch only relies on <a href="https://github.com/onnx/onnx" rel="noopener nofollow ugc">onnx</a> for exporting the .onnx file?</p>

ptrblck: <p>I don’t know since I cannot find any reference to <code>FusedMatMul</code> in <a href="https://github.com/search?q=repo%3Aonnx%2Fonnx+FusedMatMul&amp;type=code"><code>onnx/onnx</code></a>.</p>
Pytorch GPU utilization does not match model number of parameters

EloyAnguiano: <p>I’m running occupancy tests with an empty GPU, and when I move the following PyTorch module to the GPU (cuda:1), it occupies <code>843 MiB</code> according to NVIDIA SMI:</p>
<p>I can’t make sense of it; the module is trivial.</p>
<p>This happens even with a freshly created script that creates a Linear layer and moves it to the GPU:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; Linear(1, 3).to("cuda:1")
Linear(in_features=1, out_features=3, bias=True)
</code></pre>
<p>Can anyone explain why such a simple operation results in such high GPU memory usage?</p>

ptrblck: <p>The first CUDA operation will initialize the CUDA context containing the driver, kernels, etc., and thus the memory usage is expected. All PyTorch binaries shipping with CUDA&gt;=11.7 enabled lazy loading which lazily loads all needed kernels but reduces the context size.</p>

EloyAnguiano: <p>Okey, this was just a small trial on a bigger error I have. I’ve got a model that is <code>0.615MB</code> and the GPU ocupation when I put it on the GPU is <code>1689MiB</code>. Does it seem feasible to you? How could I see what are those 1689MiB on the device?</p>

ptrblck: <p>You can use <code>torch.cuda.memory_summary()</code> to check PyTorch’s memory usage and <code>nvidia-smi</code> to check which processes consume GPU memory. The CUDA context will not be shown in <code>memory_summary()</code>.</p>
Why multiply sqrt(d_model) before TransformerEncoderLayer?

yuzhenmao: <p>Hi,</p>
<p>Thank you so much for providing the tutorial! I notice that in <a href="https://github.com/pytorch/tutorials/blob/main/beginner_source/transformer_tutorial.py#L92" rel="noopener nofollow ugc">https://github.com/pytorch/tutorials/blob/main/beginner_source/transformer_tutorial.py#L92 </a>, you multiply sqrt(d_model) before TransformerEncoderLayer. May I ask why we need to do this?</p>
<p>Thanks!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db402b494318890461c5dcdb637a04d9d14418f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/db402b494318890461c5dcdb637a04d9d14418f6" title="Screenshot 2023-11-07 at 10.48.30 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db402b494318890461c5dcdb637a04d9d14418f6_2_690x319.png" alt="Screenshot 2023-11-07 at 10.48.30 AM" data-base62-sha1="vhA2XdhOx6mZZzTexVAjTqqMXga" width="690" height="319" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db402b494318890461c5dcdb637a04d9d14418f6_2_690x319.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db402b494318890461c5dcdb637a04d9d14418f6_2_1035x478.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db402b494318890461c5dcdb637a04d9d14418f6_2_1380x638.png 2x" data-dominant-color="16191E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-11-07 at 10.48.30 AM</span><span class="informations">1382×640 68.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Use scheduler with specific param_group of an optimizer

ptrch_c_m: <p>Hello,</p>
<p>I create an optimizer with different <code>lr</code>s for different parts of the model</p>
<pre><code class="lang-auto">optimizer = AdamW(
    [
        {'params': model.rnn.parameters(), 'lr': 3e-3},
        {'params': model.clf.parameters(), 'lr': 3e-4}
    ]
)
</code></pre>
<p>and I want to use a scheduler just for the RNN part of the model.</p>
<p>A possible solution is to create two optimizers but the code gets less clean. Is it possible to use a scheduler just for the RNN part without creating two optimizers?</p>
How can I insert a node of built-in call_function type into fx.graph.Graph?

shengchunnan: <p>There is an example on this page <a href="https://pytorch.org/docs/stable/fx.html" rel="noopener nofollow ugc">here</a> inserting a new node of <strong>call_function</strong> type into <strong><code>fx.graph.Graph</code></strong>.</p>
<pre><code class="lang-auto"># Specifies the insertion point. Any nodes added to the
# Graph within this scope will be inserted after `node`
with traced.graph.inserting_after(node):
    # Insert a new `call_function` node calling `torch.relu`
    new_node = traced.graph.call_function(
        torch.relu, args=(node,))

    # We want all places that used the value of `node` to
    # now use that value after the `relu` call we've added.
    # We use the `replace_all_uses_with` API to do this.
    node.replace_all_uses_with(new_node)
</code></pre>
<p>The functional callable is <strong><code>torch.relu</code></strong> which can be explicitly mentioned in the above python script. However, in many cases <strong>call_function</strong> types are built-in types that cannot be explicitly mentioned. In the following example, I’d like to gather specific channels from a 4-D tensor:</p>
<pre><code class="lang-auto">class GatherExample (nn.Module):
    def __init__(self) -&gt; None:
        super(GatherExample, self).__init__()
        self.channel_idxs = [0, 2, 3, 5, 10]

    def forward(self, x):
        out = x[:, self.channel_idxs]

        return out
</code></pre>
<pre><code class="lang-auto">opcode         name     target                       args                                              kwargs
-------------  -------  ---------------------------  ------------------------------------------------  --------
placeholder    x        x                            ()                                                {}
call_function  getitem  &lt;built-in function getitem&gt;  (x, (slice(None, None, None), [0, 2, 3, 5, 10]))  {}
output         output   output                       (getitem,)                                        {}
</code></pre>
<p>I can indirectly get an instance of the built-in callable type by running this small piece of module definition here. But this way is still a bit messy for me. So, is there a better way to insert or delete these built-in <strong>call_function</strong> types in <strong><code>torch.fx</code></strong>?</p>

Nick_Young: <p>you should use <code>call_module</code> instead</p>
<pre><code class="lang-auto">gm = symbolic_trace(model)
gm.add_module("gather_op", GatherExample())

with gm.graph.inserting_after(ori_node):
     new_node = gm.graph.call_module("gather_op")
</code></pre>

alaskra: <p>getitem can be import from operator</p>
<pre><code class="lang-auto">from operator import getitem
traced.graph.call_function(getitem, args=(node, idx))
</code></pre>
Export with torch.export for Android

adonnini: <p>Hi,<br>
I produced an exported model with <code>torch.export</code>.<br>
If I save it using torch.export.save can I use the model file for inference in my Android application?</p>
<p>Put another way what is the equivalent of<br>
<code>open("tfmodel.pte", "wb").write(.exir.capture().to_edge().to_executorch().buffer)</code><br>
when using <code>torch.export</code>?</p>
<p>Thanks</p>

Martin_Yuan: <p><code>torch.export</code> is the first step of lowering to ExecuTorch. The serialized file can only be loaded and executed by Python, which may not be suitable for Android app, where Python may not be available. Any reason you prefer to run exported graph directly from Android?</p>

adonnini: <p>Hi Martin,</p>
<p>Thanks. I tried <code>torch.export</code> only as an experiment because<br>
<code>exit.capture</code> is deprecated and I was told that eventually exir<br>
will use <code>torch.export</code>.<br>
In my Android app I am using a neural network to help predict the next<br>
location a user will go to.<br>
Python is not available on Android.<br>
This is why I first used torchscript and pytorch mobile, which worked.<br>
Now I am using executorch as I understand that executorch will be the<br>
main vehicle for running pytorch modles o edge devices.</p>

Martin_Yuan: <p>Got it. Then you could follow the example in this tutorial: <a href="https://pytorch.org/executorch/stable/tutorials/export-to-executorch-tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Exporting to ExecuTorch Tutorial — ExecuTorch documentation</a></p>
RHEL 8 installing Pytorch Cuda with glibc error (after upgrading glibc)

rglenn: <p>I upgraded my glibc to 2.28. I am trying to install pytroch-cuda.<br>
<em>conda install -c pytorch pytorch-cuda</em><br>
I get the error :<br>
<em>feature:/linux-64::__glibic==2.28=0</em><br>
python==3.9.0-&gt;libgcc-ng[version=&gt;‘7.3.0’]-&gt;__glibc[version’&gt;=2.17’]</p>
<p>I using<br>
RHEL 8.8<br>
python 3.9<br>
cuda 12.0<br>
cudnn 8.9.3<br>
nvidia drivers 525.125</p>
<p>What can I do to fix this issue? Is the RHEL version? Should I use an older version of pytorch? Is it my cuda version?</p>
Torch doesn't detect GPU on Azure VM

tomwagstaff-opml: <p>Hi everyone,</p>
<p>It’s my first time trying to use torch with a GPU and I’m struggling.</p>
<p>I’m using an Azure Virtual Machine (NCasT4_v3-series) which requires manual configuration as detailed here: <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/windows/n-series-driver-setup" rel="noopener nofollow ugc">https://learn.microsoft.com/en-us/azure/virtual-machines/windows/n-series-driver-setup</a>. This was apparently successful.</p>
<p>Then I had to download and install CUDA 12.1 from here: <a href="https://developer.nvidia.com/cuda-downloads" rel="noopener nofollow ugc">https://developer.nvidia.com/cuda-downloads</a>. This also seemed to work.</p>
<p>Finally I installed miniconda and installed an existing environment (I’m transferring an existing project to this VM). I then duly ran the code I found on pytorch dot org slash get-started slash locally i.e. <code>conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia</code> - which also ran successfully.</p>
<p>But after all that, torch.cuda.is_available() returns False.</p>
<p>I’m at a loss for what to do next or any possible troubleshooting steps. Any help would be much appreciated.</p>

tomwagstaff-opml: <p>I’ve fixed this now, having found on logging back in that there was a problem with the NVIDIA drivers. Undoing all the manual installations and adding the NVIDIA extension via the Azure portal solved it.</p>
Divergence of convolutional neural network?

TriKri: <p>Given a convolutional neural network f that takes an image as input and outputs another image of the same dimension, is it somehow possible to calculate the <a href="https://en.wikipedia.org/wiki/Divergence" rel="noopener nofollow ugc">divergence</a> of f for a specific input image (the sum of each partial derivative of an output pixel value with respect to the corresponding input pixel value), without having to evaluate the network once per output pixel?</p>