akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can not install torchvision 0.5.0 using conda

username: xu555
cooked: <p>I used the command on the <a href="https://pytorch.org/get-started/previous-versions/#:~:text=%23%20CUDA%2010.1%0Aconda%20install%20pytorch%3D%3D1.4.0%20torchvision%3D%3D0.5.0%20cudatoolkit%3D10.1%20%2Dc%20pytorch" rel="noopener nofollow ugc">pytorch website</a>.</p>
<pre><code class="lang-auto"># CUDA 10.1
conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch
</code></pre>
<p>But it returned</p>
<pre><code class="lang-auto">PackagesNotFoundError: The following packages are not available from current channels:

  - torchvision==0.5.0
</code></pre>
<p>Any suggestions? Thank you!!!</p>

username: AbdulsalamBande
cooked: <p>It seems that you are trying to install PyTorch 1.4.0 and torchvision 0.5.0 with CUDA 10.1 support. It’s important to note that these are relatively old versions of PyTorch and torchvision. If you don’t specifically require these older versions, I recommend installing the latest versions instead.</p>
<p>For the latest PyTorch and torchvision versions with CUDA 10.1 support, you can use the following command:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio -c pytorch -c conda-forge
</code></pre>
<p>However, if you still need to install the older versions (PyTorch 1.4.0 and torchvision 0.5.0) with CUDA 10.1 support, you can try the following command:</p>
<pre><code class="lang-auto">conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch -c conda-forge
</code></pre>
<p>By adding the <code>-c conda-forge</code> channel, you may have better luck finding the required packages. If you still encounter issues, consider creating a new conda environment and then installing the packages within that environment:</p>
<pre><code class="lang-auto">conda create -n myenv
conda activate myenv
conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch -c conda-forge
</code></pre>
<p>Remember to replace <code>myenv</code> with a name of your choice for the new conda environment.</p>

username: mfernezir
cooked: <p>I’m also trying to install those specific versions on Linux. It’s some old repo that has it as a dependency.</p>
<p><code>conda search torchvision -c pytorch -c conda-forge</code> doesn’t list the 0.5 version.</p>

username: No_Yeah
cooked: <p>I did it but got the same error. If i not describe specific version, I got CPU version torch and torchvision</p>

username: Julia_Zhou
cooked: <p>I finally solve the problem by using pip instead of conda</p>
<pre><code class="lang-auto">pip install torch==1.4.0 torchvision==0.5.0
</code></pre>
Are arithmetic operations supported on `nn.Module` instances?

username: omarfoq
cooked: <p>Hello,</p>
<p>Is there a native support in pytorch for arithmetic operations on <code>nn.Modules</code> objects? Say for example we want to compute the difference between two nn.Modules having the same architectures, currently I am looping over the parameters of the model and update them iteratively. I don’t think this is the optimal way to do it, since it requires running a for loop, which can be very slow on python. Does anyone have a better approach to do this?</p>

username: tal123
cooked: <p>Check <a href="https://pytorch.org/docs/stable/optim.html#weight-averaging-swa-and-ema" rel="noopener nofollow ugc">Stochastic Weight Averaging</a></p>
How do you load datasets bigger than RAM?

username: Letmehome
cooked: <p>I got 25 datasets (data as .mtx format (it contains transposed scipy.csr sparse matrix), and label data as .txt format), when I make datasets as normal matrix, total datasets size is quite large,  larger than my RAM. (&gt; 100GB)</p>
<p>I want to use all datasets as train data.</p>
<p>The first thing I could think of was to replace each .mtx with a matrix file, save it to disk, shuffle the 25 matrix files so that there is no bias during training, and then call the dataset one at a time.</p>
<p>However, this method takes up additional disk memory, so I’m wondering if there is a better way to do it.</p>
<p>Is there a better way to do this?</p>
Mixed Precision Training and Quantisation Aware Training together

username: iamexperimentingnow
cooked: <p>Hi,</p>
<p>I’m just wondering whether it is possible to perform mixed precision training and quantisation aware training together?</p>
<p>I’m working on image classification model with DDP approach. I tried mixed precision training separately and quantisation aware training separately.</p>
<p>Thought to get clarification before I implement that.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/suraj.pt">@suraj.pt</a> <a class="mention" href="/u/alband">@albanD</a></p>

username: supriyar
cooked: <p>there is a known issue with AMP + QAT as mentioned in <a href="https://github.com/pytorch/pytorch/issues/94371" class="inline-onebox" rel="noopener nofollow ugc">QAT + torch.autocast does not work with default settings, missing fused fake_quant support for half · Issue #94371 · pytorch/pytorch · GitHub</a>. There is a workaround that you can use though to get it to work by using <code>version=0</code> in <code>get_default_qat_qconfig_mapping</code> as mentioned in the issue.</p>
A100 showing false cuda vultr server

username: AmirC
cooked: <p>Hi I am using a vultr a100 server and using the pip command:<br>
pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu121</a></p>
<p>everytime i import torch and print(torch.cuda.is_available()) it prints false.</p>
<p>How do i make it true? Im using anaconda python enviroment.</p>

username: ptrblck
cooked: <p>The wheel works on my A100 system so could you post the full install log shown after the <code>pip install</code> command?</p>
2D grid positional encoding

username: avalon1511
cooked: <p>Hey everyone,</p>
<p>I have a 100 x 100 grid where certain (x,y) coordinates have a 32 dimension feature vector. I want to find the 2D positional encoding for this grid and add that to the feature vector at that position. So, for each x,y coordinate:</p>
<p>new feature = original feature + PE(x,y)</p>
<p>but  I am confused about the implementation from: <a href="https://github.com/tatp22/multidim-positional-encoding" class="inline-onebox" rel="noopener nofollow ugc">GitHub - tatp22/multidim-positional-encoding: An implementation of 1D, 2D, and 3D positional encoding in Pytorch and TensorFlow</a>.</p>
<p>The input has to be of the form <code>(batch size, x, y, ch)</code> where <code>ch = 32</code> and <code>x, y</code> are the coordinates. But, the calculated PE is also of size <code>(batch size, x, y, ch)</code>. How  do I extract the PE for certain positions in this case?</p>
<p>Thanks a lot</p>

username: J_Johnson
cooked: <p>Currently traveling in China and can’t access Github. But I do recall seeing an implementation in PyTorch on Stable Diffusion with the UNet module by OpenAI.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/Stability-AI/StableDiffusion">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/Stability-AI/StableDiffusion" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/bc0f3c4fd86052aa7832b5bcedecbd68ad06c299a48ec814a3f2a33207f370ca/Stability-AI/stablediffusion" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/Stability-AI/StableDiffusion" target="_blank" rel="noopener nofollow ugc">GitHub - Stability-AI/stablediffusion: High-Resolution Image Synthesis with...</a></h3>

  <p>High-Resolution Image Synthesis with Latent Diffusion Models - GitHub - Stability-AI/stablediffusion: High-Resolution Image Synthesis with Latent Diffusion Models</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>You need to find a folder called ldm and one more called modules or submodules and the folder for OpenAI. There should be a positional encoder that covers 2d and possibly generalized for 1d and 3d.</p>

username: avalon1511
cooked: <p>Thanks a lot for this! Appreciate it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
[Reporting Bug] Expected String but got Blob Torchvision 0.15.2

username: Jenelle_Millison
cooked: <pre><code class="lang-auto">import torchvision

img = torchvision.io.read_image('&lt;path&gt;.png', torchvision.io.ImageReadMode.RGB)
</code></pre>
<p><strong>RuntimeError: isString() INTERNAL ASSERT FAILED as “lib/python3.11/site-packages/torch/include/ATen/core/ivalue_inl.h”:2289, please report a bug to PyTorch. Expected String but got Blob</strong></p>
<pre><code class="lang-auto">torchvision.__version__
'0.15.2'
torch.__version__
'2.1.0+cu121'
</code></pre>
<p>From <a href="https://discuss.pytorch.org/t/reporting-bug-expected-string-but-got-blob/170127">this</a> post, installed libpng by running</p>
<pre><code class="lang-auto">sudo apt-get install libpng-dev
</code></pre>

username: ptrblck
cooked: <p>Thanks for reporting the issue. Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> so that the code owners can track and fix it, please?</p>

username: Jenelle_Millison
cooked: <p>Sure. Just created the issue. Thank you.</p>

username: ptrblck
cooked: <p>Great, thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Network digital twin for cybersecurity

username: xXAkutabi_Xx
cooked: <p>Hi all,<br>
for a text work of mine I am trying to do a project based on generating digital twin of networks. My goal is to create a digital twin of a network and then work on it from a cyber security point of view. I will briefly explain what I would like to do.<br>
I am currently using software for network vulnerability scans (OpenVAS). I use this software to perform network vulnerability scans at the network level, so basically to OpenVAS I pass a network (for example 192.168.xx.xx/24) to automatically identify all the vulnerabilities that are there.<br>
The next step ( what I’d like to do and that’s why I’m asking for your advice) is to create a digital twin of the newly scanned network and then perform a penetration test on this digital twin of the network, without going to stress the actual network.<br>
Ideally, I would like to pass the output of the OpenVAS vulnerability scans, routing rules, and firewall rules to some tool that will then generate for me the digital twin of the network, which will then be used for offensive cybersecurity, so exploits, privilege escalation, etc… will be tested on this digital twin without worrying about breaking some kind of service or stressing the real network.<br>
What I am asking is, do you know of any tool that would do the trick for me? So some tool that allows me to generate a digital twin of a network by providing as input vulnerability scans (xml,json,csv etc…), routing rules, firewall rules, pcap traces etc…<br>
Do you have any references or documentation?<br>
Are you aware of any open source tools?<br>
I thank you for your helpfulness!</p>

username: ptrblck
cooked: <p>It seems your question is unrelated to PyTorch so you might want to ask it in another discussion board.</p>
Most efficient way to run model on gpu while optimizing input

username: Quasiordnung
cooked: <p>Hello everyone,</p>
<p>I want to to optimize the input of a model, here the somewhat pythonic pseudocode:</p>
<pre><code class="lang-auto">var = torch.rand()
optim = Adam(var)

x= model1(var)

for i in range(10):
  with torch.autocast(device_type="cuda",dtype=torch.float16):
    x = model2(x)

with torch.autocast(device_type="cuda",dtype=torch.float16):
  x = model3(x)

loss = x.sum()
loss.backward()
otpim.step()
</code></pre>
<p>The problem being that (at some step of the for loop) the gpu memory overflows. When using torch.no_grad() the complete forward pass fits on the gpu .The question being, for this specific type of optimization architecture, does anybody know a blog post or has any clue on how this can be implemented efficiently while being easy on the gpu memory? Is it e.g. possible to do the forward pass on the gpu and get gradients on the cpu?</p>
<p>Thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>You could apply CPU-offloading as described <a href="https://discuss.pytorch.org/t/does-the-pytorch-has-a-tool-to-convert-data-from-gpu-to-cpu-and-from-cpu-to-gpu-automatically-when-the-gpu-memory-is-not-enough/186612/2">here</a> or reduce the number of iterations, if possible.</p>
How to predict the memory required to store a graph for higher order gradients

username: leifjacobson
cooked: <p>Is it possible to predict the memory required to store the graph needed if I call <code>torch.autograd.grad</code> with create_graph=True?</p>
<p>Here is a concrete example.  Suppose I have a simple neural net with some layer sizes 100 x 10 x 1 so my feature dimension is 100 and I output a scalar.  I want to add the gradients of the output with respect to my inputs to my loss.  Let’s further suppose I have 50 examples of features so my input is of shape (50, 100), let’s call this x.  I then compute</p>
<pre><code class="lang-auto">x = torch.rand(50, 100).to(device).requires_grad_(True)
predictions = model(x)
gradients = torch.autograd.grad(predictions.sum(), x, create_graph=True, retain_graph=True)
</code></pre>
<p>How much memory does each call need?</p>

username: soulitzer
cooked: <p>It would depend on the operators, like if my function in forward is squaring, my backward would be linear and hence would not need to save any tensors!</p>

username: leifjacobson
cooked: <p>See concrete example given.  I feel the general case involves non-linearities.</p>
"Break" the forward function using hook

username: malioboro
cooked: <p>I want to “break” the inference if a condition is met and just return a random output, and I want to do it without changing the code in the forward function.</p>
<p>So, something that came up in my mind is using a forward hook. But is there any function/API to “break” the forward function via hook? Or is there any better approach to do it?</p>

username: soulitzer
cooked: <p>Going off your hook idea, you could raise an error inside the forward hook for your module and catch it out side.</p>

username: malioboro
cooked: <p>Ah… you are right… I believe it will work. Thank you!</p>
Heat map for misclassified image

username: alihamza
cooked: <p>i am getting this error<br>
Traceback (most recent call last):<br>
File “C:\Users\Ali\PycharmProjects\pythonProject1\heatMap.py”, line 629, in <br>
heatmap_overlay = cv2.addWeighted(np.array(image), 0.7, np.array(heatmap), 0.3, 0)<br>
TypeError: Expected cv::UMat for argument 'src<br>
from captum.attr import IntegratedGradients</p>
<h1><a name="function-to-generate-attributions-using-integrated-gradients-1" class="anchor" href="#function-to-generate-attributions-using-integrated-gradients-1"></a>Function to generate attributions using Integrated Gradients</h1>
<p>def generate_integrated_gradients_attributions(model, input_image, target_label):<br>
integrated_gradients = IntegratedGradients(model)<br>
attributions = integrated_gradients.attribute(input_image, target=target_label)<br>
return attributions</p>
<h1><a name="example-usage-2" class="anchor" href="#example-usage-2"></a>Example usage:</h1>
<h1><a name="load-and-preprocess-the-image-3" class="anchor" href="#load-and-preprocess-the-image-3"></a>Load and preprocess the image</h1>
<p>image_path = “I:\K_Folds\PlantDoc\Fold1\Testing\AP_C\AP_C_72.jpg”<br>
image = Image.open(image_path)<br>
input_image = heatmap_transform(image)<br>
input_image = input_image.unsqueeze(0).to(device)</p>
<h1><a name="define-your-model-4" class="anchor" href="#define-your-model-4"></a>Define your model</h1>
<h1><a name="model-yourmodelclass-5" class="anchor" href="#model-yourmodelclass-5"></a>model = YourModelClass()</h1>
<p>model.to(device)<br>
model.eval()</p>
<h1><a name="get-the-true-label-for-the-image-replace-with-your-label-6" class="anchor" href="#get-the-true-label-for-the-image-replace-with-your-label-6"></a>Get the true label for the image (replace with your label)</h1>
<p>true_label = 0</p>
<h1><a name="generate-attributions-using-integrated-gradients-7" class="anchor" href="#generate-attributions-using-integrated-gradients-7"></a>Generate attributions using Integrated Gradients</h1>
<p>attributions = generate_integrated_gradients_attributions(model, input_image, target_label=true_label)</p>
<p>heatmap = viz.visualize_image_attr(attributions[0].cpu().numpy(),<br>
original_image=input_image[0].cpu().numpy(),<br>
method=“heat_map”)[0]  # Extract the heatmap from the tuple<br>
heatmap_overlay = cv2.addWeighted(np.array(image), 0.7, np.array(heatmap), 0.3, 0)</p>
<h1><a name="display-the-original-image-and-the-heatmap-overlay-8" class="anchor" href="#display-the-original-image-and-the-heatmap-overlay-8"></a>Display the original image and the heatmap overlay</h1>
<p>plt.subplot(1, 2, 1)<br>
plt.imshow(image)<br>
plt.title(‘Original Image’)<br>
plt.subplot(1, 2, 2)<br>
plt.imshow(heatmap_overlay)<br>
plt.title(‘Heatmap Overlay’)</p>

username: ptrblck
cooked: <p>Based on the error message it seems to be an OpenCV related error.<br>
If you get stuck, post a properly formatted, minimal, and executable code snippet reproducing the error.</p>
Multiprocessing leak, only when 2 models are running on the same computer (different gpus)

username: apytorch
cooked: <p>I was tracing down a memory leak that kept showing up when 2 models are running on the same computer, but training on 2 different gpus (with 2 workers each for the data loaders). That’s when I learned that I should only be using numpy arrays or pytorch tensors for all instance variables inside the workers, instead of python objects. Once I re-wrote all of that code, the leak is much smaller, but still there. BUT, only when 2 models are running on the same machine. When one model is running, there is no leak. I suspect it has to do with the multiprocessing, because there is no leak for 2 models if i set the num_workers to 0.</p>
<p>To clarify, each model is trained using the same code, independently (i.e., running one script per model).</p>
<p>1 model: no leak (+0GB at 10+ days)<br>
2 models: ~ +2GB/day of ram usage</p>
<p>I don’t have a clue how to chase this down, because there is both a leak and no leak. Any suggestions?</p>
<p><strong>New info:</strong><br>
So, I’m testing setting pin_memory=False to both my train and evaluation dataloaders. The first thing that happened is with an epoch it complains about running out of shared memory for the workers (nothing else changed). And, right now the ram usage doesn’t seem to be growing. Will check back in 12hrs to see if that remains true.</p>
<p><strong>Update:</strong><br>
So far it looks that that’s the conclusion: pinning the memory of two jobs on the same machine leads to leaks. I’ve found some vague mentions of over-allocating using cuda pinned memory, but no hard and fast rules. With one job there should be somewhere between 0.4GB and 1.6GB pinned at any one time (I’m assuming the pinning queue allows any ready workers to start pinning). Which only means 3.2GB for two jobs. On a machine with 96GB (and at least 15GB free), that seems like way too small to be hitting some allocation limit.</p>
Pytorch Extension issue

username: albertotono
cooked: <p>I am trying to build this extension <a href="https://github.com/daerduoCarey/PyTorchEMD" class="inline-onebox" rel="noopener nofollow ugc">GitHub - daerduoCarey/PyTorchEMD: PyTorch Wrapper for Earth-Mover-Distance (EMD) for 3D point cloud regression</a> but I am getting an errors with CUDA 11.1 and Pytorch cuda 11.3. and Pytorch cuda10.2</p>
<pre><code class="lang-auto">The detected CUDA version (11.1) mismatches the version that was used to compile
PyTorch (10.2). Please make sure to use the same CUDA versions.
</code></pre>
<p>I think I had this issue because I was using Python 3.6 . Then I got</p>
<pre><code class="lang-auto">cuda/emd_kernel.cu(385): error: identifier "TORCH_CHECK" is undefined
cuda/emd_kernel.cu(391): error: no suitable constructor exists to convert from "c10::ScalarType" to "at::Type"
6 errors detected in the compilation of "cuda/emd_kernel.cu".
error: command '/usr/bin/nvcc' failed with exit status 1
</code></pre>
<p>Moving to Python 3.8, and Installing Pytorch cuda 11.3 I run into these messages:</p>
<pre><code class="lang-auto">line 1694, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'
IndexError: list index out of range
</code></pre>
<p>So I used <strong><code>TORCH_CUDA_ARCH_LIST="7.5" python setup.py install</code></strong> but I got this</p>
<pre><code class="lang-auto">error: THC/THC.h: No such file or directory
   14 | #include &lt;THC/THC.h&gt;
      |          ^~~~~~~~~~~
compilation terminated.
</code></pre>
<p>I am not sure how could I install Pytorch extension properly, compiling them correctly, or where can I learn more about it outside of this page <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#conclusion" class="inline-onebox" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions — PyTorch Tutorials 1.12.1+cu102 documentation</a></p>

username: albertotono
cooked: <p><a href="https://github.com/daerduoCarey/PyTorchEMD/issues/6" class="inline-onebox" rel="noopener nofollow ugc">Updated CUDA and PyTorch versions · Issue #6 · daerduoCarey/PyTorchEMD · GitHub</a> I think this solved it</p>

username: maxiuw
cooked: <aside class="quote no-group" data-username="albertotono" data-post="1" data-topic="161042">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/albertotono/48/21596_2.png" class="avatar"> albertotono:</div>
<blockquote>
<p><strong><code>TORCH_CUDA_ARCH_LIST="7.5" python setup.py install</code></strong></p>
</blockquote>
</aside>
<p>I had the same issue</p>
<aside class="quote no-group" data-username="albertotono" data-post="1" data-topic="161042">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/albertotono/48/21596_2.png" class="avatar"> albertotono:</div>
<blockquote>
<pre><code class="lang-auto"> arch_list[-1] += '+PTX'
IndexError: list index out of range
</code></pre>
</blockquote>
</aside>
<p>and this worked for me for me</p>
Pytorch script model in C++ is slow on prediting the second image

username: 3yuan
cooked: <p>When using script model in C++ for image detection task, it spends about 2 seconds to predict the first image, and more than 60 seconds for the second image, and then less than 200 milliseconds (as we execpted) for the rest images.<br>
I have no idea why it spend too much time on the second image. Any ideas? Thanks</p>

username: ptrblck
cooked: <p>In case you are using the GPU, did you synchronize the code before starting and stopping the timers?<br>
If so, are you seeing the slowdown only in the first iterations (this would be expected, if e.g. <code>cudnn.benchmark</code> is used or in case the JIT is optimizing the graph)?</p>

username: 3yuan
cooked: <p>Only in the first two iterations in my experiment</p>

username: 3yuan
cooked: <p>Is there any method of optimizing the graph in advance?</p>

username: ptrblck
cooked: <p>I don’t think there is currently a way to store the optimized graph and if I’m not mistaken the first iterations would then rerun the optimization (but <a class="mention" href="/u/tom">@tom</a> would know and correct me, if I’m wrong).</p>

username: Xavier31
cooked: <p>Hi. I also have experienced this behavior with the same time gaps (second iteration x30 slower than the first one) but only on Windows. The problem does not seem to exist on Linux</p>

username: Xavier31
cooked: <p>in case anyone is still experiencing this, if your input is of fixed size, you can disable optimization and prevent unwanted compilations by using</p>
<pre><code class="lang-auto"> torch::jit::FusionStrategy static0 = { {torch::jit::FusionBehavior::STATIC, 0} }; 
 torch::jit::setFusionStrategy(static0); 
 torch::jit::setGraphExecutorOptimize(false);
</code></pre>
Guidance with model grayscale images

username: pcgmont
cooked: <p>Hello,</p>
<p>First of all, I’m new to deep learning and Pytorch, so I apologise in advance for my question.</p>
<p>I have a model that consists in 3 conv2D layers, and ReLU activations. It takes as an input grayscale images, where the dynamic range is between [0,255], that are normalised to the interval [0,1]. The input images have some black regions, some white ones, and others in between.</p>
<p>However, the outputs dynamic range is compressed to [0.4,0.41]. All images are grey, even after a renormalisation to bring them back to [0,255].</p>
<p>I’m a bit lost and I don’t find why this is the case.</p>
<p>What I tried:</p>
<ol>
<li>Plot histograms of gradients, although I’m not sure how to interpret this. It seems that for some parameters the model stops learning after a few epochs. For others one layer stops learning.</li>
<li>Modify kernel sizes</li>
<li>Add more layers</li>
<li>Try quite a lot of different values of learning rate</li>
</ol>
<p>Any help is appreciated. Please let me know if more information is needed. And if this is not the place to ask this kind of questions, please let me know where I can get some help.</p>

username: alixbird
cooked: <p>Can you share your code?</p>

username: pcgmont
cooked: <p>Thank you for taking the time to help!</p>
<p>This is the model:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.model = nn.Sequential(
           nn.Conv2d(in_channels=1, out_channels=64, kernel_size=13, stride=1, padding=5),
           nn.ReLU(True),
           nn.Conv2d(in_channels=64, out_channels=32, kernel_size=3, stride=1, padding=2),
           nn.ReLU(True),
           nn.Conv2d(in_channels=32, out_channels=1, kernel_size=5, stride=1, padding=2),
          )


    def forward(self, input_image):
        output_image = self.model(input_image)
        return output_image

</code></pre>
<p>Regarding my training script, for each epoch I do the following:</p>
<pre><code class="lang-auto">def train_one_epoch(
    model: nn.Module,
    dataloader:  torch.utils.data.DataLoader,
    loss_fn: nn.MSELoss,
    optimizer: optim.Adam,
    epoch_index,
    scaler: amp.GradScaler, 
):
   
    model.train()
    running_loss = 0
    last_loss = 0
    batch_index = 0

    for batch, loader in enumerate(train_dataloader):
                
        input_img = loader['input'].to(device, non_blocking=True)
        gt_img = loader['gt'].to(device, non_blocking=True)

        model.zero_grad(set_to_none=True)
      
        with amp.autocast():
            output = model(input_img)
            loss = loss_fn(output, gt_img) 
       
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

        running_loss += loss.item()
        if batch_index % 100 == 99:
            last_loss = running_loss / 1000
            running_loss = 0.0
        
        batch_index += 1

    return last_loss

</code></pre>
<p>Let me know if more information is needed</p>
Torch tensor == torch.amax(tensor, (-2,-1)) error

username: Matteo_Ciotola
cooked: <p>I wondered how to find a maximum index along 2 different dimensions.<br>
I followed the suggestions of the thread: <a href="https://discuss.pytorch.org/t/get-indices-of-the-max-of-a-2d-tensor/82150" class="inline-onebox">Get indices of the max of a 2D Tensor</a></p>
<p>But I have got a no deterministic problem:</p>
<p>xx = torch.rand((1, 8, 512, 512, 9, 9))<br>
pos_x = (xx == torch.amax(xx, (-1, -2), keepdim=True)).nonzero(as_tuple=True)</p>
<p>print(len(pos_x[5]))</p>
<blockquote>
<blockquote>
<p>2097155</p>
</blockquote>
</blockquote>
<p>print(xx[:,:,:,:,0,0].flatten().shape)</p>
<blockquote>
<blockquote>
<p>torch.Size([2097152])</p>
</blockquote>
</blockquote>
<p>In particular, this size changes at each iteration.</p>
<p>Where am I wrong?</p>

username: ptrblck
cooked: <p>The code works for me and outputs the same shape for each iteration:</p>
<pre><code class="lang-python">xx = torch.rand((1, 8, 512, 512, 9, 9))

for _ in range(10):
    pos_x = (xx == torch.amax(xx, (-1, -2), keepdim=True)).nonzero(as_tuple=True)
    print(len(pos_x[5]))
    print(xx[:,:,:,:,0,0].flatten().shape)
    
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
# 2097154
# torch.Size([2097152])
</code></pre>

username: Matteo_Ciotola
cooked: <p>Maybe I explained it in a bad way, sorry about that.<br>
I would extract the indices that maximize an index, taken on a kernel 9x9.<br>
I expected that len(pos_x[5]) == xx[:,:,:,:,0,0].flatten().shape, being the max calculated on the last two indices. I think that for some repeated maximum this number could change. In some other cases, the length of the list is higher than the length of the flatten tensor.</p>
<p>Nevermind, I have found a workaround:</p>
<p>c = torch.argmax(rho, dim=-1, keepdims=True)<br>
r = torch.argmax(torch.take_along_dim(rho, c, dim=-1), dim=-2, keepdims=True)<br>
c = torch.take_along_dim(c, r, dim=-2)</p>
<p>It seems to work</p>
TypeError: expected Tensor as element 0 in argument 0, but got list

username: Jamilah_Marram_Duart
cooked: <p>def custom_collate_fn(batch):<br>
input_ids = pad_sequence([item[“input_ids”] for item in batch], batch_first=True, padding_value=tokenizer.pad_token_id)<br>
attention_mask = pad_sequence([item[“attention_mask”] for item in batch], batch_first=True, padding_value=1)  # Attention mask should be 1s<br>
labels = pad_sequence([item[“labels”] for item in batch], batch_first=True, padding_value=tokenizer.pad_token_id)<br>
return {“input_ids”: input_ids, “attention_mask”: attention_mask, “labels”: labels}</p>
Can't overfit to training data with ViT

username: alixbird
cooked: <p>I’m using a visual transformer on a small medical imaging dataset (1000 images or so).</p>
<p>Usually my first go to sanity check is to overfit to the training data, but I haven’t had any luck in doing so. I have checked the input data, that the labels all look correct, the learning rate etc.</p>
<p>I saw in another forum that someone had said that you shouldn’t expect to be able to overfit as ViT’s require so many more magnitudes of data.</p>
<p>I’m not looking for advice on how to troubleshoot it not fitting to the training data, but whether or not I should expect it to be able to.</p>
<p>TIA</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="alixbird" data-post="1" data-topic="189716">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/eada6e/48.png" class="avatar"> alixbird:</div>
<blockquote>
<p>someone had said that you shouldn’t expect to be able to overfit as ViT’s require so many more magnitudes of data</p>
</blockquote>
</aside>
<p>It’s news to me that a model requires a lot of data to <em>overfit</em> as the simplest use case is to overfit any model on a single sample.<br>
What is the explanation for this alleged effect?</p>

username: alixbird
cooked: <p>Thanks for your response. There wasn’t a particular explanation, and it all sounded a bit vague. It sounded strange to me too so I think I will persist and try with a small simpler dataset to see if I can get it working.</p>
<p>Thanks a lot</p>
Understanding of output from the language modeling tutorial

username: dvijay
cooked: <p>I’m trying to understand the output from the model in the “language modeling with nn.transformer and torchtext” tutorial (<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and torchtext — PyTorch Tutorials 2.1.0+cu121 documentation</a>). I am basically trying to run part of a text and trying to decode the response from the model in plain text.</p>
<p>line = "This game was a lot of "<br>
line_t = torch.tensor(vocab(tokenizer(line)), dtype=torch.long)<br>
output = model(line_t.to(device))<br>
output_flat = output.view(-1, ntokens)</p>
<p>In this case the output shape is [6, 6, 28782] and flattened shape is [36, 28782]. 29782 is the vocabulary size. Since the input shape is [6] (for the six words), I’m not sure why the output shape is [6, 6, 28782]. How do I decode/translate this output into text? From what I understand, the model is expected to predict the next word but I’m not sure how to identify that from the above output.</p>
<p>I hope that makes sense. Any help is appreciated!</p>
Docker Image Performance Drop between 23.07 to 23.09

username: vgoklani
cooked: <p>Hey there,</p>
<p>We’re seeing a large performance drop when going from:</p>
<pre><code>nvcr.io/nvidia/pytorch:23.07-py3
</code></pre>
<p>to</p>
<pre><code>nvcr.io/nvidia/pytorch:23.08-py3
</code></pre>
<p>We assumed it would be addressed in September (it’s the summer after all) but we’re seeing the same drop with this image:</p>
<pre><code>nvcr.io/nvidia/pytorch:23.09-py3
</code></pre>
<p>In particular, when we train Llama2 using the 23.07 image (which is CUDA 12.1) we’re seeing approx 3,500 tokens per second while training, and with the 23.08 and 23.09 images, we’re seeing 2,800 tokens per second (CUDA 12.2). We are training on 3x NVIDIA A6000 Ada Lovelace cards.</p>
<p>The only thing we are installing is the base docker image and the Flash Attention module from Tri Dao’s repo.</p>
<p>What’s the best way for us to fix this?</p>
<p>Is there a standard benchmark built in the container that we could run to give us a better comparison?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Could you post a minimal and executable code snippet reproducing the regression?</p>
Segmentation of Organ At Risk using 3D Unet

username: ethan04
cooked: <p>I am CSE student, doing my Minor Project in PyTorch on Segmentation of Organ At Risk using CT Scans and 3D Unet.</p>
<p>I am very new to PyTorch and Deep Learning in general. I have read first 8 chapters of the book “Deep Learning with PyTorch” to learn about PyTorch. I have created a demo jupyter notebook for my project but it has some errors due to which the GPU runs out of memory.</p>
<p>Context of Project In Brief:</p>
<ol>
<li>Takes data from GDrive.</li>
<li>Creates file paths for the volumes and labels</li>
<li>Created two CustomDatasets to handle 3D Volumes and its Patches</li>
<li>Implemented 3D Unet From Scratch.</li>
<li>Created some Metric and Losses functions (a bit of doubt here on which losses and metrics to use for 3D segmentation)</li>
<li>After this the Training Loop</li>
</ol>
<p>I will add MarkDown of these to file for better visibility</p>
<p>I have a few questions :</p>
<ol>
<li>Is my approach good enough?</li>
<li>Is my 3D Unet implementation correct?</li>
<li>What losses and metrics should I consider for 3D segmentation?</li>
<li>Is my training loop correct?</li>
<li>I am running out of GPU? can you please suggest some code changes to tackle it?</li>
<li>How can I improve my code</li>
</ol>
<p>Link to my Jupyter Notebook <a href="https://colab.research.google.com/drive/14G2C28lt885Ru1ZuFe7841jnguT176Wy?usp=sharing" rel="noopener nofollow ugc">Here</a>.</p>
Optimization avoiding loops

username: bltzmann
cooked: <p>Hello to everyone! I’m working with PyTorch 2.1. Given that <code>s1</code> and <code>s2</code> are two tensors of size <code>(self.nrep, self.npt)</code> with indices ranging from 0 to <code>self.size-1</code>, <code>net</code> is a tensor of size <code>(self.size, self.size, K, 4)</code>, and <code>conf</code> is a tensor of size <code>(self.nrep, self.npt, self.size)</code>, i would like to optmize this code avoiding the for loops:</p>
<pre><code class="lang-auto">net_conf = torch.zeros(size=(self.nrep, self.npt, K, 4), dtype=torch.complex128, device=f'cuda:{self.dev}')
for r in range(self.nrep):
    for k in range(self.npt):
        net_conf[r,k] = conf[r,k,net[s1[r,k],s2[r,k]]]
</code></pre>

username: soulitzer
cooked: <p>torch.func’s vmap or functorch’s torch.dim might be useful here</p>
Would PyTorch (for CUDA 12.1) work with CUDA 12.2

username: GrassWarlock
cooked: <p>never used PTorch before and my internet connection is too limited to try cuda 12.2 myself.<br>
should i download cuda 12.1 or cuda 12.2 for it?<br>
and if it doesn’t, when do you expect PyTorch will support CUDA 12.2?</p>

username: ptrblck
cooked: <p>PyTorch binaries ship with their own CUDA dependencies and your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.</p>

username: GrassWarlock
cooked: <p>Alright. By the way do you know if i need to install cuDNN seperately or does it also come with PyTorch?</p>

username: ptrblck
cooked: <p>cuDNN also comes with PyTorch as well as NCCL, cuSPARSE, cuSOLVER, cuBLAS, etc.</p>

username: GrassWarlock
cooked: <p>awesome.<br>
i’ll delete my own versions of cudnn and cuda then (since i only planned to use them with pytorch)<br>
thanks alot!</p>
What is the correct way to source an env for multinode training?

username: hoanchar
cooked: <p>What is the correct way to source an environment (such as conda or venv) for PyTorch multinode training on slurm? Should I source the environment from the main node only? Or do I have to source the environment for all of the nodes that I am training with?</p>

username: wconstab
cooked: <p>as long as you have a shared filesystem between the main node and the runner nodes, i think you can just rely on the ENV propagating from the main node when you launch.</p>
<p>if you have another setup where the runner nodes have the env on their local disks, then you’d need to take care of setting up the ENV correctly on the runners before execution starts.  (You’d have to override the ENV that came from the main node if it pointed to the path of the conda env on the main node)</p>
Multiprocessing data loader with manual numpy seed

username: forgi86
cooked: <p>Hello</p>
<p>I have implemented an <code>IterableDataset</code> that generates synthetic data according to a source of randomness in numpy. I need to control the generation process, thus I manually seed the random source. My code looks like:</p>
<pre><code class="lang-auto">import numpy as np
import torch
from torch.utils.data import IterableDataset, DataLoader

class RandDataset(IterableDataset):
    def __init__(self, seed=42):
        super(RandDataset).__init__()
        self.seed = seed
        self.data_rng = np.random.default_rng(seed)

    def __iter__(self):
        while True:
            yield self.data_rng.normal()
            
def collect_data(loader, epochs=10):
    DATA = []
    for idx, data in enumerate(loader):
        if idx==epochs: break   
        DATA.append(data)
    return DATA
                       

DATA0 = collect_data(DataLoader(RandDataset(seed=42), batch_size=32))
</code></pre>
<p>My actual data-generating procedure is much more CPU-intensive than self.data_rng.normal(). To speed up data generation, I am using a multiprocessing data loader. However, this generates duplicate batches from the two workers.</p>
<pre><code class="lang-auto">dl = DataLoader(RandDataset(seed=42), num_workers=2)
DATA2_rep = collect_data(dl)
torch.all(DATA2_rep[0] == DATA2_rep[1]) # result: tensor(True)
</code></pre>
<p>My current solution is to seed each worker differently according to the worker id.<br>
I ended up doing something like:</p>
<pre><code class="lang-auto">def seed_worked(worker_id):
    worker_info = torch.utils.data.get_worker_info()
    dataset = worker_info.dataset
    worker_id = worker_info.id
    dataset.data_rng = np.random.default_rng(dataset.seed + 1000*worker_id)

dl = DataLoader(MyIterableDataset(seed=42), num_workers=2, worker_init_fn=seed_worked) 
DATA2_norep = collect_data(DataLoader(RandDataset(seed=42), num_workers=2, batch_size=32, worker_init_fn=seed_worked))
</code></pre>
<p>The sequence of generated batches DATA2_norep seems reproducible on my machine, and it does not contain repeated instances.  However, the result depends on the number of workers by design.<br>
My preferred solution would be to have a single shared source of randomness accessed<br>
sequentially by the workers, so that I would get the same sequence with 1 or more workers. Is that possible?</p>
<p>Cheers,<br>
Marco</p>
Timeline of NCCL activity

username: Epsilon_1
cooked: <p>I am following the distributed data parallel (DDP) tutorial on a server with two NVLink’d GPUs. I want to see which PyTorch code leads to NCCL collective calls (Broadcast, Gather etc). Is there a profiler option that lets me do this? So far, I run <code>nsys profile --gpu-metrics-device=all python3 ddp.py</code> to profile my code and then I analyze the generated profile with <code>nsys stats &lt;report&gt;</code>. This however does not give me a timeline of any kind.</p>

username: ptrblck
cooked: <p>Open the profile in <code>nsys-ui</code> and you will see the timeline.</p>
Memory leak from unowned inference

username: rohil
cooked: <p>I have a minimal example that increments GPU usage by ~1GB each time it is run.</p>
<pre><code class="lang-auto">simpleGPT2(**encoded_input)
</code></pre>
<p>Running this line multiple times in a Jupyter notebook increments the GPU utilization as shown by nvidia-smi by ~1GB each time I run it.  This happens even if I run the gc.collect() and torch.cuda.empty_cache(). This is surprising to me because I would expect that since there is no variable referencing the result of the inference, it should be possible to garbage collect it. Any suggestions for how to debug this further?</p>
<p>simpleGPT2 is my implementation of a Transformer:</p>
<pre><code class="lang-auto">class SimpleGPT2(t.nn.Module):
    def __init__(self, n_blocks = 1, vocab_size = 50257, context_length = 1024, hidden_size = 768, p_dropout = 0.1):
        super().__init__()
        self.wte = t.nn.Embedding(vocab_size, hidden_size)
        self.wpe = t.nn.Embedding(context_length, hidden_size)
        self.pe_matrix = t.nn.Parameter(t.arange(0, context_length).unsqueeze(0), requires_grad = False)
        self.dropout = t.nn.Dropout(p_dropout)
        self.gpt_blocks = t.nn.ModuleList([GPTBlock() for _ in range(n_blocks)])
        self.layernorm = t.nn.LayerNorm(hidden_size)
        self.final = t.nn.Linear(hidden_size, vocab_size)

        for layer in [self.wte, self.wpe, self.final]:
            init_layer(layer)
    
    def forward(self, input_ids: t.Tensor, attention_mask = t.Tensor):
        x = input_ids
        n, seq_len = x.shape
        hidden = self.wte(x) + self.wpe(self.pe_matrix.expand(n, -1))
        hidden = self.dropout(hidden)
        for gpt_block in self.gpt_blocks:
            hidden = gpt_block(hidden)
        hidden = self.layernorm(hidden)
        return self.final(hidden)
</code></pre>
<p>and encoded_input is a dictionary storing two Tensors of shape [1, 1024], i.e., the input and attention mask.</p>

username: ptrblck
cooked: <p>Check if you are storing any tensors in e.g. a <code>list</code> which might still be attached to the computation graph such as the model output or the loss. This will not only store the actual tensor but also the entire computation graph thus increasing the memory usage. Detach the tensor before storing it in the <code>list</code> to fix it.</p>