akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
huge dataset memory constraints?

clamalo: <p>I’m training a model in Pytorch to do image-to-image processing. My data is huge, shape is (64152, 3, 5, 2, 64, 144). I’m using memmapping to save on memory as much as humanly possible.</p>
<p>The issues that I’m facing due to the sheer size of this dataset:<br>
1.) shuffling before splitting into train/test sets<br>
2.) normalizing data</p>
<p>Even when memmapping, these operations cause a memory spike that inevitably exceeds the amount of memory I have available on my computer.</p>
<p>I’ve tried memmapping, normalizing along smaller dimensions at a time. Not sure what to try for shuffling the dataset before splitting into train and test sets.</p>

ptrblck: <p>Would it be possible to lazily load the data as it seems you are trying to load the entire array at once (even with mem mapping)?</p>

clamalo: <p>Well, I’m loading the data from a few dozen .npy files, and I have to do some concatenation and normalization with the data first. So I load them all in, concatenate them, and try to normalize, but both those operations take longer than expected and take up much more memory than expected considering the memmapping… am I doing something wrong?</p>
Understanding 3D Conv Feature Maps

Gautier30: <p>Hi community <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I am using 3D Conv layers in a network where the inputs are stacked images of a subject at 3 points in time. Across the network I keep the depth at 3 although the spatial dimensions of the images are reduced by pooling layers. Visualizing the feature maps for each activation, I was expecting to see patterns relating to each of the 3 slices individually, but it looks like all feature maps are superimpositions of the 3 slices.</p>
<p>For instance, on the feature map indexed depth 0 of the first convolution layer, I can see details that look like the input image at depth 3. Is that normal behavior and is my intuition about 3D conv layers completely wrong?</p>
<p>Thanks!</p>

ptrblck: <p><code>nn.Conv3d</code> layers will use kernels with 3 volumetric dimensions and will thus also perform the convolution in these 3 dimensions. I.e. if your kernel size of <code>[out_channels, in_channels, depth, height, width]</code>, the <code>depth</code> dimension of the kernel will be applied to the <code>depth</code> of the activation input. For values of <code>depth&gt;1</code> neighboring voxels are used (in the same way as neighboring pixels are using for <code>height,width&gt;1</code> in an <code>nn.Conv2d</code>).<br>
Let me know if I misunderstood your use case.</p>
How to locate part of network that leads to exploding gradients?

AlphaBetaGamma96: <p>Hi All,</p>
<p>Are there any go-to strategies in trying to locate the part of the network that leads to exploding gradients?</p>
<p>There exists a similar function for locating <code>nans</code>, with <code>torch.autograd.set_detect_analomy(True)</code>. However, there is no equivalent for locating exploding gradients. I know I could just ‘clip’ the gradients with <code>clip_grad_norm_</code>, however, that doesn’t solve the problem, but only masks it.</p>
<p>Any advice on debugging this problem would be apprecitated.</p>

JuanFMontesinos: <p>You can simply register the gradients for each layer (max min or whatever you want). You just need to iter through the parameters after backwards and compute any metric that is good for you. Some libraries such us pytorch lightning (I think) do this for you automatically.</p>
Decorator @torch.compiler.disable() backward compability

orena1: <p>Hi,</p>
<p>I have a model that I want to compile, but I need to disable compilation for some functions inside the class. I use <span class="mention">@torch.compiler.disable</span>() to disable compilation for these functions, and an argparse parameter to decide whether to attempt compilation.</p>
<p>The issue arises when using an older version of PyTorch that does not support torch.compile. I get an error because torch.compile does not exist in those versions. How can I use this decorator while maintaining backward compatibility? Note that the user should not use the compile parameter with the old PyTorch version, but the decorator still causes an issue.</p>
<p>Thanks</p>

ptrblck: <p>PyTorch does not guarantee forward compatibility, i.e. running new code in older PyTorch releases, but backwards compatibility, i.e. older code should run fine in newer PyTorch releases.<br>
You could check the <code>torch.__version__</code> inside the script and raise an error asking users to either update PyTorch or to remove the <code>compile</code> usage (you could also try to use <code>compile</code> conditionally based on the version).</p>
How to write my own LearningRate function

Farshid1: <p>hey guys<br>
I want to define my own LR<br>
the code in tensorflow is like:</p>
<pre><code class="lang-auto">class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
  def __init__(self, d_model, warmup_steps=4000):
    super().__init__()

    self.d_model = d_model
    self.d_model = tf.cast(self.d_model, tf.float32)

    self.warmup_steps = warmup_steps

  def __call__(self, step):
    step = tf.cast(step, dtype=tf.float32)
    arg1 = tf.math.rsqrt(step)
    arg2 = step * (self.warmup_steps ** -1.5)

    return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)
</code></pre>
<p>how can I write this code in pytorch format (chatGPT’s didn’t work!)</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/farshid1">@Farshid1</a>,</p>
<p>You could have a look at the existing LR Schedulers in the docs <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.StepLR.html#torch.optim.lr_scheduler.StepLR" rel="noopener nofollow ugc">here</a> and view the source of these LR schedulers as well <a href="https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html#StepLR" rel="noopener nofollow ugc">here</a>. This would you give you a rough idea of how to construct your own custom LR scheduler.</p>
Tips on creating images with StyleGAN2-ADA

nitay_shlump: <p>It’s my first time using StyleGAN2-ADA to create synthetic images. I’m currently training it using 1350 images but I’m still unsure about some things.</p>
<p>While training, the augment value rises. When it reached 0.6, I stopped the training. Later I wanted to resume training but the augment value started from 0 again. Is that fine, or should I change it to where it was before?</p>
<p>Also, if you have experience with the model, please share some tips on how to attain the best results. Thank you.</p>
Linking libtorch in my library

Rahul3527: <p>I am using cmake project and trying to link libtorch in my library but seems to have issues in linking with “cmake --target install”.<br>
In CmakeLists.txt I have<br>
<code>target_link_libraries(${MY_LIBRARY_NAME} PRIVATE "${TORCH_LIBRARIES}")</code><br>
Torch libraries are : torch;torch_library; $LIBTORCH_PATH/lib/libc10.so;$LIBTORCH_PATH//lib/libkineto.a</p>
<p>Added libtorch include directories</p>
<pre><code class="lang-auto">target_include_directories(${MY_LIBRARY_NAME} 
PUBLIC
"$&lt;BUILD_INTERFACE:${TORCH_INCLUDE_DIRS}&gt;")
</code></pre>
<p>But when I use My library in example test I get undefined reference symbol from linc10.so</p>
<pre><code class="lang-auto">undefined reference to symbol 
'_ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'
lib/libc10.so: error adding symbols: DSO missing from command line
</code></pre>
<p>Any help is well appreciated.</p>
<p>Thanks</p>

Anqin: <p>I got the same error. have you solved it?</p>

ptrblck: <p>Check if the right C++ ABI is used in <code>libtorch</code>.</p>

Anqin: <p>Thanks，have fixed it now</p>
Avoiding blowup of Gradient Tape Memory for Accumulated Tensors

sad_robot: <p>Hi, I’ve got to re-run a variational inference deep learning model (using <a href="https://arxiv.org/abs/1505.05424" rel="noopener nofollow ugc">reparameterization trick</a>) many times before each backwards step to get accumulated prediction metrics…</p>
<p>But for simplicity I’ll ask about this toy problem: <strong>Is it possible change the code below such that <code>mean_output</code> retains valid gradients but also the gradient memory footprint doesn’t grow with additional samples?</strong></p>
<pre><code class="lang-auto">batch = next(iter(train_loader))
batch_inputs = batch['x'].cuda()
mean_output = 0
n_samples = 10000 # I want gradient tape memory to remain constant with more samples
for i in range(n_samples):
    mean_output += stochastic_model(batch_inputs)/n_samples
</code></pre>
<p>P.S. A key point is that the accumulation of samples doesn’t store intermediate tensors.</p>
How to access the implementation of the computational graph nodes?

mohamadaltalli: <p>I want to access the implementation of the computational graph representation of a function I am differentiating using <code>functorch.jacfwd</code> (or alternatively, using <code>torch.Tensor.backward</code>). I know that I can use <code>.grad_fn</code>, <code>.next_functions</code>, and <code>.variable</code> to inspect the graph structure, but that is not what I want. I am looking to access the actual implementation (probably in C++) of the different nodes in the computational graph. Is that something that is possible?</p>
Torch.abs() backpropgation problem

AMIRA_BEDOUI: <p>Hello everyone,<br>
I have a model in which I do this operation <code>A = A_1/ torch.abs(A_1 + epsilon)</code><br>
A_1 is a complex matrix, and I figured out that the division by  <code> torch.abs(A_1 + epsilon)</code> prevent the model from converging, otherwise when omitting this division the model converge without problem. same issue goes for another tensor B that has been multiplied like this <code>(1/(n*s))*B</code>, where ‘n’ and ‘s’ are two real numbers.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/amira_bedoui">@AMIRA_BEDOUI</a>,</p>
<p>The derivative of the function <code>torch.abs(x)</code> is ill-defined when <code>x = 0</code> as the derivative w.r.t <code>x</code> is <code>x / torch.abs(x)</code> whereas the derivative of <code>x</code> w.r.t <code>x</code> is just 1. So, check the value of <code>A_1 + epsilon</code> throughout the convergence.</p>
Set longer timeout for torch distributed training

ralzq01: <p>Hi, I wonder if there is a mechanism to synchronize all processes with unlimited waiting time.</p>
<p>The scenario is in distributed training where one of processes in each node needs to deal with some CPU-related tasks, while other processes keep waiting until finish. Currently I use <code>torch.distributed.barrier()</code> (with nccl backend) and find it will timeout in half an hour. Is there another way that can make synchronize on all processes without timeout limitations? It would be better if torch already provides something like this.</p>

wanchaol: <p>Could you try <a href="https://pytorch.org/docs/stable/distributed.html?highlight=monitored_barrier#torch.distributed.monitored_barrier" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.13 documentation</a> instead?</p>

weichenrs: <p>Worked for me! Thank you!</p>

level1807: <p>this is not a solution to the question. <code>monitored_barrier</code> is implemented only for GLOO, whereas the question specifically states they’re using NCCL.</p>

lilamonroe: <p>Hello,<br>
I try this and it really worked, Thanks for sharing.</p>

ralzq01: <p>Hello, I found <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="noopener nofollow ugc">init_process_group</a> works for my scenario. You can customize the timeout with <code>timeout</code> argument. Then all the NCCL operations under this group will be with this timeout.</p>
Lstm model cannot run on gpu using libtorch, cnn is ok

Jas: <p>I use libtorch2.3.1+cuda12.1+vs2022 on windows11, and I convert two models, cnn and lstm, from python with torch.jit.trace.  The cnn model can run normally on GPU, the lstm model cannot, only can run on cpu.  Why?</p>
<p>and  I found that once the model contain the lstm, it cann’t run on gpu in vs c++ environment.</p>

vdw: <p>Are you about just using/calling <code>nn.LSTM</code> or a complete training?</p>
<p>For example, compared to a CNN, you typically need to initialize a hidden state for the LSTM. If that’s true in your case, do you move this hidden state to the same service as your model?</p>

Jas: <p>Thanks. I tried initial the LSMT’s  hidden state, but it does work.<br>
I check my python code, I found the problem. When I transfer the model from python using the torch.jit.trace, I also send the model.to(“cpu”), and this is the problem. I change it to model.to(“cuda”), it is ok for GPU run.<br>
But for cnn, the problem doesnot  exist, the model.to(“cpu”) and torch.jit.trace transfer the model also can run on GPU.</p>
Parallel computation of forward for multiple networks

yuchiq: <p>I have multiple independent neural networks that take different inputs, and I will need their output concatenated together. What might be the way to avoid loop in <code>forward()</code>. Particularly, if the set of networks have the same architectures and their sizes are small, there must be a way to “stack” all networks into one, and run the forward function one time for all. But any simple way to do it? Thanks.</p>
<p>My current forward() function with loop is (<a href="http://self.net" rel="noopener nofollow ugc">self.net</a> and x are two lists for networks and inputs):</p>
<pre><code class="lang-auto">def forward(self, x):
    output = [self.net[i](x[i]) for i in range(len(self.net))]
</code></pre>
CPU libtorch fails to find CUDA headers

ZealanL: <p>I am attempting to link to libtorch in my CMake project. I explicitly downloaded a CPU version of libtorch with no CUDA support (<a href="https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.3.1%2Bcpu.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.3.1%2Bcpu.zip</a>).</p>
<p>However, when trying to build and link/include the library, it fails because it cannot find CUDA headers (which it shouldn’t need because this is not a CUDA version).</p>
<pre><code class="lang-auto">C:\Users\[redacted]\libtorch\include\c10\cuda\CUDAStream.h(3): fatal error C1083: Cannot open include file: 'cuda_runtime_api.h': No such file or directory
</code></pre>
<p>This header file is part of CUDA itself. I have triple-checked that the version of the <code>\libtorch\</code> is indeed <code>2.3.1+cpu</code>, and is from the download.</p>

ptrblck: <p>Are you seeing the same issue using any of the tutorials? If so, could you link to it or post a minimal and executable code to reproduce the build issue?</p>

ZealanL: <p>Alright, I found the issue.<br>
My project was explicitly including:<br>
<code>#include &lt;c10/cuda/CUDACachingAllocator.h&gt;</code><br>
to allow manual freeing of CUDA cache (following memory clogging issues).</p>
<p>This header is still present in the CPU libtorch version for some reason, but from my testing, is normally not ever included/compiled. If you do include it from CPU torch without CUDA installed, the actual CUDA header includes from libtorch will unsurprisingly fail.<br>
So it definitely is an issue on my part. Sorry for bothering!</p>
How to know which triton version to build for a given pytorch version

ahmedius2: <p>I am trying to make the inductor backend of torchdynamo work on Jetson AGX Orin (aarch64 iGPU system). I have to use torch version 2.3, and I compiled triton v2.1 and also the main branch from the source and installed it. However, both have compatibility issues, resulting in errors like no such package found triton.common etc. How can I know which branch or commit of triton I should compile to make it work with torch 2.3 (or any other version)?</p>

ptrblck: <p>You can find the Triton commit in <a href="https://github.com/pytorch/pytorch/blob/v2.3.0/.ci/docker/ci_commit_pins/triton.txt"><code>.ci/docker/ci_commit_pins/triton.txt</code></a>.</p>

ahmedius2: <p>I was having trouble checking out that commit. I realized that I needed to fetch it like this:<br>
git fetch origin 79c6c9b209a5692b9a895398f4f3a033f8f80415<br>
git checkout 79c6c9b209a5692b9a895398f4f3a033f8f80415</p>
<p>Thanks.</p>
Quantization for Retinanet

tdavid18: <p>Are there any guides or resources for using PyTorch’s quantization tools with the Retinanet models provided in torchvision? Or any of the object detection models in general? I’ve only seen guides for the classification models.</p>

MyCenturaHealth: <p>Seeking PyTorch quantization assistance for Retinanet in torchvision lacking specific guidance beyond classification models.</p>
Parameter-Efficient Fine-Tuning for CNN

vivekstorm: <p>Dear community,</p>
<ol>
<li>
<p>Is there some Parameter Efficient Fine-Tuning technique like LORA(low rank adaptation) is available for CNN to reduce the GPU memory usage while training/fine-tuning the network?</p>
</li>
<li>
<p>Is it possible to apply LoRA for CNN kernels which will be of (3X3, 5X5, 7X7)?</p>
</li>
</ol>
<p>Any article recommendations will be highly helpful. Thanks</p>

minhnh: <p>It seem like Microsoft has written LoRAConv layer, you can find it <a href="https://github.com/microsoft/LoRA/blob/main/loralib/layers.py" rel="noopener nofollow ugc">here</a>.</p>

Raman_Dutt: <p>I implemeted a ton of PEFT methods for both CNNs and ViTs. Check my paper here - <a href="https://arxiv.org/abs/2305.08252" class="inline-onebox" rel="noopener nofollow ugc">[2305.08252] Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity</a></p>

Mees.elizabeth: <p>Do you have your code somewhere? I’m particularly interested in trying SSF, but it performs super bad so I must be doing something wrong</p>

Raman_Dutt: <p>The official code is here - <a href="https://github.com/dongzelian/SSF" class="inline-onebox" rel="noopener nofollow ugc">GitHub - dongzelian/SSF: [NeurIPS'22] This is an official implementation for "Scaling &amp; Shifting Your Features: A New Baseline for Efficient Model Tuning".</a></p>

Mees.elizabeth: <p>Thank you for your response! I read in your paper that you’ve added SSF to a ResNet50, did you add it to only the ResBlocks? And throughout the entire model? I tried to add it to the resblocks of a segresnet like<br>
` scale, shift = init_ssf_scale_shift(in_channels)<br>
self.ssf_scale = scale<br>
self.ssf_shift = shift</p>
<pre><code>def forward(self, x):
    
    if self.tuning_mode == "ssf":

        identity = x
        
        x = self.norm1(x)
        x = self.act(x)
        x = ssf_ada(self.conv1(x), self.ssf_scale, self.ssf_shift)

        x = self.norm2(x)
        x = self.act(x)
        x = ssf_ada(self.conv2(x), self.ssf_scale, self.ssf_shift)

        x += identity`. 
</code></pre>
<p>Then I only train the params with ‘ssf’ in their name, and this gives very poor results.</p>

MyCenturaHealth: <aside class="quote no-group" data-username="vivekstorm" data-post="1" data-topic="195694" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/fbc32d/48.png" class="avatar"> vivekstorm:</div>
<blockquote>
<p>Dear community,</p>
<ol>
<li>Is there some Parameter Efficient Fine-Tuning technique like LORA(low rank adaptation) is available for CNN to reduce the GPU memory usage while training/fine-tuning the network?</li>
<li>Is it possible to apply LoRA for CNN kernels which will be of (3X3, 5X5, 7X7)?<br>
Any article recommendations will be highly helpful. Thanks</li>
</ol>
</blockquote>
</aside>
<p>Yes, LoRA can be adapted for CNNs to reduce GPU memory usage. Check “LoRA: Low-Rank Adaptation of Large Language Models” for foundational concepts.</p>
Loss evaluation

L_Z: <p>I am trying to evaluate the loss for my training and validation set. But I am not sure what is loss evaluated through loss.backward(). ie. if the loss function is as cross-entropy. Like does it return the averaged loss per batch? So that means I should have times the batch size in order to get the total loss? Or looking at the loss per batch is sufficient enough?</p>

ptrblck: <aside class="quote no-group" data-username="L_Z" data-post="1" data-topic="206219">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/l_z/48/71230_2.png" class="avatar"> L_Z:</div>
<blockquote>
<p>if the loss function is as cross-entropy. Like does it return the averaged loss per batch?</p>
</blockquote>
</aside>
<p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss">docs</a>:</p>
<blockquote>
<ul>
<li><strong>reduction</strong> (<a href="https://docs.python.org/3/library/stdtypes.html#str"><em>str</em></a><em>,</em> <em>optional</em>) – Specifies the reduction to apply to the output: <code>'none'</code> | <code>'mean'</code> | <code>'sum'</code>. <code>'none'</code>: no reduction will be applied, <code>'mean'</code>: the weighted mean of the output is taken, <code>'sum'</code>: the output will be summed. Note: <code>size_average</code> and <code>reduce</code> are in the process of being deprecated, and in the meantime, specifying either of those two args will override <code>reduction</code>. Default: <code>'mean'</code></li>
</ul>
</blockquote>
<p>So by default <code>'mean'</code> will be used.</p>
A confusion in torch.__init__

chao_jiang: <p>in  torch.<strong>init</strong> we can see a lot</p>
<p>from torch.XXX import YYY</p>
<p>but the correct and robust way to build a package should be</p>
<p>from .XXX import YYY</p>
<p>My question is why “In package torch innerly , builder want to import module or funcs by abs import from outside , considering seemingly useless , isnt”</p>
Compile Pytroch missing python312_d.lib

QFuser: <p>I already compiled Pytroch with CUDA , CuDnn and distributed support for release but I want to create also the debug build for using it in a project . I’m using the build_pytroch.bat for it . I already found the normal python build with debug symbols means included the python312_d.lib.</p>
<p>The script using mini anconda just copying the files seems doesn’t work and I can’t find the anconda with the debug infos. Also I actually compile it without python support so I’m not sure where it hangs or where I can find the Anaconda with the debug symbols ?</p>

QFuser: <p>Okay the reason where this problem came from is properly version confusion on my system . I de-installed the other python 3.11 versions and made VC part in the batch file working I mentioned in another script .</p>
<p>Next was cmake which was suddenly on version 3.17 which was too low . Some pointed hardcoded from Visual Studio to old version which weren’t there anymore but with some reinstalling I solved it too . Now the active conda version is/was the one with the old cmake updating caused this :</p>
<blockquote>
<p>LibMambaUnsatisfiableError: Encountered problems while solving:</p>
<ul>
<li>package cmake-3.29.0-hf0feee3_0 requires libuv &gt;=1.48.0,&lt;2.0a0, but none of the providers &gt;can be installed</li>
</ul>
</blockquote>
<blockquote>
<p>Could not solve for environment specs<br>
├─ cmake 3.29.0**  is installable and it requires<br>
│  └─ libuv &gt;=1.48.0,&lt;2.0a0 , which can be installed;<br>
└─ libuv 1.39**  is not installable because it conflicts with any installable versions previously reported.</p>
</blockquote>
<p>I ignored the dependencies not sure if this works but we will find out</p>

QFuser: <p>Okay release works again with all. I also solved python312_d.lib problems but it also demand K python312.lib but it gives the error : LINK : fatal error LNK1104: Datei “python312.lib”</p>
<p>Stackoverflow :</p>
<p>This particular issue is caused by specifying a dependency to a lib file that had spaces in its path. The path needs to be surrounded by quotes for the project to compile correctly.</p>
<p>Funny thing I saw compiler errors not recognized compiler options: " " at the beginning. I think there is a bug in the debug cmake settings . It looks for me someone moved the command out of the quotes. I will make bug report when I find the time</p>
Slow training for transformer with parallel data feed

Matin_Ahmadi: <p>Hi everyone. I am training a transformer model. Since my data is huge, I convert my data to a bunch of files, each containing tensors. Then I use threads to open those files and feed the tensors in those files one by one into a queue. Then a generator over this queue is served as <code>__iter__</code> in my dataset class.<br>
My model has around 40M parameters (4 layers, d_model 512, d_feedforward 2048) and the batches are of size 32. Each batch takes 2.5 seconds using mixed precision training and checkpoints (to prevent <code>CUDA out of memory</code>). So each epoch (~10000 iterations per epoch) takes around 7 hours. And my GPU Is an RTX 4080 with 16G memory.<br>
My take is that this training is slow. I read somewhere that a similar model took 2 days for 40 epochs (whereas mine takes 13 days).<br>
I’m skeptical about the model. Can you provide some insights?<br>
Is this parallel data feeding legit? Am I doing something wrong here?</p>

ptrblck: <p>I would recommend profiling the code first to understand where the current bottleneck is before trying to optimize anything. E.g. if the data loading is indeed the bottleneck, you might want to profile it separately to check if e.g. the multi-threading approach is causing the slow execution time.</p>

Matin_Ahmadi: <p>I don’t think it’s the data loading, since the queue has a max size of 1000 tensors, while each batch is 32. so there’s no waiting on the queue, considering the 2.5 seconds per batch. (Unless threads themselves are somehow magically slowing everything down)<br>
Also, mixed precision training reduces the time from 2.9 to 2.5 seconds.<br>
Is it possible that <code>torch.utils.checkpoint</code> is the culprit?<br>
Or is 2.5 seconds normal for this model with these parameters?</p>

ptrblck: <p>I would recommend profiling your workload instead of guessing to understand which part of the code is slow.</p>

Matin_Ahmadi: <p>I do profile my model. Most of the time is spent in the backward pass, which, given the checkpoints, makes sense. I need to get an idea of whether this 2.5 seconds is normal for this model with ~40M parameters on this GPU.</p>
Error in PyInstaller executable with Pytorch

RAHUL_ROY: <p>I am trying to create an executable with pyinstaller from a python/conda environment. Although it is creating the executable but while running it is coming up with the following error<br>
“OSError: Can’t get source for &lt;class ‘torch_geometric.nn.pool.select.base.SelectOutput’&gt;. TorchScript requires source access in order to carry out compilation, make sure original .py files are available.<br>
[47898] Failed to execute script ‘start’ due to unhandled exception!”</p>
<p>I am using torch 2.0.1, torch geometric 2.5.3 and Pyinstaller 6.5.0</p>
<p>Have tried multiple solutions:-<br>
Ranging from adding files, binaries of torch in .spec files and all. But it is not working.</p>
MPS Back End Out of Memory on GitHub Action

sammlapp: <p>When running a pytest action on GitHub Actions Mac OS, I inconsistently get an error message:</p>
<blockquote>
<p>RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 1.70 GB). Tried to allocate 0 bytes on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).</p>
</blockquote>
<p>from a line running <code>a_tensor.to('mps')</code></p>
<p>If I keep re-running the test it eventually succeeds, but sometimes it takes &gt;5 attempts. I don’t understand  why the error is occurring or how to prevent it. Setting the environment variable <code>PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0</code> as suggested in the error message does not resolve the problem. It occurs, at least occasionally, with Python versions 3.9, 3.10, and 3.11.</p>
<p>The issue does not occur when I run pytest locally on Mac OS with Apple Silicon.</p>
<p>The github action lists the following operating system information:</p>
<pre><code class="lang-auto">macOS

12.6.9

21G726
</code></pre>
<p>The installed pytorch is <code>torch (2.0.0)</code></p>

thorinf: <p>Hi,</p>
<p>Did you find any solution to this? I have the exact same issue with GitHub workflows, but all tests pass on my local macOS.</p>
<p>Thanks</p>

gernophil: <p>Having the same issue (see <a href="https://discuss.pytorch.org/t/mps-backend-out-of-memory/183879/5">here</a> and <a href="https://github.com/pytorch/pytorch/issues/105839#issuecomment-1779116758" rel="noopener nofollow ugc">here</a>), but only since 2.1.0.</p>

sammlapp: <p>I don’t have a solution</p>

janosh: <p>even GH’s new <code>macos-14</code> runners don’t have access to MPS hardware unfortunately, see <a href="https://github.com/pytorch/pytorch/issues/111449#issuecomment-1767613883" rel="noopener nofollow ugc">pytorch/pytorch#111449 (comment)</a>.</p>

michaelfeil: <p>I used to be able to run. Since ~May, the memory seems to be more limited (or a having other issues)</p>
<pre><code class="lang-auto">RuntimeError: MPS backend out of memory (MPS allocated: 0 bytes, other allocations: 0 bytes, max allowed: 7.93 GB). Tried to allocate 44.71 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
</code></pre>

ethanwhite: <p>Switching <code>runs-on</code> from <code>macos-latest</code> (which is running on M1) to <code>macos-13</code> fixed this issue for me. Hopefully they’ll get the issue with the M1 architecture fixed, but this is working for us for now.</p>