akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch-CUDA 11.8 for Jetson Orin AGX

username: rahulswa
cooked: <p>Hello everyone,</p>
<p>I am working on a Jetson orin AGX  32GB platform with jetpack 5.1 installed. I have to updated CUDA on my jetson to 11.8 and updated all my settings to CUDA 11.8 using <a href="https://developer.nvidia.com/blog/simplifying-cuda-upgrades-for-nvidia-jetson-users/" rel="noopener nofollow ugc">link</a></p>
<p>I have checked the installation using <code>nvcc --version</code> which points to CUDA 11.8 and even the soft link to <code>/usr/local/cuda</code> and <code>/usr/local/cuda-11</code> are pointed to <code>/usr/local/cuda-11.8</code></p>
<p>Now I installed pytorch using the instructions given <a href="https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform/index.html" rel="noopener nofollow ugc">here</a></p>
<p>When I type <code>torch.version.cuda</code> it outputs 11.4.  And I’m facing issues with this, because when I try to install <code>pytorch-3d</code>.  Because it says pytorch is build for CUDA-11.4 but pytorch-3d is trying to build for CUDA-11.8.</p>
<h3><a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<p>Pytorch : torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl<br>
Jetpack : 5.1.1<br>
Platform : Jetson AGX Orin 32GB</p>
<p>I’m not sure why I’m facing this issue.  Am I doing anything wrong here? or it this a limitation of this library?</p>
<p>Could anyone help me solve this issue?</p>
<p>Thanks!!</p>

username: ptrblck
cooked: <p>The linked wheel was built with the CUDA compiler shipped in the corresponding JetPack version, which was 11.4. If you need CUDA 11.8 for some reason, you would need to build the PyTorch wheel from source manually.</p>

username: rahulswa
cooked: <p>Where can I find instructions to build from source? Are there any specific set of instructions to build torch on jetson?</p>

username: ptrblck
cooked: <p>You can find the build instructions <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a>.</p>
Unpickling Error for Detectron2

username: RadAlienware
cooked: <p>Hello, I am training a zoo model from detectron2 where I want to use transfer learning so I was adding pretrained weight. The .pth format is working properly but I am unable to use the pretrained weight which are as .pkl file</p>
<p>UnpicklingError: A load persistent id instruction was encountered,<br>
but no persistent_load function was specified.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34e7ddffb58456a368e04fb2b97458c749c140f8.png" data-download-href="https://discuss.pytorch.org/uploads/default/34e7ddffb58456a368e04fb2b97458c749c140f8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34e7ddffb58456a368e04fb2b97458c749c140f8.png" alt="image" data-base62-sha1="7y1BiAYwUw9GI26apO5dCld3c6Y" width="690" height="119" data-dominant-color="F6D7D4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1122×194 15.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried to write a new function but unable to get it work.</p>

username: ptrblck
cooked: <p>It seems you are trying to load the file multiple times.<br>
Would using <code>pickle.load</code> without <code>PathManager.open</code> work?</p>
Do enumerate()/indexing cause a host-GPU sync?

username: yiftach
cooked: <p>Consider the following code:</p>
<pre><code class="lang-auto">import torch
x = torch.arange(100, device='cuda').reshape(50, 2)
y = torch.empty(100, device='cuda').reshape(50, 2)
for i, batch in enumerate(x):
    y[i] = batch ** 2
</code></pre>
<p>My basic knowledge is that CUDA operations run asynchronously on the GPU, and that operations like <code>.item()</code> or <code>print()</code> require a sync for copying them back to the host memory.<br>
Is anything in the above code require such a sync? Specifically I’m worried that either:<br>
(1) using <code>enumerate</code>, which iterates the GPU memory but returns ints on the host memory<br>
(2) using these ints to index an array which resides in GPU memory<br>
require a GPU to host sync (which I would hope to avoid). Is that the case here?</p>
<p>Additionally, how would I verify this on my own?</p>
<p>Thanks,<br>
Yiftach</p>

username: ptrblck
cooked: <p>Your code shouldn’t synchronize the code and you can double check it by adding <code>torch.cuda.set_sync_debug_mode("warn")</code> to your code.</p>
'MPS' Resource Exhaustion Issue Generating -Inf's and Nan's?

username: dbl001
cooked: <p>Running the Transformer Model NanoGPT with ‘MPS’ on MacOS 13.4 with an AMD Radeon Pro 5700 XT,<br>
I started getting -Inf and NaNs after several thousand training iterations.<br>
When I switched the backend from ‘MPS’ to ‘CPU’, there were no -Inf’s or -NaNs.<br>
When I decreased the block_size and the batch_size, the problem stopped.<br>
With the original larger block_size and batch_size, but, saving the tensor in the forward method it stopped the issue (change in timing?)</p>
<p>The -Infs first appeared in the</p>
<pre><code class="lang-auto">LayerNorm: F.layer_norm(input, self.weight.shape, self.weight, self.bias, 1e-5)
</code></pre>
<p>Does this sound like an ‘MPS’ resource issue?<br>
Is there a way to display the resource usage in the Forward method?</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/karpathy/nanoGPT">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/karpathy/nanoGPT" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/006c9738fe59736ddee4ea436c1d7c6c9af7f4c5756ea7c772249893b56cc894/karpathy/nanoGPT" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/karpathy/nanoGPT" target="_blank" rel="noopener nofollow ugc">GitHub - karpathy/nanoGPT: The simplest, fastest repository for...</a></h3>

  <p>The simplest, fastest repository for training/finetuning medium-sized GPTs. - GitHub - karpathy/nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Can not set num_workers>0

username: wickked1204
cooked: <p>I’m running this code:<a href="https://github.com/alibaba-mmai-research/CLIP-FSAR.I" rel="noopener nofollow ugc">https://github.com/alibaba-mmai-research/CLIP-FSAR.I</a> find that if I set num_workers =2 / 4.After training for some time, the child process will be suspended, and the entire python program will be stuck, with no return value and no reaction. I was convinced that my CPU, memory, and GPU were sufficient for this experiment, and I wanted to know what I could do to solve the problem.</p>
CUDA Initialization Issue and Proposed Workaround with PyTorch in wsl2

username: Peter_Deng
cooked: <p>Dear PyTorch and NVIDIA teams,</p>
<p>I am writing to report an unexpected behavior I’ve encountered when working with PyTorch and CUDA on a wsl2 on Windows 11 system equipped with multiple NVIDIA RTX 3090 GPUs.</p>
<p>Environment Details:</p>
<ul>
<li>Operating System: Windows 11</li>
<li>CUDA Version: 12.2</li>
<li>WSL Version: 2</li>
<li>GPUs: 4x NVIDIA RTX 3090</li>
<li>PyTorch Version: 2.01 (CUDA 11.8)</li>
</ul>
<p>Problem Statement: When I set the <code>CUDA_VISIBLE_DEVICES</code> environment variable to enable all the GPUs (0,1,2,3) on the system and then run a PyTorch script that calls <code>torch.cuda.is_available()</code>, I encounter an “Out of Memory” error. Notably, this error does not occur if I only enable GPU 1 or a combination of GPU 0,2,3. Furthermore, this error can be circumvented if I call <code>torch.cuda.device_count()</code> before <code>torch.cuda.is_available()</code>.</p>
<p>Steps to Reproduce:</p>
<ol>
<li>Set the environment variable: <code>export CUDA_VISIBLE_DEVICES=0,1,2,3</code></li>
<li>Run a Python script that imports PyTorch and calls <code>torch.cuda.is_available()</code></li>
</ol>
<p>Expected Behavior: The <code>torch.cuda.is_available()</code> function should return <code>True</code> if GPUs are available and accessible.</p>
<p>Observed Behavior: An “Out of Memory” error is triggered internally at <code>../c10/cuda/CUDAFunctions.cpp:109</code>. The <code>torch.cuda.is_available()</code> function returns <code>False</code>.</p>
<p>Workaround: I found that calling <code>torch.cuda.device_count()</code> before <code>torch.cuda.is_available()</code> circumvents the error. However, this workaround requires modifying each script to include this extra call.</p>
<p>While the workaround is effective, it may be beneficial to investigate and address the root cause of this issue. I wanted to bring this to your attention and look forward to any insights or potential solutions you might provide.</p>
<p>Thank you for your time and assistance.</p>
<p>Best Regards, Peter Deng, Language and Technology Research Group at TC, Columbia</p>
GPU Cuda out of memory error

username: Karl_Roberts
cooked: <p>I am getting the following error message in the vscode app, when I run a jupyter notebook, using python 3.11.3 as a jupyter kernel:</p>
<p>‘“name”: “OutOfMemoryError”,<br>
“message”: “CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 8.00 GiB total capacity; 7.00 GiB already allocated; 0 bytes free; 7.13 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF”’</p>
<p>I ran the following iterative tests, disabling each GPU one by one:</p>
<p>Test 1.<br>
NVIDIA GeForceRTX 2080 1 disabled, 2, 3 and 4 enabled. Run the segment/mask step in the ‘Segment Anything Model’ jupyter notebook in VSCODE. Run successful, no errors.</p>
<p>Test 2.<br>
NVIDIA GeForceRTX 2080 2 disabled, 1, 3 and 4 enabled. Run the segment/mask step in the ‘Segment Anything Model’ jupyter notebook in VSCODE. Run successful, no errors.</p>
<p>Test 3.<br>
NVIDIA GeForceRTX 2080 3 disabled, 1, 2 and 4 enabled. Run the segment/mask step in the ‘Segment Anything Model’ jupyter notebook in VSCODE. Run unsuccessful, CUDA Out of memory error (‘Test 3 error’ attached).</p>
<p>Test 4.</p>
<p>NVIDIA GeForceRTX 2080 4 disabled, 1, 2 and 3 enabled. Run the segment/mask step in the ‘Segment Anything Model’ jupyter notebook in VSCODE. Run unsuccessful, Unknown error ‘Test 4, 5 and 6 error’ attached.</p>
<p>Test 5.</p>
<p>NVIDIA GeForceRTX 2080 1 disabled, 2, 3 and 4 enabled. Run the segment/mask step in the ‘Segment Anything Model’ jupyter notebook in VSCODE. Run unsuccessful, Unknown error ‘Test 4, 5 and 6 error’ attached.</p>
<p>Test 6.</p>
<p>NVIDIA GeForceRTX 2080 2 disabled, 1, 3 and 4 enabled. Run the segment/mask step in the ‘Segment Anything Model’ jupyter notebook in VSCODE. Run unsuccessful, Unknown error ‘Test 4, 5 and 6 error’ attached.</p>
<p>To me these results don’t point to an issue with a specific GPU, but instead to a problem that is exacerbated as the number of runs increases. Perhaps a memory cache issue… although I have integrated a line to clear the cache, and the error persists. Has anyone come across this before?</p>
<p>Thanks,</p>
<p>Karl</p>
TypeError: unsupported operand type(s) for +: 'diceLoss' and 'focalTverskyLoss'

username: jeremy_eudaric
cooked: <p>Hello many thanks for your support l am trying add two loss functions but l got this error message. They 2 loss functions has been made my myself in 2 .py file.</p>
<p>TypeError                                 Traceback (most recent call last)<br>
Cell In[20], line 3<br>
1 epochs=10<br>
2 for epoch in range(1,epochs+1):<br>
----&gt; 3       training(epoch)</p>
<p>Cell In[19], line 78, in training(epochs)<br>
72   loss2 = focalTverskyLoss(output,label)<br>
74   <span class="hashtag-raw">#label</span> =  torch.randn(3) .softmax(dim=0)<br>
75<br>
76   <span class="hashtag-raw">#loss3</span> =torch.nn.CrossEntropyLoss()(output,label)<br>
—&gt; 78   total_loss =  (loss1 + loss2)<br>
80   total_loss.backward()<br>
82 # Update Weight</p>
<p>TypeError: unsupported operand type(s) for +: ‘diceLoss’ and ‘focalTverskyLoss’</p>
<p>my two loss functions :</p>
<p>class focalTverskyLoss(_Loss):<br>
def <strong>init</strong>(self, weight=None, size_average=True):<br>
super(focalTverskyLoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA, gamma=GAMMA):
    
    #comment out if your model contains a sigmoid or equivalent activation layer
    inputs = F.sigmoid(inputs)       
    
    #flatten label and prediction tensors
    inputs = inputs.view(-1)
    targets = targets.view(-1)
    
    #True Positives, False Positives &amp; False Negatives
    TP = (inputs * targets).sum()    
    FP = ((1-targets) * inputs).sum()
    FN = (targets * (1-inputs)).sum()
    
    Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  
    FocalTversky = (1 - Tversky)**gamma
                   
    return FocalTversky
</code></pre>
<p>class diceLoss(_Loss):<br>
def <strong>init</strong>(self, weight=None, size_average=True):<br>
super(diceLoss, self).<strong>init</strong>()</p>
<pre><code>def forward(self, inputs, targets, smooth=1):
    
    #comment out if your model contains a sigmoid or equivalent activation layer
    inputs = F.sigmoid(inputs)       
    
    #flatten label and prediction tensors
    inputs = inputs.view(-1)
    targets = targets.view(-1)
    
    intersection = (inputs * targets).sum()                            
    dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)  
    loss=1 - dice

    return loss
</code></pre>

username: ptrblck
cooked: <p>based on the error message you are initializing <code>loss1</code> and <code>loss2</code> as loss functions:</p>
<pre><code class="lang-python">loss1 = diceLoss(output, label)
loss2 = focalTverskyLoss(output, label)
</code></pre>
<p>which is wrong, since <code>output</code> and <code>label</code> would be passed to the <code>__init__</code> method and interpreted as <code>weight</code> and <code>size_average</code>, respectively.<br>
You need to create the objects first and call them afterwards:</p>
<pre><code class="lang-python">criterion1 = diceLoss()
criterion2 = focalTverskyLoss()

loss1 = criterion1(output, label)
loss2 = criterion2(output, label)
</code></pre>
What are possible reasons for BUS errors in DDP?

username: nmamie
cooked: <p>Hi,</p>
<p>What could be possible reasons for encountering a BUS error in DDP? My guess would be an OOM error but as far as I can see RAM should be sufficient for the task. Could the shuffling of the data play a role when creating a torch dataloader? This is my implementation:</p>
<pre><code class="lang-auto">data = DataLoader(
        dataset,
        batch_size=batch_size,
        pin_memory=True,
        shuffle=False,
        num_workers=0,
        sampler=DistributedSampler(dataset)
    )
</code></pre>
Compute the gradient of functions for many data points

username: Namjoon_Suh
cooked: <p>Hi all. Thanks for your help in advance.<br>
I am trying to compute the gradient of log probability with respect to the data X.<br>
The probability is computed through sigmoid of the output of some neural networks.<br>
Here, I use the network “prob_ScoreNet” that I implemented and freeze the model parameters.<br>
Following is the code that I am trying to run.</p>
<blockquote>
<p>for param in prob_ScoreNet.parameters():<br>
param.requires_grad = False<br>
X = torch.tensor(X, requires_grad = True)<br>
class_prob = prob_ScoreNet(X, time_step.unsqueeze(0))<br>
function = torch.log(F.sigmoid(class_prob)+1e-10)<br>
gradients = torch.autograd.grad(function, X)[0]</p>
</blockquote>
<p>The data X is of size [100000, 8] and keep facing the memory allocation problem.<br>
I don’t know how to fix this, and not sure if the above code is correct.</p>
Image normalisation in preprocessing

username: robertvava
cooked: <p>Apologies if this has been discussed before, but so far I couldn’t find an answer that answers my “why”. So I have the following transformation:</p>
<p>transformation = transforms.Compose([<br>
transforms.Resize((244,244)),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
])</p>
<p>Pretty standard. However, I am confused about the negative values. I have been told that I shouldn’t work with negative values. Moreover, this is an example of an image that I get from this transform:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/99aafe14be38078ac765eb8b98ff888a2ee18c4f.jpeg" alt="image" data-base62-sha1="lVpoY5AxyY83Qk1g2jaBA2cHKZF" width="500" height="500"></p>
<p>I know that it doesn’t have to make sense to the human eye, but I can’t imagine that the pixel values here are appropriate for a machine learning pipeline.</p>
<p>Could someone explain why this is actually fine to work with (if it is)?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="robertvava" data-post="1" data-topic="183585">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/robertvava/48/61937_2.png" class="avatar"> robertvava:</div>
<blockquote>
<p>but I can’t imagine that the pixel values here are appropriate for a machine learning pipeline.</p>
</blockquote>
</aside>
<p>Could you explain your concern and why your neural network won’t be able to use normalized values (containing positive and negative values)?</p>
<p>The normalization will subtract the mean and divide by the stddev to crease a standardized input, which contains the same “information” but in another range, which is usually helping the model training.</p>
Can't use torch.hann_window in ScriptModule

username: vbtraz
cooked: <p>I’m learning jit.script. I got error when I use torch.hann_window.<br>
Code:</p>
<pre><code>import torch
class MyModule(torch.nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()

    def forward(self) -&gt; torch.Tensor:
        return torch.hann_window(5)

if __name__ == "__main__":
    script_module = torch.jit.script(MyModule())
    print(script_module())
</code></pre>
<p>The error message is:</p>
<pre><code>Traceback (most recent call last):
  File "D:\Coding\Work\test_jit.py", line 15, in &lt;module&gt;
    print(script_module())
  File "D:\PublicVENV\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "D:\Coding\Work\test_jit.py", line 10, in forward
    def forward(self) -&gt; torch.Tensor:
        return torch.hann_window(5, True)
               ~~~~~~~~~~~~~~~~~ &lt;--- HERE
RuntimeError: result type Float can't be cast to the desired output type Long
</code></pre>
<p>How can I use hann_window in jit script ? Thanks !!</p>

username: ptrblck
cooked: <p>Could you create a <a href="https://github.com/pytorch/pytorch/issues">GitHub issue</a> for this error, as I would expect <code>torch.hann_window</code> to work in the same way as e.g. <code>torch.randn</code> (the latter works in your code snippet).</p>

username: miblue119
cooked: <p>If we set the <code>dtype</code> at the calling, it could scripted correctly</p>
<pre><code class="lang-auto">import torch
class MyModule(torch.nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()

    def forward(self) -&gt; torch.Tensor:
        return torch.hann_window(5,dtype=torch.float32)

if __name__ == "__main__":
    script_module = torch.jit.script(MyModule())
    print(script_module())
</code></pre>
Update internal variable across different forward pass

username: new_user
cooked: <p>How can i update an existing variable across different inferences in a computational graph (a memory variable that gets updated for every input during a forward pass) making sure that the gradients are computed correctly for that variable?</p>
<p>For example, I want to do self.x = self.x + input every inference, where self.x is an internal variable for a layer.</p>
<p>I get an error that says the following:</p>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access<br>
saved tensors after they have already been freed). Saved intermediate values of the<br>
graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=<br>
True if you need to backward through the graph a second time or if you need to acces<br>
s saved tensors after calling backward.</p>
<p>Upon setting retain_graph to True, I get the following error:</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified<br>
by an inplace operation: [torch.cuda.FloatTensor [70, 32, 4, 4]] is at version 2; e<br>
xpected version 1 instead. Hint: enable anomaly detection to find the operation that<br>
failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>As a workaround, is it functionally equivalent if I detach() and reassign value to the same variable?</p>

username: eqy
cooked: <p>This looks like some kind of backpropgation through time. Could you consider doing a single backward step for multiple forward steps? <a href="https://discuss.pytorch.org/t/correct-way-to-do-backpropagation-through-time/11701">Correct way to do backpropagation through time? - PyTorch Forums</a></p>
<p>I guess another question is what it would mean to compute a gradient for that variable across multiple forward passes. It’s not really a parameter as it is being overwritten during every forward pass.</p>
BufferError: memoryview has 1 exported buffer

username: John4
cooked: <p>Hi. I’m doing parameters tuning of XGBoost classifier for a multilabel problem:</p>
<pre><code>cv_inner = StratifiedShuffleSplit(n_splits=5, test_size=0.3, random_state=109)
clf = RandomizedSearchCV(cl, dist, scoring='r2', cv=cv_inner, refit=True, n_jobs=-1)
search = clf.fit(Xtrain, ytrain)
cv_outer = StratifiedShuffleSplit(n_splits=3, test_size=0.3, random_state=109)
scores = cross_val_score(search, Xtrain, ytrain, scoring='accuracy', cv=cv_outer)
</code></pre>
<p>Could someone explain me why I get this error (BufferError: memoryview has 1 exported buffer) and how to fix it?</p>

username: ptrblck
cooked: <p>This question seems to be unrelated to PyTorch, so I would recommend to cross-post it into an XGBoost-specific discussion board or their GitHub repository in case you think it’s a bug in their library.</p>

username: John4
cooked: <p>Actually, I tried also with other classifiers such as SVM and RF and the same error happens. In an other pc the code works so it is my pc problem. Do you have any idea of what is the cause?</p>

username: ptrblck
cooked: <p>I don’t know what might be causing the issue and cannot find a lot of references.<br>
My first idea would be to create a new and clean virtual environment and reinstall the used packages there so check if somehow the environment is broken.</p>

username: John4
cooked: <p>Ok. I’ll try it. Thank you for the help!</p>

username: John4
cooked: <p>I created a new virtual environment but the error still persists. Please, if you have other ideas, let me know!</p>

username: ptrblck
cooked: <p>Try to narrow down the issue by checking the stacktrace via <code>gdb</code> which should give you more information about the root cause.</p>

username: Tom_Johnson
cooked: <p><a class="mention" href="/u/john4">@John4</a>  I’ve been getting the same error in generic multprocessing.Pool code. This discussion <a href="https://youtrack.jetbrains.com/issue/PY-54447/BufferError-memoryview-has-1-exported-buffer" rel="noopener nofollow ugc">https://youtrack.jetbrains.com/issue/PY-54447/BufferError-memoryview-has-1-exported-buffer</a> suggests that the issue is related to Python 3.10 and PyCharm interaction (my experience too). Switching back to Python 3.9 I had no issues. What you are seeing may be unrelated to PyTorch at all. Hope it helps.</p>
How to use PyTorch-Direct, "Enabling direct memory access" or similar method?

username: Yorai_Levi
cooked: <p>I am reading <a href="https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s32038/" rel="noopener nofollow ugc">PyTorch-Direct: Introducing Deep Learning Framework with GPU-Centric Data Access for Faster Large GNN Training | NVIDIA On-Demand</a><br>
and</p>
<p><a href="https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s32038/" rel="noopener nofollow ugc">PyTorch-Direct: Introducing Deep Learning Framework with GPU-Centric Data Access for Faster Large GNN Training | NVIDIA On-Demand</a></p>
<p>It makes me wonder how to use these and it it’s supported as a first party feature nowadays since some time has passed.</p>
Understanding Model's Parameters in Distributed DataParallel

username: Super_Villain
cooked: <p>Hi guys,</p>
<p>I’ve been trying to figure out wether  <code>model_with_ddp</code>  and  <code>model_without_ddp</code>  share the same state dict or not.</p>
<p>For example, if I create <code>model_without_ddp</code> via<br>
<code>model_without_ddp = model_with_ddp.module</code><br>
and update  <code>model_with_ddp</code>  parameters via some loss and some optimizer, will<br>
<code>model_without_ddp</code>  change correspondingly?</p>
<p>Many thanks,<br>
Yiming</p>

username: ptrblck
cooked: <p>Changing the internal parameters via <code>ddp.module</code> should be reflected in the parent <code>DDP</code> module on this rank. However, you would run into the risk of diverging the models on different ranks if you are not properly communicating the gradients between all ranks.</p>

username: Super_Villain
cooked: <p>Hi Piotr,</p>
<p>Sorry for the my poor explanation.<br>
How about the reverse? I mean if there is a <code>model_without_ddp</code> defined (via <code>ddp.module</code>) and stored in the memory, and later the parallelized model changes its params, will this also be reflected in previously defined <code>model_without_ddp</code>?<br>
Or, do they share the same memory?<br>
Many thanks,<br>
Yiming</p>
Why does nn.Transformer enforce the input embedding size to be equal to transformed embedding size?

username: Goldname
cooked: <p>Let src be the input to the decoder. src.size() = (seq_len, embed_dim_1). Then the linearly transformation yields Q= src*W_query has new dimensions of (seq_len, embed_dim_2)</p>
<p>However it appears that the Pytorch implementation of the Transformer enforces embed_dim_1 = embed_dim_2.</p>
<ol>
<li>Why is this?</li>
<li>Is this the standard variation of transformers used in Large language models like BERT, GPT?</li>
<li>Is there a way for me to allow the dimensions to be different, or do I need to build my own transformer?</li>
</ol>
nn.Transformer: is d_model the input embedding or transformed embedding size?

username: Goldname
cooked: <p>I am trying to use nn.Transformer. I am confused if the d_model parameter is the embedding dimension of the input, or if it’s the embedding dimension of the linearly transformed input</p>
<p>I.E., Let X = input where X.size() = (seq_len, embed_dim_1). Then the linearly transformed input = X*W_query with new dimensions of (seq_len, embed__dim_2)</p>
<p>I first treated d_model as the input embedding but then I got an error about how d_model must be divisible by num_heads which doesn’t make sense. The input embedding * W needs to be divisible by num_heads, not the input embedding itself.</p>
Can I run any deepspeed application with torch.distributed.run?

username: mathewjacob1002
cooked: <p>I was interested in learning that, with deepspeed, you can use torch.distributed.run to run HuggingFace and Pytorch Lightning application with deepspeed optimizations. Is this also true for any arbitrary deepspeed application? Additionally, I was also curious as to what the differences were between the deepspeed launcher and using the torch.distribued.run launcher?</p>
<p>Thank you so much!</p>
Mask not making any difference for time series transformers

username: Spencerfonbuena
cooked: <p>I am using transformers for a time series forecasting task. The following is my mask code, and transformer code. I am having a few issues, one of which is that the validation MAE is 0, but during inference the model performs very poorly (I think there may be other problems allowing the model to perform perfectly, but I wanted to start here). I figured it may be due to an improper implementation of the mask. Any help would be greatly appreciated. In order to check my hypothesis, I also set the masks to all 0’s, and nothing changed. Any ideas as to why that would be as well?</p>
<p>mask code:</p>
<pre><code>def mask(dim1: int, dim2: int):

return torch.triu(torch.ones(dim1, dim2) * float('-inf'), diagonal=1)
</code></pre>
<p>transformer code:</p>
<p>class Model(nn.Module):</p>
<pre><code>def __init__(self,
             d_model = int,
             heads = int,
             dropout = float,
             dim_feedforward = int,
             stack = int, 

             # [Embedding]
             channel_in = int,
             window_size = int,
             pred_size = int
             ):
    super(Model, self).__init__()
    
    def mask(dim1: int, dim2: int):
        return torch.triu(torch.ones(dim1, dim2) * float('-inf'), diagonal=1)


    self.embedding = Embedding(channel_in=channel_in, window_size=window_size)

    # [Encoder]
    encoder_layer = nn.TransformerEncoderLayer(d_model=d_model, nhead=heads, dim_feedforward=dim_feedforward, dropout=dropout, activation='gelu',batch_first=True, norm_first=True,)
    self.encoder = nn.TransformerEncoder(encoder_layer=encoder_layer, num_layers=stack, norm=nn.LayerNorm(d_model))
    
    # [Mask]
    self.tgt_mask = mask(pred_size, pred_size).to(DEVICE)
    self.src_mask = mask(pred_size, window_size).to(DEVICE)

    # [Decoder]
    decoder_layer = nn.TransformerDecoderLayer(d_model=d_model, nhead=heads, dim_feedforward=dim_feedforward, dropout=dropout, activation='gelu', batch_first=True, norm_first=True,)
    self.decoder = nn.TransformerDecoder(decoder_layer=decoder_layer, num_layers=stack, norm=nn.LayerNorm(d_model))

    self.out = nn.Linear(d_model, 1)


def forward(self, x, tgt):
    
    x = self.embedding(x)
    tgt = self.embedding(tgt)
    memory = self.encoder(x)
    out = self.decoder(tgt, memory, self.tgt_mask, self.src_mask)
    out = self.out(out)

    

    return out
</code></pre>
<p>P.S. I apologize in advance for the wierd formatting. I am new to pytorch forums, and don’t know how to make it so all the code bits are greyed out. If anyone knows how to do that, it may help in future questions.</p>

username: Spencerfonbuena
cooked: <p>I still haven’t figured it out completely, but just in case someone has these issues, I figured out that the mask is an additive mask when implemented with torch.transformerdecoder, which means that even though I set the mask to all zeros, it essentially got rid of the whole mask.</p>
Upstream einops library?

username: hoangmit
cooked: <p>Can we add einops library into pytroch ?</p>
<aside class="quote quote-modified" data-post="1" data-topic="29639">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arogozhnikov/48/1584_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/meet-einops-new-operations-for-pytorch/29639">Meet einops - new operations for pytorch</a> 
  </div>
  <blockquote>
    I’d like to present a new project called einops. 
Einops provides flexible and powerful operations for readable and reliable code (developed right for deep learning). 
[einops package examples] 
Einops provides complete support for pytorch. 
Project lives at github: <a href="http://github.com/arogozhnikov/einops" rel="noopener nofollow ugc">http://github.com/arogozhnikov/einops</a>
  </blockquote>
</aside>

<p>The ops are low level and popular in both vision an NLP codes. Having this inside pytorch improve usability and management</p>
What are the differences between Flash Attention and Memory Efficient Attention?

username: Puellaquae
cooked: <p>I’m learning about PyTorch and Transformer. While reading the source code of PyTorch, I noticed that if I don’t enable the <code>USE_FLASH_ATTENTION</code> compilation condition, the memory efficient attention won’t be compiled into PyTorch. Does this mean that the implementation of memory-efficient attention depends on the implementation of flash attention? And,  I am confused about the specific differences between memory efficient attention and flash attention in the implementation.</p>
How cudnn v8 benchmark different convolution algorithms

username: Wang_Albert
cooked: <p>Hello,</p>
<p>I have some questions about cudnn v8 benchmarking for convolution algorithms.</p>
<ol>
<li>
<p>Does the “valid_plans” pass to <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cudnn/Conv_v8.cpp#L403-L429" rel="noopener nofollow ugc">time_sorted_plan</a>  represent different convolution algorithms?</p>
</li>
<li>
<p>If that’s the case, is it because the <code>desc</code> parameter in this function is of type <code>cudnnBackendDescriptorType_t</code> with the value <code>CUDNN_BACKEND_OPERATION_CONVOLUTION_FORWARD_DESCRIPTOR</code> ? and then generate an opGraph with corresponding operation?</p>
</li>
</ol>
<p>Thanks for your help!!</p>
Which type of CUDA Math API does torch.sin() use?

username: tianyma
cooked: <p>CUDA has two types of math API to compute transcendental functions such as sin, cos, sqrt, exp, etc.<br>
One is the accurate version, e.g.</p>
<pre><code class="lang-auto">__device__​ double sin ( double  x )
</code></pre>
<p>which can be found in<br>
<code>https://docs.nvidia.com/cuda/cuda-math-api/group__CUDA__MATH__DOUBLE.html#group__CUDA__MATH__DOUBLE_1g3ebbca20a2937d1fe51329402880df85</code><br>
The other is the fast approximate version, e.g.</p>
<pre><code class="lang-auto">__device__​ float __sinf ( float  x )
</code></pre>
<p>which can be found in<br>
<code>https://docs.nvidia.com/cuda/cuda-math-api/group__CUDA__MATH__INTRINSIC__SINGLE.html#group__CUDA__MATH__INTRINSIC__SINGLE_1gfa0ea4b2cee94521792ead0deb03addb</code></p>
<p>MY question is, if I call  torch.sin(), which type of CUDA API does it use?<br>
If it uses the accurate version, how can I use the fast approximate version?</p>

username: ptrblck
cooked: <p>The native implementation should use <code>std::sin</code> as seen <a href="https://github.com/pytorch/pytorch/blob/40f53912cf797b55f63adf0e5ac0ec08ccfafe4a/aten/src/ATen/native/cuda/UnaryGeometricSinKernel.cu#L24">here</a>.<br>
To use the fast approximate implementation you could use a custom CUDA extension.</p>

username: tianyma
cooked: <p>Thanks for your fast reply! I solved the problem following <code>https://pytorch.org/tutorials/advanced/cpp_extension.html</code>.</p>

username: tianyma
cooked: <p>BWT I found my implementation of the fast version is slower than I directly call torch.xxx. I know this is maybe a CUDA problem, but it’ll be very appreciated if you could help me.</p>
<p>for example, when I implement the fast version of exp().<br>
In the cuda file, my implementation is</p>
<pre data-code-wrap="cu"><code class="lang-plaintext">template &lt;typename scalar_t&gt;
__global__ void fastexp_cuda_kernel(
    const torch::PackedTensorAccessor32&lt;scalar_t, 1, torch::RestrictPtrTraits&gt; input,
    torch::PackedTensorAccessor32&lt;scalar_t, 1, torch::RestrictPtrTraits&gt; output) {
  const int index = blockIdx.x * blockDim.x + threadIdx.x;
  output[index] = expf(input[index]);

torch::Tensor fastexp_cuda(torch::Tensor input) {
  const auto size = input.size(0);
  auto output = torch::empty_like(input);
  const int threads = 1024;
  const dim3 blocks((size + threads - 1) / threads);
  AT_DISPATCH_FLOATING_TYPES(input.type(), "fastexp_cuda", ([&amp;] {
    fastexp_cuda_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(
        input.packed_accessor32&lt;scalar_t, 1, torch::RestrictPtrTraits&gt;(),
        output.packed_accessor32&lt;scalar_t, 1, torch::RestrictPtrTraits&gt;());
  }));
  return output;
}
}
</code></pre>
<p>I use the JIT compiling method, I do find it can be faster when I use <code>--use_fast_math</code> flag.<br>
BUT anyway it is slower than <code>torch.exp()</code>.<br>
does <code>torch.exp()</code> use any optimization? Or does my implementation have any redundant?</p>