akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
List all leaves in a backward graph

Pierre_Quinton: <p>I am trying to get a set of all leaves that are involved in a backward graph of some <code>Tensor</code>.<br>
<a href="https://discuss.pytorch.org/t/print-all-graph-leaves/8355">This</a> was supposed to be similar but saddly the answer contains a link to a 404 page.</p>
<p>Anyone has an idea ?</p>
GradScaler for CPU with AMP

Hossein_Ghafarian: <p>Hi,<br>
Here <a href="https://pytorch.org/docs/stable/amp.html" rel="noopener nofollow ugc">AMP in pytorch</a> it is stated that we can use uses <code>torch.autocast</code> and <code>torch.cpu.amp.GradScaler</code> or <code>torch.cuda.amp.GradScaler</code>. But when I try to import the torch.amp.GradScaler, or torch.cpu.amp.GradScaler, it says that there is no GradScaler in it.</p>
<p>The pytorch version is 2.2.2+cpu, I have tried it with 2.2.1+cpu. How to resolve this issue?</p>

ptrblck: <p><code>amp</code> on CPU should use <code>bfloat16</code> only, which does not need gradient scaling.</p>

Hossein_Ghafarian: <p>But in the documentation it specifically states that you can use gradient scaling with cpu and amp. It even shows the way to call the gradient scaling in AMP, in two different ways.</p>

ptrblck: <p>Could you point me to the section in the docs showing gradient scaling with <code>bfloat16</code> on the CPU, please?</p>

Hossein_Ghafarian: <p>In second paragraph in this <a href="https://pytorch.org/docs/stable/amp.html" rel="noopener nofollow ugc">pytorch.org about amp with CPU</a>. It is stated that</p>
<p>Ordinarily, “automatic mixed precision training” with datatype of <code>torch.float16</code> uses and <code>torch.cpu.amp.GradScaler</code> or<code>torch.cuda.amp.GradScaler</code>together…</p>
<p>Moreover, next in the same document says that</p>
<ul>
<li><code>torch.GradScaler("cpu", args...)</code> is equivalent to <code>torch.cpu.amp.GradScaler(args...)</code>.</li>
</ul>

ptrblck: <p>Thanks for pointing to this section. It seems <code>float16</code> was implemented for CPU ops (although I have no idea if any performance benefits are expected) and indeed the <code>GradScaler</code> is also available:</p>
<pre data-code-wrap="python"><code class="lang-python">conv = nn.Conv2d(3, 3, 1, 1)
x = torch.randn(1, 3, 24, 24)
with torch.autocast(device_type="cpu", dtype=torch.float16):
    x = conv(x)

scaler = torch.GradScaler("cpu")
scaler
# &lt;torch.amp.grad_scaler.GradScaler at 0x7fdc4e202200&gt;
</code></pre>

Hossein_Ghafarian: <p>Thank you very much for your answer. But your code gives me the following error:</p>
<pre><code class="lang-auto">scaler = torch.GradScaler("cpu")
AttributeError: module 'torch' has no attribute 'GradScaler'
</code></pre>
<p>My pytorch version is 2.2.2. What is yours?</p>

ptrblck: <p>I am using a current nightly binary, but could you try updating to the recent stable release (2.3)?</p>

Hossein_Ghafarian: <p>Sorry for late reply. Because I couldn’t upgrade torch. Now, that I upgraded torch to 2.3.0. and now, your code works fine.</p>
<p>Thank you very muc.</p>
Trace all matrix vector multiplication in a nn.Module

GoldenalCheese: <p>Say I have a noisy matrix-vector multiplication (MVM) operation enabled by some novel hardware. I want to simulate how this noisy MVM will impact a nn.Module inference. How can I trace and then replace all MVMs in the given nn.Module?</p>
<p>nn.Linear perhaps is the easiest to deal with. But attention and Conv layers are also based on MVMs. I wonder if there is a general way that can deal with any layers?</p>
<p>Additionally, sometimes I guess a nn.Module might directly use torch.matmul or nn.funcitonal.linear instead of defining a nn.Linear module. How can I trace these operations?</p>
<pre><code class="lang-auto">class CustomNN(nn.Module):
    def __init__(self, input_dim, linear_dim, attention_dim, conv_in_channels, conv_out_channels, conv_kernel_size):
        super(CustomNN, self).__init__()
        self.linear = nn.Linear(input_dim, linear_dim)
        self.attention = Attention(linear_dim, attention_dim)
        self.conv = nn.Conv2d(conv_in_channels, conv_out_channels, conv_kernel_size)
        self.fc = nn.Linear(conv_out_channels, 1)
        
    def forward(self, x):
        x = self.linear(x)
        x, attention_weights = self.attention(x)
        x = x.view(x.size(0), -1, 1, 1)
        x = F.relu(self.conv(x))
        x = F.adaptive_avg_pool2d(x, 1)
        x = x.view(x.size(0), -1)
        x = self.fc(x)
        return x, attention_weights

model = CustomNN(input_dim, linear_dim, attention_dim, conv_in_channels, conv_out_channels, conv_kernel_size)

traced_model = trace(model)

# Will run MVMs performed with error. For simplicity, say here the MVM error
# is just MVM(A,B) = A*B + Gauss(0,1). How to implement the trace()?
# I want it to be able to track the MVMs of the 'self.linear, self.attention, self.fc' 
# in the CustomNN. More generally, trace() should ideally be able to 
# deal with any layer types which use MVMs inside.
traced_model(x) 

</code></pre>
<p>Thanks</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/goldenalcheese">@GoldenalCheese</a>,</p>
<p>Can you give a minimal reproducible example in pytorch and show the desired output/behavior?</p>
How to solve TracerWarning during onnx export

Pierre_Carceller-Meu: <p>Hello !</p>
<p>My question is very simple: The first 3 lines of my forward are these:</p>
<pre><code class="lang-auto">    def forward(self, x):
        h, w = x.shape[2:]
        max_offset = max(h, w) / 4.
</code></pre>
<p>Everything works fine during training and evaluation. When I want to export my model in onnx format I get this warning :</p>
<pre><code class="lang-auto">TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  max_offset = max(h, w) / 4.
</code></pre>
<p>I understand that the max_offset value will be calculated once during export and then stored as a constant value. But how can I make the value dynamic?</p>
<p>I’ve been stuck on this point for hours…</p>
<p>Thanks in advance</p>

ptrblck: <p>I’ve edited your title and removed the expletive so please watch your language in future posts.</p>

Pierre_Carceller-Meu: <p>You’re right. Sorry about that… After several hours of being stuck, frustration mounts…</p>
OSError when importing torch in Python script

AkiraS: <p>Hello, to whom it concerns</p>
<p>I have stumbled upon a roadblock trying to run a Python script that uses PyTorch. Needless to say, the error message is being quite stubborn when I try to import the <code>torch</code> module. Here’s the error message I’m seeing:</p>
<p>“G:\Project Raphael\env\Lib\site-packages\torch_<em>init</em>_.py”, line 110, in <br>
os.add_dll_directory(dll_path)<br>
File “”, line 1125, in add_dll_directory<br>
OSError: [WinError 87] The parameter is incorrect: ‘G:Library\bin’</p>
<p>I’m using Python version 3.12.3 and PyTorch version 2.3.0 on Windows 11. I’ve tried setting the <code>dll_path</code> to <code>'G:\Project Raphael\.venv\Lib\site-packages\torch\lib'</code>, which is where my <code>torch</code> package is installed, but that did not change the outcome.</p>
<p>Snippet of the problem:</p>
<h1><a name="add-dll-directory-before-importing-torch-1" class="anchor" href="#add-dll-directory-before-importing-torch-1"></a>Add DLL directory before importing torch</h1>
<p>dll_path = r’G:\Project Raphael.venv\Lib\site-packages\torch\lib’<br>
if os.path.exists(dll_path):<br>
os.environ[‘PATH’] = dll_path + os.pathsep + os.environ[‘PATH’]<br>
print(f"Successfully added DLL directory: {dll_path}“)<br>
else:<br>
print(f"Warning: The DLL path ‘{dll_path}’ does not exist”)</p>
<h1><a name="now-import-torch-2" class="anchor" href="#now-import-torch-2"></a>Now import torch</h1>
<p>import torch</p>

AkiraS: <p>I resolved the issue by downgrading python to Python 3.9.16. After digging further into lots of forums. There was a member that presented a possible solution. I am not exactly sure but from the explanation of the forums I read the finger was pointed at ‘Python 3.12’ But I just wanted to put this here so if anyone else has a similar issue try downgrading to isolate the problem.</p>
WeightedRandomSampler has no effect

nastaran_93th: <p>Dear all,<br>
I have an imbalanced dataset for training which I have used WeightedRandomSampler for train, dev_train and test separately:</p>
<pre><code>  class_weights=[1,112/75,112/32] # class 0 (112), class 1 (75)  class 2 (32) 
  train_sample_weights=[0]*len(datasets_list['train'])                     #172
  dev_train_sample_weights=[0]*len(datasets_list['dev_train'])     #21
  test_sample_weights=[0]*len(datasets_list['test'])                       #26
  
  from torch.utils.data import WeightedRandomSampler
  for idx, data in enumerate(datasets_list['train']):
      label=data['label']
      class_weight=class_weights[label]
      train_sample_weights[idx]=class_weight
  sampler_train=WeightedRandomSampler(train_sample_weights,num_samples=len(train_sample_weights),replacement=True)
  
  for idx, data in enumerate(datasets_list['dev_train']):
      label=data['label']
      class_weight=class_weights[label]
      dev_train_sample_weights[idx]=class_weight
  sampler_dev_train=WeightedRandomSampler(dev_train_sample_weights,num_samples=len(dev_train_sample_weights),replacement=True)

  for idx, data in enumerate(datasets_list['test']):
      label=data['label']
      class_weight=class_weights[label]
      test_sample_weights[idx]=class_weight
  sampler_test=WeightedRandomSampler(test_sample_weights,num_samples=len(test_sample_weights),replacement=True)


  datasets_list={}
  for group in dict_expl_MCclassifier_training.keys():  #created with all the data from 3 classes 
  
     if group =='train':
  
      datasets_list[group] = tio.SubjectsDataset(dict_expl_MCclassifier_training[group], transform=transform_train)
      
     else:
      
      datasets_list[group] = tio.SubjectsDataset(dict_expl_MCclassifier_training[group], transform=transform_dev)


  dataloaders_glaucoma_training_MCclassifier={'train': DataLoader(dataset=datasets_list['train'], batch_size=10,sampler= sampler_train, num_workers=8),
               'dev_train': DataLoader(dataset=datasets_list['dev_train'], batch_size=10,sampler=sampler_dev_train, num_workers=8),
               'test': DataLoader(dataset=datasets_list['test'], batch_size=10,sampler=sampler_test, num_workers=8)}
</code></pre>
<p>The problem is after testing the trained model, I can see data from class 1 and 2 are misclassified as class 0(the largest class).<br>
So apparently the weightedsampler has not worked properly but how can I make sure if the  weightedsampler is the problem?<br>
p.s.:when I manually and randomly selected similar number of data from 3 classes the classifier is working much better</p>
Is using @torch.no_grad() for a whole evaluate function a good idea?

Zhou-Yucheng: <p>Typically I use the <code>with torch.no_grad()</code> context in a evaluate function like the following example:</p>
<pre><code class="lang-auto">def evaluate(model: nn.Module, val_loader):
    model.eval()
    val_loss = 0
    correct = 0
    criterion = nn.BCEWithLogitsLoss()

    with torch.no_grad():
        for images_1, images_2, targets in val_loader:
            images_1, images_2, targets = images_1.to(device), images_2.to(device), targets.to(device)

            with autocast(enabled=True):
                outputs = model(images_1, images_2).squeeze()
                val_loss += criterion(outputs, targets).sum().item()  # sum up batch loss

            pred = torch.where(outputs &gt; 0.5, 1, 0)  # get the index of the max log-probability
            correct += pred.eq(targets.view_as(pred)).sum().item()

    val_loss /= len(val_loader.dataset)
    print(f'Valid set: Loss: {val_loss:.4f}, Accuracy: {100.0 * correct / len(val_loader.dataset):.1f}%')
</code></pre>
<p>Now I know we can use the <code>@torch.no_grad()</code> decorator above a function to act like using <code>with torch.no_grad()</code> context at the beginning and for the all function body (ref: <a href="https://stackoverflow.com/questions/77875298" class="inline-onebox" rel="noopener nofollow ugc">pytorch - What is the difference between '''@torch.no_grad()''' and '''with torch.no_grad()''' - Stack Overflow</a>).</p>
<p>So, I want to know for the above typical example, will it have difference if I use <code>@torch.no_grad()</code> decorator to replace <code>with torch.no_grad()</code> context? That is, the whole function body is in no_grad mode including the <code>model.eval()</code> line and others.</p>
<p>My motivation is using <code>@torch.no_grad()</code> decorator to improve the readability (less indention in the function body).</p>

JuanFMontesinos: <p>They are analogous as far as I know. The context manager just address cases where you may want to exclude part of a fn.</p>
Help on node classification with only a single positive node per graph

ThijsTUE: <p>I got a Geometric Deep Learning problem I cannot find the solution for. I think that is an easy question for someone experienced in the subject.</p>
<p>I got around 1000 graphs on which I use GAT to do node classification. Every graph has 22 nodes. For every graph, only one node is labeled as 1 (the receiver).</p>
<p>So it looks like this:<br>
example 1 y: [0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]<br>
example 2 y: [0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0]<br>
example 3 y: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0]<br>
example 4 y: [0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]</p>
<p>I think I should use Cross Entropy Loss and threat it as there are 22 classes. However, I am afraid that the model will try to understand when to predict each of the classes. Instead, it should handle this problem as evaluating node per node and finally selecting the one node that has the highest chance of being the receiver. By using BCEloss I am afraid that the model does not understand that there only can be one receiver per graph.</p>
<p>Could someone please help me with this problem.</p>
<p>I am sorry if I explained it too vaguely, please ask any questions that will make it more clear. I will try to respond as fast as I can.</p>
import spconv.pytorch will slow down torch.multiprocessing.Pool().apply_async()

dwbzxc: <p>Hi, when I try using spconv-cu116(2.3.6) and torch.multiprocessing.Pool().apply_async() in 1.13 Pytorch version together, I find that the time for executing multiprocess mission is much longer after importing spconv (4.728647470474243 seconds before importing spconv, 11.85951018333435 after importing spconv, while the singleprocess takes 6.01546311378479 seconds to run the codes below with a NVIDIA GeForce 4090 GPU). Do you have any ideas how to solve this problem? Thanks!</p>
<pre><code class="lang-auto">import spconv.pytorch as spconv
import time
import torch.multiprocessing as mp

def multipreocess_func(x, y):
    time.sleep(1.5)
    return x, y, x + y

if __name__ == '__main__':
    torch.multiprocessing.set_start_method('spawn')
    with mp.Pool(processes=4) as pool:
        pytorch_device = torch.device('cuda:0')
        num = torch.ones(1)
        
        a = time.time()
        res = [pool.apply_async(multipreocess_func, (num, i)) for i in range(4)]
        for i, x in enumerate(res):
            pam_x = x.get()[0]
            pam_y = x.get()[1]
            pam_xy = x.get()[2]
            print(f"{pam_x},{pam_y},{pam_xy}")
        b = time.time()

        c = time.time()
        for i in range(4):
            res = multipreocess_func(num, i)
            pam_x = res[0]
            pam_y = res[1]
            pam_xy = res[2]
            print(f"{pam_x},{pam_y},{pam_xy}")
        d = time.time()
        print(f"multiprocess time:{b - a}, singleprocess time:{d - c}")
</code></pre>

ptrblck: <p>CUDA operations are executed asynchronously so could you describe what exactly you are trying to profile in your code? Do you want to profile the kernel dispatching or launching only or the actual execution time? In the latter case you would need to synchronize the code before starting and stopping the timers.</p>

dwbzxc: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="203548">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>launchi</p>
</blockquote>
</aside>
<p>Thanks for your reply. Actually I want to use the multiprocessing to calculate the variance in parallel so as to reduce the execution time(see <a href="https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm" class="inline-onebox" rel="noopener nofollow ugc">Algorithms for calculating variance - Wikipedia</a>). But I found that when trying to do so, it was slower with multiprocessing.</p>
<p>The code above is a tiny demo summarized after I found out the time differences before and after importing <em>spconv.pytorch</em>. Sorry I’m not quite clear about how to <em>synchronize the code before starting and stopping the timers</em>.</p>
Flash Attention with variable-length sequences

gal_avineri: <p>I would like to use the flash implementation of attention on sequences of variable length. However, i’m not sure how this can be achieved.</p>
<p>For example, I attempted to perform self-attention on padded sequences together with the padding mask as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn
from torch.nn.utils.rnn import pad_sequence

n_features = 8
batch_size = 2
lengths = torch.tensor([1, 2])

model = nn.TransformerEncoderLayer(d_model=n_features, nhead=1, batch_first=True, dtype=torch.float16, device='cuda')

# Define data
data = [torch.rand((lengths[i], n_features), dtype=torch.float16, device='cuda') for i in range(batch_size)]
data_padded = pad_sequence(data, batch_first=True)
pad_mask = (torch.arange(max(lengths))[None] &gt;= lengths[:, None]).cuda()

with torch.backends.cuda.sdp_kernel(enable_math=False, enable_mem_efficient=False, enable_flash=True):
    model(data_padded, src_key_padding_mask=pad_mask)
</code></pre>
<p>This results in the following error:</p>
<pre><code class="lang-auto">RuntimeError: No available kernel. Aborting execution.
</code></pre>
<p>Which is preceded by the following warning:</p>
<pre><code class="lang-auto">UserWarning: Both fused kernels do not support non-null attn_mask.
</code></pre>
<p>Is there another method to perform flash SDPA on variable-length sequence?</p>

gal_avineri: <p>It is achievable using Nested Tensors, as shown in the <a href="https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html#nestedtensor-and-dense-tensor-support" rel="noopener nofollow ugc">SDPA tutorial</a></p>
Error with the build wheel

raf789: <p>(raf© kali-raspberry-pi-zero-2-w)-[~] pip install pytorch</p>
<p>Defaulting to user installation because normal site-packages is not writeable Collecting pytorch Using cached pytorch-1.0.2. tar.gz (689 bytes) Preparing metadata (setup.py) … done Building wheels for collected packages: pytorch Building wheel for pytorch (setup.py) … error error: subprocess-exited-with-error python setup.py bdist wheel did not rull successfully. exit code: 1 &gt; [6 lines of output] Traceback (most recent call last): File “‹string&gt;”, line 2, in ‹module&gt; File “‹pip-setuptools-caller&gt;”, line 34, in ‹module&gt; File “/tmp/pip-install-lhoczi82/pytorch_573a1798d654826a953d5c2bbd7c f96/setup-py”, line 15, in ‹module&gt; raise Exception(message) Exception: You tried to install “pytorch”. The package named for PyTorch is “torch” [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pytorch Running setup.py clean for pytorch Failed to build pytorch ERROR: Could not build wheels for pytorch, which is required to install pyproject.toml-based projects</p>
<p>I have no clue what this means but gitcloned one of the Pytorch repositories but then deleted it after it didn’t work. Then I tried this after I couldn’t find any nightly builds that worked with my pi zero 2 W.</p>

brandon698sherrick: <p><a href="https://www-kynect.com" rel="noopener nofollow ugc">Kynect</a><a class="mention" href="/u/raf789">@raf789</a> ,</p>
<p>The error you’re encountering is due to trying to install a package named <code>pytorch</code> instead of the correct package name, which is <code>torch</code>. Here’s how you can correctly install PyTorch on your Raspberry Pi Zero 2 W.<br>
Use the correct package name torch instead of pytorch. Additionally, since you’re on a Raspberry Pi, you might need to install a version that’s compatible with your device’s architecture.<br>
Installing PyTorch on a Raspberry Pi can be a bit tricky due to compatibility issues. There are pre-built wheels available for ARM architectures that you can try. First, ensure you have all the necessary dependencies installed. Download the appropriate wheel file for your Raspberry Pi Zero 2 W from the PyTorch ARM build repository.</p>
<p>Pre-built wheels can simplify the installation process. You can also check sites like piwheels for PyTorch wheels tailored for Raspberry Pi devices.</p>
<p>If pre-built wheels are not available or do not work, you can try building PyTorch from source. This is a more advanced and time-consuming process. Detailed instructions can be found in the official PyTorch documentation.</p>
Reimplementing conv2d as a c++ extension

71M073J: <p>Hi, i’m trying to (re)implement 2d convolution manually, as i need to change some details in how it works due to implementing a low-level convolution approximation algorithm (also described in a bit more detail <a href="https://discuss.pytorch.org/t/custom-conv2d-algorithm-c-extension/203177">here</a></p>
<p>My general idea is to take the pytorch cpp files and simply adjust the needed values; However, when taking the ConvolutionMM2d.cpp, and registering it as an extension (in the same way as the LLTM tutorial extension), the cpp file does not compile. Obviously i am missing something, as i am inexperienced in the ways of cpp and pytorch, but i would think that taking an (obviously working) part of a library would work and be a good starting point for adjustments.</p>
<p>Does anyone know how i could progress with this?</p>
Requires_grad() = True for ReLU() activation?

jasonnr626: <p>Why does my relu activation function has requires_grad = True? What happens if I set it to False?</p>
<p>My model:</p>
<blockquote>
<p>ConvAE(<br>
(encoder): Sequential(<br>
(0): Conv1d(1, 32, kernel_size=(220,), stride=(2,))<br>
(1): ReLU()<br>
(2): Conv1d(32, 21, kernel_size=(88,), stride=(2,))<br>
(3): ReLU()<br>
(4): Conv1d(21, 11, kernel_size=(35,), stride=(2,))<br>
(5): ReLU()<br>
(6): Conv1d(11, 1, kernel_size=(14,), stride=(2,))<br>
(7): ReLU()<br>
)<br>
(decoder): Sequential(<br>
(0): ConvTranspose1d(1, 11, kernel_size=(14,), stride=(2,))<br>
(1): ReLU()<br>
(2): ConvTranspose1d(11, 21, kernel_size=(35,), stride=(2,))<br>
(3): ReLU()<br>
(4): ConvTranspose1d(21, 32, kernel_size=(88,), stride=(2,), output_padding=(1,))<br>
(5): ReLU()<br>
(6): ConvTranspose1d(32, 1, kernel_size=(220,), stride=(2,), output_padding=(1,))<br>
)<br>
)</p>
</blockquote>
<p>When I run the code:</p>
<pre><code class="lang-auto"> for param in model.encoder.parameters():
     print(param.shape)
     print(param.requires_grad)
</code></pre>
<p>the output is:</p>
<pre><code class="lang-auto">torch.Size([32, 1, 220])
True
torch.Size([32])
True
torch.Size([21, 32, 88])
True
torch.Size([21])
True
torch.Size([11, 21, 35])
True
torch.Size([11])
True
torch.Size([1, 11, 14])
True
torch.Size([1])
True
</code></pre>
<p>I guess the torch.Size([“only one value”]) are my relu activation functions for the filters. But why do they have require_grad=True? And what happens if I set requires_grad=false for them? Will the training happen in the same way? Or will this make the gradient chain break somehow?</p>
<p>The idea is, that I want to train single conv layers / kernel why keeping all other layers fixed.</p>

ptrblck: <p><code>nn.ReLU</code> does not have trainable parameters as it’s a pure functional call.</p>
Pytorch install with conda

Skeptic9793: <p>When I use<code> conda install pytorch=1.9.1 torch vision=0.10.1 torch studio=0.9.1 cudatoolkit=11.3-c pytorch - c conda forge</code> to install the GPU version of pytorch, I can only get one CPU version. Can someone tell me the reason</p>

ptrblck: <p>The command seems to have syntax errors and tries to use <code>conda-forge</code> which contains community builds not maintained by us. I don’t know where this install command comes from, but you should stick to the ones posted on our website.</p>

Skeptic9793: <p>no，this command comes literally from <a href="https://pytorch.org/get-started/previous-versions/#linux-and-windows-30" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a></p>

wilhelm: <p>He means:<br>
there is no space between <em>torch</em> and <em>vision</em><br>
or <em>torch</em> and <em>studio</em><br>
or <em>conda</em> and <em>forge</em></p>
<p>You wrote:<br>
<code>conda install pytorch=1.9.1 torch vision=0.10.1 torch studio=0.9.1 cudatoolkit=11.3-c pytorch - c conda forge</code></p>
<p>But it must be:</p>
<pre><code class="lang-auto">conda install pytorch==1.9.1 torchvision==0.10.1 torchaudio==0.9.1 cudatoolkit=11.3 -c pytorch -c conda-forge
</code></pre>
<p>Clear now?</p>
Inference using model weights and without pytorch

payman: <p>Is it possible to extract weights of a trained model, and after that, do the inference just by matrix multiplication in any language or system, without the need for torchscript or exporting model to other formats?</p>
<p>(My problem is that I have a model trained in python (Tacotron 2 for TTS), now I want to use this model in C++ and also in mobile devices, for this it is required to first use torchscript to compile the model, but it seems that this model is not scriptable! I’ve already posted about this <a href="https://discuss.pytorch.org/t/can-not-port-checkpoint-to-mobile-using-torchscritp-cant-redefine-method-forward-on-class-torch-model-encoder/164343">here </a>)</p>

sree_harsha: <p>Hi,</p>
<p>While it is not exactly the model you are referring to (TacoTron) – people have exactly shown this to be possible:</p>
<p>Some examples:</p>
<p>[1] <a href="https://github.com/ggerganov/ggml" rel="noopener nofollow ugc">GGML</a> - has structures implemented in C for inference of several cutting edge models.<br>
[2] <a href="https://github.com/ggerganov/whisper.cpp" rel="noopener nofollow ugc">whisper.cpp</a> - same author, but for one model.</p>
<p>Could adapt a similar approach and build on top of these methods <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> – hope this helps!</p>
How to Accelerate Inference for a Deep Slim Network?

111414: <p><strong>Background:</strong></p>
<p>I have a deep network with 20 layers, where both the input and intermediate feature maps have very few channels. Here’s a simplified example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn

model = nn.Sequential(*[nn.Conv2d(2, 2, 3, padding=1) for _ in range(20)]).cuda()
x = torch.rand(16, 2, 64, 64).cuda()
while True:
    model(x)
</code></pre>
<p>When running this on an NVIDIA RTX 4090 GPU, the utilization is around 15% or lower. This suggests the GPU isn’t fully utilized, leading to suboptimal inference speed.</p>
<p><strong>Question:</strong><br>
How can I accelerate inference for such a deep and slim network to better utilize the GPU? The input shape and intermediate feature shapes are fixed in my case. Any suggestions or insights would be greatly appreciated.</p>

ptrblck: <p>Since your input shapes are static you could apply CUDA Graphs to your workflow as explained <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graph-semantics">here</a> manually or via <code>torch.compile</code>.</p>

111414: <p>I sincerely appreciate your suggestions. After reviewing the <code>torch.compile</code> documentation, I integrated the following lines into my training script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.set_float32_matmul_precision('high')  # 1
...
torch.backends.cudnn.enabled = True  # 2
torch.backends.cudnn.benchmark = True  # 3
...
model = torch.compile(model, mode="max-autotune", fullgraph=True)  # 4
model = DDP(model, device_ids=[rank])
...
</code></pre>
<p>These changes significantly boosted the training speed. The GPU utilization increased from 70% to 95%. Please note that my actual training script and network are much more complex than the toy example I previously posted.</p>
<p>Once again, thank you for your kind and valuable assistance. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Automatically cast input to Huggingface model’s device map

souryadey: <p>This is a question on the Huggingface transformers library.</p>
<p>Is there a way to automatically infer the device of the model when using auto device map, and cast the input tensor to that?</p>
<p>Here’s what I have now:</p>
<pre data-code-wrap="python"><code class="lang-python">import transformers
import torch

DEVICE = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"

tokenizer = transformers.AutoTokenizer.from_pretrained(&lt;model_id&gt;)
model = transformers.AutoModelForCausalLM.from_pretrained(&lt;model_id&gt;, device_map = 'auto')

prompt = "Hello how are you"
prompt_obj = tokenizer(prompt, return_tensors = 'pt').to(DEVICE)
# proceed with model.generate
</code></pre>
<p>Instead of hardcoding <code>DEVICE</code>, I’d like to infer it from the model’s device map. Something like:</p>
<pre data-code-wrap="python"><code class="lang-python"># inferred_device = &lt;some code that maybe involves model.hf_device_map&gt;
prompt_obj = tokenizer(prompt, return_tensors = 'pt').to(inferred_device)
</code></pre>
<p>Is there a way to do this?</p>

nicomanzonelli: <p>Hi, the model loaded using Huggingface will have an attribute named <code>hf_device_map</code> which maps the names of certain layers to the device that the layer is physically on. You can use this to map the input to the first layer’s device.</p>
<pre><code class="lang-auto">first_layer_name = list(model.hf_device_map.keys())[0]
self.device = model.hf_device_map[first_layer_name]
</code></pre>
<p>Keep in mind, you could run into memory issues if the if there’s no memory left on the device with the first layer on it.</p>
<p>The pipeline functions help abstract the need to do this manually away.</p>
Packaging: How do a build Python PyTorch from a `-DBUILD_PYTHON=OFF`, library-only build?

Geremia: <p>I have packaged a library-only (non-Python) build of PyTorch (<code>-DBUILD_PYTHON=OFF</code> passed to <code>cmake</code>) for Slackware (using <a href="https://slackbuilds.org/repository/15.0/libraries/pytorch/?search=pytorch" rel="noopener nofollow ugc">this SlackBuild</a>), but now I’d like to make another Slackware package for Python PyTorch.</p>
<p>How do a build Python PyTorch? It seems <code>-DBUILD_PYTHON=ON</code> would have created all the necessary files, but when I run <code>python3 setup.py install --root=$PKG</code>, it complains that I’m not in a <code>git</code> repo; I downloaded <code>pytorch-v2.3.0.tar.gz</code> to make the library-only package.</p>

Geremia: <p>It actually did build the <code>site-package</code> files. I just needed to manually move them to the <code>$PKG</code> by adding these lines to the SlackBuild (right before <code>cd ..</code>):</p>
<pre data-code-wrap="sh"><code class="lang-sh">  mkdir -p $PKG/usr/lib$LIBDIRSUFFIX/python3.11
  mv tmpxxx/usr/lib/python3.11/site-packages $PKG/usr/lib$LIBDIRSUFFIX/python3.11
</code></pre>
<p>But the problem here is that that <code>site-packages</code> directory doesn’t have any <code>__init__.py</code> files or anything. Why didn’t <code>cmake</code> make them? Must I use <code>setup.py</code> (which requires I be in a <code>git</code> directory)?</p>

Geremia: <p>I tried using <code>setup.py</code> in a <code>git</code> clone:</p>
<pre><code class="lang-auto">git clone --recursive https://github.com/pytorch/pytorch
</code></pre>
<p>In the SlackBuild, I used this instead (paths specific to my CUDA install):</p>
<pre><code class="lang-auto">export CMAKE_PREFIX_PATH=$PKG
MAX_JOBS=32 \
USE_CUDA=1 \
CUDA_PATH=/usr/share/cuda \
CFLAGS="$SLKCFLAGS" \
CUDAHOSTCXX=/home/geremia/gcc-11/usr/bin/gcc \
USE_CUDNN=1 \
CUDNN_LIBRARY=/usr/share/cuda/lib64/libcudnn.so \
NVTOOLSEXT_PATH=/usr/lib64/libnvToolsExt.so.1 \
python setup.py develop
</code></pre>
<p>But ran into errors I didn’t have in the CMake build. I wish I didn’t have to use <code>setup.py</code> at all to build the Python bindings.</p>

Geremia: <p>I needed to use Ninja, run <code>setup.py build</code> with the <code>--cmake-only</code> flag, and manually install Python site-package stuff, as described <a href="https://www.linuxquestions.org/questions/slackware-14/building-python-bindings-for-pytorch-4175737251/" rel="noopener nofollow ugc">here</a>.</p>
Exporting DinoV2

anaandreea1228: <p>Hello!<br>
Has anyone managed to successfully export the DinoV2 model, specifically the depth estimation head? I’ve been trying for a while now but haven’t had any success. I used the code from the official notebook on GitHub to initialize the model, and for export, I used torch.jit.trace on a preprocessed example image as in the notebook, then the save function, but the trace method crash. I’d like to achieve an export as a .pt file, but ONNX would also be an option.</p>
How to start with jit?

deepfailure: <p>I have an RL Python code that I want to speed up with JIT.</p>
<p>I have changed from the class definition (torch.nn.Module) to (torch.jit.ScriptModule) and added the decorator <span class="mention">@torch.jit.script_method</span>. I need to rerun the numbers, but my impression is that it speeds up slightly the training.</p>
<p>If I print the layers I can see: (conv2_q1): RecursiveScriptModule(original_name=Conv2d)</p>
<p>What else can I speed up with JIT? Can I set up the training part with JIT?</p>
<p>Also, how does this all tie with torch.jit.trace and torch.jit.script?</p>
<p>It is a beginner question, I am quite new to this possible optimization. Feel free to refer to any training material to understand everything.</p>
<p>Thanks!</p>

ptrblck: <p>The recommended approach is to use <code>torch.compile</code> as <code>TorchScript</code> is in “maintenance” mode and won’t receive any major updated anymore.</p>

deepfailure: <p>Thanks for the reply.</p>
<p>In fact, I have realized after posting that torch.compile is better. It speeds up much more the model for training/inference.</p>
<p>However, I was wondering if there are more ways to speed up other than torch.compile(model) with torch.jit or other pytorch methods (without using distributed training).</p>
<p>I have a class Trainer which contains an actor and critic CNN models. Moreover, it contains the training methods. Can I pass this trainer class to the compile? Or just the models themselves?</p>
<p>Could you provide some references?</p>
How to uninstall/remove PyTorch when built and installed from source?

wilhelm: <p>I followed <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">this</a> guide to install the latest PyTorch with Cuda support on my system.</p>
<p>and importing torch in a terminal I can see the following:</p>
<pre><code class="lang-auto">torch.__version__
'2.4.0x0+gitf14cdc5'
</code></pre>
<p>But the problem now is that that installation overwrites all the other different pytorch versions in my other environments. Activating other conda environments show me exactly the same pytorch version.</p>
<p>Now, how do I get rid of the build and installed version of pytorch?</p>
<p>Thanks</p>
<p>EDIT: I read in many other threads, that when you install PyTorch using conda then it comes with it own cuda environment and so on. But I do not understand, why the installation from source did overwrite all my pytorchs on my system. If I create a new environment and install pytorch using conda, then it  recognize always and everywhere the installed version from source.</p>

ptrblck: <aside class="quote no-group" data-username="wilhelm" data-post="1" data-topic="203382">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/ce7236/48.png" class="avatar"> Dave:</div>
<blockquote>
<p>But the problem now is that that installation overwrites all the other different pytorch versions in my other environments. Activating other conda environments show me exactly the same pytorch version.</p>
</blockquote>
</aside>
<p>Build PyTorch in an own environment instead of the <code>base</code> env.</p>

wilhelm: <p>This is exactly what I did, because I didn’t want to infest my server. I created a new conda environment then I built the conda following the tutorial.</p>
<p>I even looked in stackoverflow for similar problems. Usually there should be a<br>
<code>$ make uninstall</code><br>
command in torch but it seems there is not.</p>

ptrblck: <aside class="quote no-group" data-username="wilhelm" data-post="3" data-topic="203382">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wilhelm/48/70280_2.png" class="avatar"> Dave:</div>
<blockquote>
<p>This is exactly what I did, because I didn’t want to infest my server.</p>
</blockquote>
</aside>
<p>I have 35 virtual environments in <code>conda</code> on my current system with stable, nightly, and source-built binaries spanning from <code>torch==0.4.1</code>, <code>1.12.2+cu113</code>, to the latest nightlies using CUDA 12.4 including source builds using my locally installed CUDA toolkit and have never seen that any libraries “leak” from one virtual environment to another one (not speaking of the <code>base</code> env which is entirely empty on my system).</p>
<p><code>make uninstall</code> does not exist, but <code>pip uninstall torch -y</code> will do it.</p>
Computation of loss in distributed context

Thomas_Sisk: <p>Hello,</p>
<p>I have a question about the computation of loss when using either DP or DDP. I have a particular loss function (physics related) which is highly sensitive to batch size. Thus, I am wondering if there is a way to use a distributed method which aggregates all network outputs onto a single device BEFORE computing the loss. The computation of the loss is analytically more effective when using more data, i.e it involves estimating a linear operator constructed from covariance matrices. It’s not hard to show that computing this loss separately for each scattered batch and then aggregating the  is not desirable for the application. While I understand that under normal circumstances what I am shooting for doesn’t make much sense and is not desirable computationally, I am certain that for this particular application, aggregation of network outputs before estimating the loss would be highly beneficial. Is there a way to do this? Answers explaining that I don’t want to do this in the first place are not appreciated, my motivation is entirely application based and outside the scope of this forum (physics / applied math).</p>

Thomas_Sisk: <p>I’ll add that the network architecture is essentially an encoder, so the network outputs a substantially lower dimensional representation than is input.</p>
Getting set up with libtorch c++

Brandon_Fowler: <p>Hey yall, having some trouble getting a basic nn Module working with libtorch. I’m trying to copy the example found here <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" class="inline-onebox" rel="noopener nofollow ugc">Using the PyTorch C++ Frontend — PyTorch Tutorials 2.3.0+cu121 documentation</a></p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;

struct Net : torch::nn::Module {
  Net(int64_t N, int64_t M) {
    W = register_parameter("W", torch::randn({N, M}), true);
    b = register_parameter("b", torch::randn(M), true);
  }
  torch::Tensor forward(torch::Tensor input) {
    return torch::addmm(b, input, W);
  }
  torch::Tensor W, b;
};
</code></pre>
<p>But I’m getting this error:</p>
<pre><code class="lang-auto">Exception has occurred: W32/0xC0000005
Unhandled exception at 0x00007FF868671241 (vcruntime140.dll) in bmo.exe: 0xC0000005: Access violation reading location 0x0000001B8210038C.
</code></pre>
<p>On this line:</p>
<pre><code class="lang-auto">W = register_parameter("W", torch::randn({N, M}), true);
</code></pre>
<p>Libtorch version 2.2.2<br>
C++ compiler MSVC v142 - VS 2019 C++ x64/x86<br>
Windows</p>
Custom loss function based on model parameters

pytorchuser101: <p>Hi, I am trying to create a custom loss function to induce gradients based on change or no change in certain filter parameters within certain layers of the model.</p>
<p>I am creating a custom loss function which has cross entropy loss and the L2 norm of change in filter values added or subtracted based on the filter index within a layer.</p>
<p>This is an implementation of my code, how can I model the custom loss to obtain gradients based on model parameter’s L2 norm to an identical model ?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

def get_nested_attr(obj, attr):
    attributes = attr.split('.')
    for attribute in attributes:
        obj = getattr(obj, attribute)
    return obj

def filter_change_loss(model, model_zero, loss, layer_filter_index):
    for each_layer, filter_indexes in layer_filter_index.items():
        for i in range(len(get_nested_attr(model, each_layer).weight)):
            l2_norm = torch.norm(get_nested_attr(model, each_layer).weight[i].data, get_nested_attr(model_zero, each_layer).weight[i].data)
            if i in filter_indexes:
                loss += l2_norm
            else:
                loss -= l2_norm
    return loss

model_zero = #Some model
model = #Current model
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9, weight_decay=weight_decay)
layer_filter_index = {
    "layer1.1.conv1" = [1, 4, 18, 24, 56, 57, 52, 54, 58, 61, 63],
    "layer1.1.conv2" = [2, 5, 7, 8, 27, 29, 34, 38, 47, 49, 61]
}

for epoch in range(total_epochs):
    for i, (x, y) in enumerate(trainloader):
        optimizer.zero_grad()
        x = x.to(device)
        y = y.to(device)
        outputs = model(x)
        loss = criterion(outputs, y)
        loss = filter_change_loss(model, model_zero, loss, layer_filter_index)
        loss.backward()
        optimizer.step()
</code></pre>

ptrblck: <aside class="quote no-group" data-username="pytorchuser101" data-post="1" data-topic="203240">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pytorchuser101/48/63239_2.png" class="avatar"> pytorchuser101:</div>
<blockquote>
<p>how can I model the custom loss to obtain gradients based on model parameter’s L2 norm to an identical model ?</p>
</blockquote>
</aside>
<p>I don’t understand this question, but based on your code you would have to remove the <code>.data</code> attribute as it’s usage will skip Autograd and the L2 weight loss will thus be a constant not influencing the gradients.</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)
loss = model.weight.norm(2)
loss.backward()
print(model.weight.grad)
# tensor([[-1.]])

model = nn.Linear(1, 1)
loss = model.weight.data.norm(2)
loss.backward()
# RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn```</code></pre>

pytorchuser101: <p>Understood, I will make the changes and try it out. Thank you</p>
TorchServe log retention

Dave_Jones: <p>Hello,</p>
<p>what is the default retention policy for TorchServe logs?</p>
<p>Logs are stored inside “./logs” directory. Every day they are compressed and stored.<br>
I was wondering what is the default retention policy for these logs, and if you can set a custom log expiration date?</p>
<p>If there is no log removal, after some time, logs will take up all free space and docker container will fail.</p>
<p>I am using TorchServe version <a href="https://hub.docker.com/layers/pytorch/torchserve/0.6.0-gpu/images/sha256-cb3c483e85cfb38dfc35b921edbb64203f4237ad3f6f91b330b6231e6d718b6e?context=explore" rel="noopener nofollow ugc">0.6.0</a></p>
<p>Thanks</p>

Dave_Jones: <p>Hi,</p>
<p>it seems that TorchServe doesn’t have an inbuilt log cleaner yet.<br>
This is a problem because it will collapse a production server in time.</p>
<p>To address this, I wrote a custom python script that deletes old “*.log.gz” files.<br>
As a template, I used this <a href="https://geekflare.com/python-delete-files/" rel="noopener nofollow ugc">code</a>, and for scheduled runs, I implemented the PyPI library <a href="https://pypi.org/project/APScheduler/" rel="noopener nofollow ugc">APScheduler</a>.<br>
In the end, from my Dockerfile, I called torchserve and python script as background processes. Something like:</p>
<pre><code class="lang-auto">python ./log_cleaner.py &amp;
torchserve start
</code></pre>
<p>Hope it helps,<br>
Have a good one</p>

marksaroufim: <p>Feel free to open an issue here <a href="https://github.com/pytorch/serve" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/serve: Serve, optimize and scale PyTorch models in production</a> - sometimes it’s a diificult tradeoff for us to figure out what are common things we should support vs integrating with tools that do it better for you</p>
torchServce async handling

Yoann_Buzenet: <p>Hey there,</p>
<p>I’m trying to deploy a torchServe microservice that would handle asynchronously some embedding working then give result back to my api.</p>
<p>I can’t find if there is already an intern queue in torchServe and if I can query it 500 times and let it handle it, or if I have to build my own queue on front of it and have another service give work to it ?</p>
<p>Thank you for your help, I’m quite stuck on this!<br>
Yoann</p>