akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA driver, CUDA toolkit and Pytorch

JakobAK: <p>I have installed:</p>
<ul>
<li>NVIDIA-SMI 551.61,</li>
<li>Driver Version: 551.61,</li>
<li>CUDA Version: 12.4,<br>
and I’m about to install CUDA toolkit to enable pytorch locally.<br>
I see that the latest pytorch version is compatible with CUDA toolkit 12.1.<br>
Should I install CUDA toolkit 12.4 or 12.1?<br>
I am running windows 11 with GPU =  NVIDIA RTX 2000 Ada<br>
Thanks in advance!</li>
</ul>

ptrblck: <p>The PyTorch binaries ship with their own CUDA dependencies and your locally installed CUDA toolkit will be used if you build from source or custom CUDA extensions.</p>
While training RLHF model I am getting error like, ValueError: num_samples should be a positive integer value, but got num_samples=0

OpenEyes_VO: <p>I am using above code for training RLHF for answer generation.</p>
<pre><code class="lang-auto">learning_rate=1.41e-5
max_ppo_epochs=1
mini_batch_size=4
batch_size=16

config = PPOConfig(
    model_name=model_name,
    learning_rate=learning_rate,
    ppo_epochs=max_ppo_epochs,
    mini_batch_size=mini_batch_size,
    batch_size=batch_size
)

ppo_trainer = PPOTrainer(config=config,
                         model=ppo_model,
                         ref_model=ref_model,
                         tokenizer=tokenizer,
                         dataset=dataset_train,
                         data_collator=collator)
</code></pre>
<p>but while running this code I am getting error like</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-41-5fa467fb1768&gt; in &lt;cell line: 14&gt;()
     12 )
     13 
---&gt; 14 ppo_trainer = PPOTrainer(config=config,
     15                          model=ppo_model,
     16                          ref_model=ref_model,

3 frames
/usr/local/lib/python3.10/dist-packages/torch/utils/data/sampler.py in __init__(self, data_source, replacement, num_samples, generator)
    141 
    142         if not isinstance(self.num_samples, int) or self.num_samples &lt;= 0:
--&gt; 143             raise ValueError(f"num_samples should be a positive integer value, but got num_samples={self.num_samples}")
    144 
    145     @property

ValueError: num_samples should be a positive integer value, but got num_samples=0
</code></pre>
<p>How to solve this error? Thanks in advance.</p>
Model suggestions for pixelated enhancement of QR codes

Abdul_Rahmaan: <p>I’ve been trying to run models for deblurring and enhancing blurry and distorted qr code images via torch. However, none of them have been able to return the desired output despite being trained on customized QR code datasets (10,000 images) with variations to the models parameters.</p>
<p>Are there any models which can provide an accurate result with respect to image enhancement of grayscaled images in general that could help with this task?</p>
Applying Low rank approximation to learnable parameters

Immobile_Imbecile: <p>I am trying to understand whether it makes sense to apply Low-rank approximations to learnable parameters in a class. The goal is to reduce parameter counts.</p>
<p>I have the following custom module :</p>
<pre><code class="lang-auto">class CustomPara(nn.Module):
    
    def __init__(self, num_blocks, in_planes, out_planes, kernel_size):
        super(CustomPara, self).__init__()
        self.coefficient_shape = (num_blocks,1,1,1,1)
        blocks = [torch.Tensor(out_planes, in_planes, kernel_size, kernel_size) for _ in range(num_blocks)]
        for i in range(num_blocks): init.kaiming_normal_(blocks[i])
        self.blocks = nn.Parameter(torch.stack(blocks)) # this is what we will freeze later

    def forward(self, coefficients):
        final_blocks =  (self.blocks*coefficients).sum(0)
        return final_blocks
</code></pre>
<p>Is it possible to reduce the number of learnable parameters here using Low-rank adaptation on the <code>blocks</code> parameter?</p>
<p>Thanks.</p>
[ERROR] Problem in updating batched variable

Srinjoy_Mukherjee: <p>Hey all,<br>
I am working in RL where I am using A2C algorithm to train an agent to follow instructions. I am spawning multiple environments simultaneously and batching outputs of the environment to improve efficiency. I am attaching the code for model training <a href="https://pastebin.com/SrLZG365" rel="noopener nofollow ugc">Here</a>.</p>
<p>But the above code is giving this <a href="https://pastebin.com/rrdT0hGY" rel="noopener nofollow ugc">error</a>. and the model is defined <a href="https://pastebin.com/qLGGYJNG" rel="noopener nofollow ugc">here</a></p>
<p>I figured out the problem is due to updation of a particular hx and cx.The hx and cs are outputs of model(inputs), which depend on the scene’s/agent’s history. So if any episode starts, hx[i] and cx[i] in i^th environment, we have to zero it.This causes problems.</p>
<p>I would like to know how to solve this. Thanks and looking forward to your input.Please leet me know if you need more clarifications.</p>

soulitzer: <p>This context manager can be used to work around that error <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.graph.allow_mutation_on_saved_tensors" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.2 documentation</a>. Let me know if that works for you.</p>

Srinjoy_Mukherjee: <p>Thanks for the reply <a class="mention" href="/u/soulitzer">@soulitzer</a> , will try it and report back.</p>
Modifying weight update gradient in my training loop

Mathieu_Chene: <p>Hello,</p>
<p>I am working on a project in which I have to study the impact of low precision quantization (int8) on accuracy.<br>
I found this work that has been presented in NEURIPS 2018 that quantize the gradient used for “backprop” and keep the gradient for weight update in full resolution (FP32). [<a href="https://github.com/eladhoffer/quantized.pytorch/tree/master" class="inline-onebox" rel="noopener nofollow ugc">GitHub - eladhoffer/quantized.pytorch</a>](<a href="https://Quantized" rel="noopener nofollow ugc">https://Quantized</a> Convolutional networks using PyTorch)</p>
<p>I want also to quantize the weight update gradient before the update operation and I was wondering if it was better to quantize out1 with autograd as done in the previous workfor “backprop” gradient (quantize.py file) or to get and update the parameters gradients with .grad.</p>
<p>Modifying github code:</p>
<pre><code class="lang-auto">def conv2d_biprec(input, weight, bias=None, stride=1, padding=0, dilation=1, groups=1, num_bits_grad=None):
    out1 = F.conv2d(input.detach(), weight, bias,
                    stride, padding, dilation, groups)
    out1 = quantize_grad(out1, num_bits=num_bits_grad)
    out2 = F.conv2d(input, weight.detach(), bias.detach() if bias is not None else None,
                    stride, padding, dilation, groups)
    out2 = quantize_grad(out2, num_bits=num_bits_grad)
    return out1 + out2 - out1.detach()


def linear_biprec(input, weight, bias=None, num_bits_grad=None):
    out1 = F.linear(input.detach(), weight, bias)
    out1 = quantize_grad(out1, num_bits=num_bits)
    out2 = F.linear(input, weight.detach(), bias.detach()
                    if bias is not None else None)
    out2 = quantize_grad(out2, num_bits=num_bits_grad)
    return out1 + out2 - out1.detach()
</code></pre>
<p>Modifying gradient using .grad:</p>
<pre><code class="lang-auto">if training:
           optimizer.zero_grad()
           loss.backward()
           for p in model.parameters():
            p.grad=gemmlowpquantization(p.grad,8)
           optimizer.step() 
</code></pre>
<p>I don’t have the same impact on training (speed, final precision) when I try both, so I want to know which is more rigorous.</p>
<p>Thanks in advance for your help.</p>
<p>Mathieu</p>

Ushavilash: <p>It’s recommended to quantize the gradients directly using <code>.grad</code> for each parameter tensor.</p>

Mathieu_Chene: <p>Thank you for your answer. I’ll use the second method.</p>

Ushavilash: <p>You’re welcome and it sounds good.</p>
How to turn off gradient tracking without using 'with torch.no_grad():'

matthewleigh: <p>Hello,</p>
<p>My request is pretty simple. I want to be able to turn off autograd mechanics without having to indent my entire code in a ‘with’ block. This way I can use it dynamically. I have so far tried this:</p>
<pre><code class="lang-auto">if do_grad:
   torch.enable_grad()
else:
   torch.no_grad()
...
</code></pre>
<p>But so far I have noticed that gradients are being tracked no matter what.</p>

matthewleigh: <p>Nevermind! I found the correct function to call:</p>
<pre><code class="lang-auto">torch.set_grad_enabled(True)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">torch.set_grad_enabled(False)
</code></pre>
<p>do exactly what I want.</p>

debg48: <p>I know I am late and you have found the solution almost two years ago but for anyone who is looking for a similar solution can check out the .detach() method .</p>
NaN gradient after complex square root of 0 value

rehno: <p>Can anyone think of a nice work-around for this issue?</p>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; torch.__version__
'2.0.1+cu118'
&gt;&gt;&gt; z = torch.tensor(0. + 0.j, requires_grad=True)
&gt;&gt;&gt; z.sqrt().abs().backward()
&gt;&gt;&gt; z.grad
tensor(nan+nanj)
</code></pre>
<p>(I’m willing to post an issue, but I’d like to characterize the problem a bit more. Does it look like a bug? I haven’t properly dived into the math behind wirtinger derivatives yet)</p>
<p>I suspect I could work around this with something like <code>torch.where(z != 0, z, epsilon)</code> or by zero’ing out all nans but both seem rather awkward with complex numbers / gradients.</p>

KFrank: <p>Hi Rehno!</p>
<aside class="quote no-group quote-modified" data-username="rehno" data-post="1" data-topic="198832" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/ea5d25/48.png" class="avatar"> rehno:</div>
<blockquote>
<p>Can anyone think of a nice work-around for this issue?</p>
<pre data-code-wrap="python"><code class="lang-python">tensor(nan+nanj)
</code></pre>
<p>…<br>
Does it look like a bug?</p>
</blockquote>
</aside>
<p>This is not a bug – it is the mathematically correct result (and is to be<br>
expected).</p>
<p>The square-root function has a <a href="https://en.wikipedia.org/wiki/Branch_point" rel="noopener nofollow ugc">branch cut</a> at <code>z = 0. + 0.j</code> and its<br>
derivative there is not well defined, so <code>nan</code> is the appropriate result.</p>
<aside class="quote no-group">
<blockquote>
<p>I could work around this with something like <code>torch.where(z != 0, z, epsilon)</code> or by zero’ing out all nans but both seem rather awkward with complex numbers / gradients.</p>
</blockquote>
</aside>
<p>It would indeed be awkward.  The core problem is that you want to compute<br>
a derivative at the singular point <code>z = 0.  + 0.j</code>.</p>
<p>Let’s say you have some small, positive, real <code>epsilon</code>.  You might imagine<br>
trying to clamp <code>z</code> at some <code>epsilon</code> away from zero.  But should you set<br>
<code>z = epsilon</code> or <code>z = -epsilon</code> or <code>z = 1.j * epsilon</code>, and so on?  The<br>
ambiguity remains, even if you try to sweep it under the rug.</p>
<p>Figure out in the context of your larger computation why you are trying to<br>
compute the gradient of <code>z.sqrt()</code> when <code>z</code> is zero, and avoid doing that<br>
in the first place.</p>
<p>Best.</p>
<p>K. Frank</p>
Running inference on multiple Images on a Single on Device GPU using pytorch mobile

nbansal90: <p><strong>Use Case</strong>: I currently have a <code>model_dynamic_quantization.ptl</code>, which I have created using <code>pytorch mobile</code> using the following steps.</p>
<pre><code class="lang-auto">pth_path = '../models/net_g_latest.pth'
checkpoint = torch.load(pth_path)
model.load_state_dict(checkpoint['params'])
model.eval()

# dynamic quantization
model_quantized = torch.quantization.quantize_dynamic(model, qconfig_spec={torch.nn.Linear}, dtype=torch.qint8)

# Convert to torch mobile (ptl)
example = torch.rand(1, 3, 128, 128)
traced_script_module = torch.jit.trace(model_quantized, example)
traced_script_module_optimized = optimize_for_mobile(traced_script_module)

# Save the model
traced_script_module_optimized._save_for_lite_interpreter("model_dynamic_quantization.ptl")
</code></pre>
<p>I am looking to run this model on my end device, which has GPU as the compute. I am wondering if there is a way to parallelize model inference over multiple images on the device(GPU) using the above model.</p>
<p>Any help/suggestion is appreciated.<br>
Regards!</p>
Python3.9 dataloader and torch.nn.DataParallel errors

nikiguo93: <p>hi, when I use python3.9 torch.nn.parallel, I got the following errors:<br>
Traceback (most recent call last):<br>
File “/home/modelrep/manshan/python_examples/DataLoader/other_seeds/test_robustness.py”, line 319, in <br>
rob_acc_multi_real = run_attack(TestdataLoader,fmodel_multi_real,epsilons,device).numpy()<br>
File “/home/modelrep/manshan/python_examples/DataLoader/other_seeds/test_robustness.py”, line 169, in run_attack<br>
for i_batch, sample_batched in enumerate(TestdataLoader):<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 631, in <strong>next</strong><br>
data = self._next_data()<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 1329, in _next_data<br>
idx, data = self._get_data()<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 1295, in _get_data<br>
success, data = self._try_get_data()<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 1133, in _try_get_data<br>
data = self._data_queue.get(timeout=timeout)<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/multiprocessing/queues.py”, line 122, in get<br>
return _ForkingPickler.loads(res)<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/site-packages/torch/multiprocessing/reductions.py”, line 495, in rebuild_storage_fd<br>
fd = df.detach()<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/multiprocessing/resource_sharer.py”, line 57, in detach<br>
with _resource_sharer.get_connection(self._id) as conn:<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/multiprocessing/resource_sharer.py”, line 86, in get_connection<br>
c = Client(address, authkey=process.current_process().authkey)<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/multiprocessing/connection.py”, line 502, in Client<br>
c = SocketClient(address)<br>
File “/scratch/modelrep/manshan/anaconda/envs/branch-env/lib/python3.9/multiprocessing/connection.py”, line 630, in SocketClient<br>
s.connect(address)<br>
FileNotFoundError: [Errno 2] No such file or directory</p>
<p>Can any body tell me how to fix this problem?</p>
Poor performance with Masking/Sparcity in linear layers

Ono: <p>I have a model that uses linear layers where most of the parameters are zero. I’d like to maintain the structure of the linear layers while reducing the effective number of trainable parameters. I’ve been experimenting with sparse tensors and masked tensors, but I’m not getting the performance boost I’d hoped from my simple experiments. Does anyone have any suggestions for how to use these properly?</p>
<p>More details here… Here are a few simple modules that capture the essence of what I need. The first is just a classic NxN linear module. The second uses masked tensors to reduce the number of non-zero parameters to 1/10 of the fully connected layer. The third attempts to do the same thing using sparsity.</p>
<pre><code class="lang-auto">class Classic_Linear_Module(nn.Module): 
    def __init__(self, N=256, device="cpu"): 
        super().__init__()
        self.device = device
        self.Linear = nn.Linear(N, N, bias=False, device=device)
        self.Linear.weight = torch.nn.init.uniform_(self.Linear.weight, a=0.0, b=1.0)
        self = self.to(device)
        
    def forward(self, x):
        return self.Linear(x)
    
class Masked_Linear_Module(nn.Module): 
    def __init__(self, N=256, device="cpu"): 
        super().__init__()
        self.device = device
        Linear = torch.rand(N, N, device=device)
        mask = Linear &lt;= .1
        Masked_Linear = masked_tensor(Linear, mask).to(device)
        self.Masked_Linear = torch.nn.Parameter(Masked_Linear)
        self = self.to(device)
        
    def forward(self, x):
        Linear = self.Masked_Linear.to_tensor(0).to(self.device)
        return torch.matmul(x, Linear)
    
class Sparse_Linear_Module(nn.Module): 
    def __init__(self, N=256, device="cpu"): 
        super().__init__()
        Linear = torch.rand(N, N)
        mask = Linear &lt;= .1
        Sparse_Linear = masked_tensor(Linear, mask).to_tensor(0.0).to_sparse()
        self.Sparse_Linear = torch.nn.Parameter(Sparse_Linear)
        
    def forward(self, x):
        return torch.matmul(x, self.Sparse_Linear)
</code></pre>
<p>Note that the masked tensor first converts to an ordinary tensor before applying matrix multiplication because matmul and other matrix operations don’t appear to be implemented yet (maybe there’s a better way?).</p>
<p>I tested out the performance of the forward pass and backward gradient decent step with a simple loss function by simply applying it to a big batch (batch_size=256, N=2048, 100 epochs) and measuring the speed. The masked case was about 50% lower than the fully connected layer despite having one-tenth of the parameters, and the sparse implementation was about 10 times slower:</p>
<p>Classic training time: 1.2331788539886475<br>
Masked training time: 1.8363850116729736<br>
Sparse training time: 12.60797905921936</p>
<p>Similar results on gpu, but even more pronounced (masked is about 10 times slower and sparse about 100 times slower than fully connected).</p>
<p>Any ideas on how this could be implemented better?</p>
Hybrid CNN + SVM Feature Extraction

Gregory_Lanier: <p>Hello everyone,</p>
<p>First time on the PyTorch forum. This is the first time I’m working with a PyTorch project, so bare with me if this is an easy misunderstanding.</p>
<p>I have a CNN that I trained/tested using the PyTorch basic CNN tutorial, and I believe I have a feature extractor working with it. I am trying to add the features into a new dataset that I can pass to a SVM model for training. These features are saved after a FC layer, so with a batch size of 4, each feature is (4, 84). The reason I am trying to create a new dataset from the features is that I have a SVM model that iterates over a trainingloader.</p>
<p>If there is an easier way of grabbing features to pass to another model, I am all open to suggestions. Below I have attached some of the code I am using.</p>
<p>Thanks,<br>
Greg</p>
<pre><code class="lang-auto"># CNN Architecture: INPUT - CONV - RELU - MAXPOOL - CONV - RELU - MAXPOOL - FC1 - FC2 - FC3
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        self.feature_ext = create_feature_extractor(self, return_nodes=[feature_layer])

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
    
    def extract_features(self, x):
        x = self.feature_ext(x)
        return x

# Training occurs here in my code.

# Extract features from TRAINING set (as an example)
feat_list = list()
label_list = list()
with torch.no_grad():   
    for data in trainloader:
        images, labels = data
        feat_list.append(net.extract_features(images)[feature_layer])
        label_list.append(labels)

# How to turn features into a dataset from here???
</code></pre>

Gregory_Lanier: <p>I think I finally figured it out. Is there a more efficient way to pass features apart from this:</p>
<pre><code class="lang-auto"># Extract features from TRAINING set
feat_list = list()
label_list = list()
with torch.no_grad():   
    for data in trainloader:
        images, labels = data
        feat_list.append(net.extract_features(images)[feature_layer])
        label_list.append(labels)

# Convert extracted features to a dataset
feat_tensor = torch.cat(feat_list, dim=0)
label_tensor = torch.cat(label_list, dim=0)
svm_dataset = TensorDataset(feat_tensor, label_tensor)
svm_train_size = int(0.8 * len(svm_dataset))  # 80% of the data for training
svm_test_size = len(svm_dataset) - svm_train_size  # Remaining 20% for testing
svm_train_dataset, svm_test_dataset = random_split(dataset, [svm_train_size, svm_test_size])
svm_batch_size = 4
svm_trainloader = DataLoader(svm_train_dataset, batch_size=svm_batch_size, shuffle=True)
</code></pre>
Run two nets in parallel on single machine with multiple GPUs

ydzhang12345: <p>Hi, I want to run 2 different networks that take the same input in parallel, with multiple gpus.<br>
The main purpose is to achieve inference speed up. So I did:</p>
<pre><code class="lang-auto">    inputs = torch.randn(1, 3, 256, 256)
    inputs1 = inputs.to('cuda:0')
    inputs2 = inputs.to('cuda:1')
    model1 = model1.to('cuda:0')
    model2 = model2.to('cuda:1')

    s1 = torch.cuda.Stream()
    s2 = torch.cuda.Stream()
    torch.cuda.synchronize()

    i = 0
    time_spent = []
    while i &lt; 1000:
        with torch.no_grad():
            start_time = time.time()
            with torch.cuda.stream(s1):
                _ = model1(inputs1)
            with torch.cuda.stream(s2):
                _ = model2(inputs2)

        if device == 'cuda':
            torch.cuda.synchronize()
        if i &gt; 100:
            time_spent.append(time.time() - start_time) 
</code></pre>
<p>However, when I measure the average speed, I did not see any speed improvement compared to running model1 and model2 in a sequential manner on a single gpu. Am I missing something here? I am running the test on ubuntu 18.04 with two RTX-2080TIs. Many thanks!</p>

ptrblck: <p>Depending on the model and thus the workload, the CPU might not be able to run ahead and schedule the kernel launches fast enough.<br>
You could profile it using e.g. Nsight Systems and check, if the kernels are overlapping or if they are so short, that they are executed “sequentially” on these two devices.</p>

RylanSchaeffer: <p>Can anyone point me towards a clear tutorial about running multple networks in parallel, where each network is on a different device?</p>
Kernel killed due to RAM memory leak

Adex: <p>I have a utility code that takes as input the class probabilities as predicted by two image classification models <code>M1</code> and <code>M2</code>. The utility code calculates a modified version of the AUC metric. The output of model <code>M1</code> is passed to the utility code and no error is thrown. However, the output of model <code>M2</code> when passed to the utility code throws an error as described next.</p>
<p>The jupyter notebook shows ‘<code>kernel has been killed</code>’. A little bit more digging into the error shows that the error comes due to RAM memory being full (GPU memory is 24GB is less than 80% occupied) and the kernel is getting killed. I have also localized the line of code where the error is actually happening. The kernel is killed i.e. RAM is overflows (it is worth noting that I am using a 128GB RAM machine with pytorch 2.1, ubuntu 20.04) when the model executes the following code</p>
<blockquote>
<p>score[~mask]+=bias</p>
</blockquote>
<p>I have the following dimensions of the variables and data types</p>
<p><code> print(score.data_type, score.shape, mask.data_type, mask.shape)</code></p>
<p><code>&lt;class 'torch.Tensor'&gt;, torch.Size([7280]), &lt;class 'torch.Tensor'&gt;,torch.Size([7280, 278362]),</code></p>
<p>Also the output from <code>M1</code> and <code>M2</code> are not same. The outputs are dictionaries with class labels as keys and values as probabilities. The output of M1 is as follows,</p>
<blockquote>
<p>print(pred_M1[‘class_0’])</p>
</blockquote>
<p><code>tensor(0.5000),</code></p>
<blockquote>
<p>print(pred_M2[‘class_0’])</p>
</blockquote>
<p><code>tensor(0.5000))</code></p>
<p>However the following output is also observed,</p>
<blockquote>
<p>print(pred_M1[‘class_0’].item())</p>
</blockquote>
<p><code>0.5000015497207642</code></p>
<p>What can be the cause of memory leak in the RAM? Any help in this regard will be highly helpful.</p>
Understanding the GRUCell and its inputs

Max_Unhold: <p>I would like to understand the inputs of the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRUCell.html" rel="noopener nofollow ugc">GRUCell</a> but unfortunately, the documentation is very unprecise in this:</p>
<p><strong>Inputs:</strong> The inputs are declared as (N, H_in) where N is undefined.<br>
<strong>Hidden:</strong> The hidden are declared as (N,H_out) where N is undefined.</p>
<p>The exmaple takes a three dimensional input</p>
<pre><code class="lang-auto">rnn = nn.GRUCell(10, 20)
input = torch.randn(6, 3, 10)
hx = torch.randn(3, 20)
output = []
for i in range(6):
    hx = rnn(input[i], hx)
    output.append(hx)
</code></pre>
<p>If we loop over the first dimension of the input, I assume, it must be the time, making the second dimension the batch dimension, which defines N.</p>
<p>However, why is the first dimension of the hidden parameter <strong>hx</strong> a batch dimension?</p>
<p>Or, if the second is the time and the first the batch dimension, why do you loop over the batch dimension here?</p>
Light weight libtorch-cxx11 for C++ projects

tlm: <p>Hi all,<br>
I have a C++ based project (that compiles in Windows, MacOS and Linux). I have a pre-trained model in pytorch and I am performing inference (<strong>only inference and no training</strong>) using this model in my C++ project. When I distribute a standalone installer for this C++ project, I also have to ship the pytorch dependencies (libtorch-cxx11-abi-shared-with-deps-1.3.0.zip) with my software. The current size of pytorch dependencies are 1.4GB. My application size is around 2MB.</p>
<p>Is there any possibility of creating a pytorch distributable library that only contains the minimal code required for inference (no training)? I feel that 1.4GB of dependency is too big and unnecessary if I only want to perform inference.</p>
<p>I would really appreciate if the members of this forum provide me with pointers and insight on how I can embed pytorch inference functionality in my C++ projects without the burden of 1.4GB of libtorch dependency.</p>
<p>Abhishek</p>

tom: <p>If you drop GPU support, you save some. However, as long as you want MKL, you have a pretty large chunk that won’t go away.<br>
You could imitate the android builds to get a small footprint.</p>

soldierofhell: <p>Take for example Tensorflow Serving. Docker image for CPU is 50 MB, but for GPU 1 GB <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

soldierofhell: <p>And libtensorflow_cc.so compiled with CUDA and TensorRT ~ 0.5 GB</p>

tlm: <p>Thank you. The “android builds” pointer is a very useful.</p>

Aakash_Srivastava: <p><a class="mention" href="/u/tlm">@tlm</a> <a class="mention" href="/u/tom">@tom</a> Can you please provide me some hints of how this android builds helps you. I have checked the libs folder in my apk.debug. I got three .so files but how will I use it without .h file? Stuck in same situation as you were. Can you please help me as I need to showcase this as my College Projects.Please help me!!!.</p>
Can anyone help me, i want to make project anomaly detection water consumption using dqn, below is my dataset

Andhika_Ahnaf_Danisw: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa23a9e92f36aa70118706e8280b9aa8fd1f561f.png" data-download-href="https://discuss.pytorch.org/uploads/default/fa23a9e92f36aa70118706e8280b9aa8fd1f561f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa23a9e92f36aa70118706e8280b9aa8fd1f561f.png" alt="image" data-base62-sha1="zGPM9TxjxNpAkBygFuc1W22N8aX" width="690" height="374" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1249×678 27 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I don’t think anyone is able to help you if the only information you are sharing is a screenshot of (a subset of) data.<br>
Explain what kind of issue you are running into and where you are stuck.</p>
Extracting features using my QNN Hybrid model

sleepingcat4: <p>I’ve created a Hybrid Quantum CNN model. And now after calling the instance: I want to use the model as feature extractor and receive the features from training images. I’m a bit confused, how to achieve this in Pytorch. <strong>[I can do this on Tensorflow but not on Pytorch yet]</strong></p>
<p><strong>Code</strong></p>
<pre data-code-wrap="Python"><code class="lang-Python">weights = torchvision.models.ResNet18_Weights.IMAGENET1K_V1
model_hybrid = torchvision.models.resnet18(weights=weights)

for param in model_hybrid.parameters():
    param.requires_grad = False

# Notice that model_hybrid.fc is the last layer of ResNet18
model_hybrid.fc = DressedQuantumNet()

# Use CUDA or CPU according to the "device" object.
model_hybrid = model_hybrid.to(device)
</code></pre>
<p>Can someone help on how can I extract features from training images? using <code>model_hybrid</code>?</p>

ptrblck: <p>You could use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> or could alternatively replace the classifier with <code>nn.Identity</code> layers if you won’t need the actual classifier output anymore.</p>
CNN-LSTM Image Captioner: Testing Output Problems

Jack_Archer: <pre><code>def forward(self, features, captions): #for the training phase
    #e.g input feature -&gt; (b, 768), caption -&gt; (b,seq_length, vocab_size)
    features = features.unsqueeze(1)
    # features = (b,1,758)
    h_0= torch.zeros(self.num_layers, features.shape[0], self.hidden_size).to(device)
    c_0 = torch.zeros(self.num_layers, features.shape[0], self.hidden_size).to(device)
    # teacher forcing
    if captions is not None:       
        #stacking image features and caption embeddings in 1d array
        all_embeddings = torch.cat((features, captions[:,:-1,:]), dim =1) #captions shifted
        # (b,1,embed_dim) + (b,50-1,embed_dim)
        #LSTM
        logits, _ = self.lstm(all_embeddings)
        
        #linear that feeds to the next LSTM cell and also contains the previous state
        prob_dist = self.linear(logits)
        # softmax handled is cross entropy
        return prob_dist
    
    else:
        inputs = features 
        states = (h_0,c_0)
        decoder_output = torch.zeros(features.shape[0],self.word_max,self.vocab_size)
        # decoder_output (b, seq_length, vocab_size) stores output for batch 
        for t in range(self.word_max):
            # inputs = (b,1, embed_dim = 768)
            output_step, states = self.lstm(inputs,states)
            # output_step = b,1,hidden_size 
            logits = self.linear(output_step)
            # logits (b,1, vocab_size)
            pred_word = self.greedy_word(logits)
            #(b,1,3000) one hot to plug into decoder_output for loss calc in testing
            decoder_output[:, t:t+1, :] = pred_word
            # fill in the n th  word for each caption
            
            # now embed next word to feed into lstm
            input_ids = logits.argmax(dim=-1).to(device)
           # (b,1)
            bert_outputs = self.bert_model(input_ids,attention_mask=None)
            inputs = bert_outputs.last_hidden_state
            # (b,1, 768)
        return decoder_output
</code></pre>
<p>[/quote]<br>
Training is excellent. 90%+ accuracy. Testing I’m getting nonsensical sentences like “on there there there there.”  I am using BERT embeddings. I’m new to both this forum and machine learning so any help would be appreciated.</p>
Creating a heatmap on 3 points of the image. The key points have the highest intensity which fades in all directions following a gaussian distribution

Bfuraide: <p>Could you help with a code that creates a heatmap with highest intensity at the key point and the intensity fades in all directions follow a guassian distribution with a standard deviation of 4</p>

JuanFMontesinos: <pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt

def gaussian_heatmap(height, width, center, std_dev=4):
    """
    Args:
    - height (int): Height of the heatmap.
    - width (int): Width of the heatmap.
    - center (tuple): The (x, y) coordinates of the Gaussian peak.
    - std_dev (int, optional): Standard deviation of the Gaussian.

    """
    x_axis = torch.arange(width).float() - center[0]
    y_axis = torch.arange(height).float() - center[1]
    x, y = torch.meshgrid(x_axis, y_axis, indexing='ij')
    
    return  torch.exp(-((x ** 2 + y ** 2) / (2 * std_dev ** 2)))

height, width = 20, 20
center = (10, 10)

heatmap = gaussian_heatmap(height, width, center)
plt.imshow(heatmap, cmap='hot', interpolation='nearest')
plt.colorbar()
plt.show()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d606f2d16ebd15d36271fa4ef29812bf5a3e4dd0.png" data-download-href="https://discuss.pytorch.org/uploads/default/d606f2d16ebd15d36271fa4ef29812bf5a3e4dd0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d606f2d16ebd15d36271fa4ef29812bf5a3e4dd0_2_605x500.png" alt="image" data-base62-sha1="uxn44SHrtGL52X2FFVub43bTv56" width="605" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d606f2d16ebd15d36271fa4ef29812bf5a3e4dd0_2_605x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d606f2d16ebd15d36271fa4ef29812bf5a3e4dd0_2_907x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d606f2d16ebd15d36271fa4ef29812bf5a3e4dd0_2_1210x1000.png 2x" data-dominant-color="A66043"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1235×1019 49.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Bfuraide: <p>Thanks, i have the image key points and their cordinates in a json file. I should be able to have this heat map on each point shown on the image. Then the center cordinated returned and distances between these cordinates calculated.</p>

Bfuraide: <p>How can i have it imposed on the key points. Can each point have different centre colour intensity that fades to zero instead of having different colours.</p>
CUDA illegal memory access for cuda extension in multiprocessing

lhai37: <p>I wrote a simple CUDA extension to multiply a tensor in-place on GPU, it works fine on single process but in multiprocessing mode it gets the illegal memory access error. I’ve tried <code>compute-sanitizer</code> as well but didn’t get any more meaningful info. Here’s <a href="https://drive.google.com/drive/folders/1FSBLD2PvoahNmw5-TWt6x5nQmGwNx3ym?usp=sharing" rel="noopener nofollow ugc">a minimal reproducible code package</a>, simply run <code>python setup.py install &amp;&amp; python test_inplacemul.py</code>.</p>
<p>Here are the main snippets for a quick preview:</p>
<p>CUDA implementation:</p>
<pre><code class="lang-auto">__global__ void raw_mul_cuda_forward_kernel(float* x) {
  const int r = blockIdx.x;
  const int c = blockIdx.y;
  const int d = threadIdx.x;
  const int D = blockDim.x;
  x[r * D * D + c * D + d] *= 2.f;
}

void mul_cuda_forward(torch::Tensor x) {
  const auto H = x.size(0);
  const auto W = x.size(1);
  const auto D = x.size(2);
  const int threads = D;
  const dim3 blocks(H, W);
  raw_mul_cuda_forward_kernel&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(x.data&lt;float&gt;());
  cudaDeviceSynchronize();
}
</code></pre>
<p>In Python:</p>
<pre><code class="lang-auto">import torch
import inplacemul_cuda
from tqdm import tqdm

def error_callback(error):
	print(error, flush=True)

def process(rank):
    device = torch.device(f"cuda:{rank}")
    x = torch.rand(32, 32, 32).to(device)
    gold = x * 2
    inplacemul_cuda.forward(x)
    assert torch.abs(gold - x).sum() == 0

if __name__ == '__main__':
    assert torch.cuda.is_available()

    # sequential test
    for _ in tqdm(range(10)):
        process(0)
    print('Single-process test passed')

    # multiproc test
    import torch.multiprocessing as mp
    ctx = mp.get_context("spawn")
    N = 1000
    pool = ctx.Pool(processes=2)
    pbar = tqdm(total=N)
    rets = []
    num_gpus = torch.cuda.device_count()
    gpu = 0
    for _ in range(N):
        ret = pool.apply_async(process, args=(gpu,), 
            callback=lambda _: pbar.update(1), 
            error_callback=error_callback)
        rets.append(ret)
        gpu = (gpu + 1) % num_gpus
    pool.close()
    pool.join()
    for ret in rets:
        ret.get()
</code></pre>

ptrblck: <p>You might need to add a device guard via:</p>
<pre><code class="lang-auto">at::cuda::OptionalCUDAGuard device_guard(id)
</code></pre>

lhai37: <p>Yes that works, thanks for the quick reply. For reference: I added <code>#include &lt;c10/cuda/CUDAGuard.h&gt;</code> at the top, and <code>at::cuda::OptionalCUDAGuard device_guard(x.device());</code> before the kernel launch.</p>
Pytorch Model to access a Tensor store on GPU

torchlearner: <p>Hello team,</p>
<p>I have a use-case like this. A small tensor store is created on GPU. We can think of this store as a hashmap where the keys represent the entities and the values represent a set of feature tensors for each entity. During inference the model gets a request with some context which has request tensors. For every inference (forward() call), the model needs to run operations using the request tensors and each of the entity tensors which is on GPU. The tensors in the store could be updated continuously by an external process. New entities with its set of feature tensors could also be added externally.</p>
<p>How can I have the forward pass of the model refer to these feature tensors as part of its operations and also get the most updated value and also consider the new entities ? Technically the interface looks like a  pointer from model variables to a gpu data structure.</p>
<p>If the store was not managed externally, I would have defined the feature tensors as part of the model and maintained a dictionary of entity to feature tensors, and forward() would have accessed that dictionary.</p>
<p>Thanks</p>
OutOfMemoryError: CUDA out of memory error during Gradient loss accumulation

Saimon: <p>Hi,<br>
The following code is showing CUDA out of memory error. Is there any way to make this calculation computationally efficient? I also need to keep track of the computation graph so that the graph doesn’t break; as I need to backpropagate through it during training. Here, I am basically trying to calculate the partial differential equation loss where the right hand side is just zero. Many thanks!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7ef7333877ae38235d0fa35e90cc0bbfae3d14a7.png" data-download-href="https://discuss.pytorch.org/uploads/default/7ef7333877ae38235d0fa35e90cc0bbfae3d14a7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7ef7333877ae38235d0fa35e90cc0bbfae3d14a7_2_690x358.png" alt="image" data-base62-sha1="i7bMFyXboCDVdjHECU0eJEYuQfB" width="690" height="358" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7ef7333877ae38235d0fa35e90cc0bbfae3d14a7_2_690x358.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7ef7333877ae38235d0fa35e90cc0bbfae3d14a7_2_1035x537.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7ef7333877ae38235d0fa35e90cc0bbfae3d14a7_2_1380x716.png 2x" data-dominant-color="EEF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1464×760 86.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You are most likely running OOM since you are explicitly keeping the computation graph with its intermediate activations alive via <code>retain_graph=True</code>.<br>
Are you sure you need to keep it alive? Since you are recomputing <code>u_x_y</code> in each iteration I would speculate you might be able to delete the graphs after the gradients are computed.</p>
How to build it with cuda for best performance?

nooodles2023: <p>There are so many environment variables need to set, such as USE_CUDA, USE_CUDNN…<br>
For best performance, what vars are needed to be set  with cuda?</p>

ptrblck: <p>The pre-built binaries already ship with all libraries and you could additionally enabled TF32 for <code>matmul</code>s as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices">here</a> and check the <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a>.</p>

nooodles2023: <p>I want to forward cuda api, so I have to compile pytorch myself to make it not use static cudart.<br>
But I found that the proformance of my cuda forwarding was poor, so I had to find the difference between official builds and mine.</p>
Why does autograd::engine::evaluate_function: CudnnConvolutionBackward0 takes too much host time?

Kenan_Yip: <p>I ran my training code on different machines and got varied time consumption, from 20min per epoch to 60min per epoch. (AMD 5950x and Intel 4210R)<br>
Thus, I used the torch.profiler to check the bottleneck. While from the overview summary it seems to be the CPU bottleneck, the operator profile shows top 4 cudnn backward operations contributing most device total time duration with 0 self-time duration (3 in 4 operators).<br>
Does this mean they are waiting for the CPU?<br>
Additionally, my code is using GRU cells with for loop to iterate sequences, and I found a page similarly suffering from autograd overhead (<a href="https://discuss.pytorch.org/t/why-is-my-training-bottlenecked-on-cpu/7176/2" class="inline-onebox">Why is my training bottlenecked on CPU? - #2 by smth</a>).<br>
I understand in my case the CPU performance gap seems to be the dominant cause, I still wondering if there’s any chance to reduce this executing time gap.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Category</th>
<th>Time Duration (us)</th>
<th>Percentage (%)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Average Step Time</td>
<td>8,347,976</td>
<td>100</td>
</tr>
<tr>
<td>Kernel</td>
<td>1,211,537</td>
<td>14.51</td>
</tr>
<tr>
<td>Memcpy</td>
<td>73,742</td>
<td>0.88</td>
</tr>
<tr>
<td>Memset</td>
<td>7,329</td>
<td>0.09</td>
</tr>
<tr>
<td>Runtime</td>
<td>574,164</td>
<td>6.88</td>
</tr>
<tr>
<td>DataLoader</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>CPU Exec</td>
<td>5,360,633</td>
<td>64.21</td>
</tr>
<tr>
<td>Other</td>
<td>1,120,572</td>
<td>13.42</td>
</tr>
</tbody>
</table>
</div><div class="md-table">
<table>
<thead>
<tr>
<th>Name</th>
<th>Calls</th>
<th>Device Self Duration (us)</th>
<th>Device Total Duration (us)</th>
<th>Host Self Duration (us)</th>
<th>Host Total Duration (us)</th>
<th>Tensor Cores Eligible</th>
<th>Tensor Cores Self(%)</th>
<th>Tensor Cores Total(%)</th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<td>autograd::engine::evaluate_function: CudnnConvolutionBackward0</td>
<td>3924</td>
<td>0</td>
<td>1386348</td>
<td>128347</td>
<td>1348531</td>
<td>Yes</td>
<td>0</td>
<td>3.53</td>
<td>View CallStack</td>
</tr>
<tr>
<td>aten::cudnn_convolution_backward</td>
<td>3924</td>
<td>0</td>
<td>1375288</td>
<td>93613</td>
<td>1157994</td>
<td>Yes</td>
<td>0</td>
<td>3.55</td>
<td>View CallStack</td>
</tr>
<tr>
<td>CudnnConvolutionBackward0</td>
<td>3924</td>
<td>0</td>
<td>1375288</td>
<td>58735</td>
<td>1216729</td>
<td>Yes</td>
<td>0</td>
<td>3.55</td>
<td>View CallStack</td>
</tr>
<tr>
<td>aten::cudnn_convolution_backward_weight</td>
<td>3924</td>
<td>1119203</td>
<td>1119203</td>
<td>301089</td>
<td>520406</td>
<td>Yes</td>
<td>0</td>
<td>0</td>
<td></td>
</tr>
</tbody>
</table>
</div>

CCLDArjun: <p><a class="mention" href="/u/kenan_yip">@Kenan_Yip</a> were you able to find a solution? I’m having a similar issue with <code>autograd::engine::evaluate_function: CudnnConvolutionBackward0 </code> taking a lot of time</p>
How to use only one of the RGB channels in forward function?

lemonja: <p>I want to build a CNN for image classification. The input images should be of type RGB but every channel is just the same grayscale image (intentionally). Nevertheless, I only want to use the first channel of these three in the forward function for the conv2d layers. I’ve tried something like:</p>
<pre><code class="lang-auto">...

def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, padding=2)
        ...

def forward(self, x):
        x = x[:,0]
        x = F.relu(self.conv1(x))
        ...
</code></pre>
<p>I’ve got the following error message while training (batch size is 64, images are 32x32):</p>
<blockquote>
<p>RuntimeError: Given groups=1, weight of size [32, 1, 3, 3], expected input[1, 64, 30, 30] to have 1 channels, but got 64 channels instead</p>
</blockquote>
<p>I can imagine some mistake in the slicing of <code>x</code>. I’ve tried different slicing versions (<code>x[0]</code>, <code>x[:,0]</code>, <code>x[:,:,0]</code>) for this reason but nothing worked.</p>
<p>Does anyone has a clue what I am doing wrong here?</p>

i4ata: <p>The issue is that indexing x like this removes the ‘channels’ dimension. That is, if you run:</p>
<pre><code class="lang-auto">x = torch.rand(64, 3, 32, 32) # [batch_size, channels, height, width]
x = x[:, 0]
print(x.size())
</code></pre>
<p>The output is [64, 32, 32]. However, for x to be compatible with the convolutional layer, you would need shape [64, 1, 32, 32]. That can be done with ‘unsqueeze’, e.g.:</p>
<pre><code class="lang-auto">def forward(self, x):
    x = x[:,0].unsqueeze(1) # 3 channels -&gt; 1 channel
    x = F.relu(self.conv1(x)
    ...
</code></pre>

lemonja: <p><a class="mention" href="/u/i4ata">@i4ata</a> thanks for your help and the nice explanation!</p>