akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torchvision Faster R-CNN custom

asdfv904: <p>I’m currently implementing the model introduced in <a href="https://dl.acm.org/doi/10.1145/3402597.3402605" rel="noopener nofollow ugc">here</a>, where it just simply adds a branch that detects rotated bounding boxes to Faster R-CNN roi head. I’ve adapted torchvision Faster R-CNN model at <a href="https://github.com/crazyboy9103/oriented_detection" class="inline-onebox" rel="noopener nofollow ugc">GitHub - crazyboy9103/oriented_detection</a>, but it does not seem to work well. It is able to detect bounding boxes, but their classifications are mostly wrong and I don’t know where to fix. The dataset I’m using is MVTec rotated screws dataset.</p>
<p>Could anyone help me with this matter?<br>
In the repository, I’ve mainly adapted models/detection/box_coders.py, roi_heads.py, rotated_fatster_rcnn.py.</p>
How to resolve runtime error related to compile PyTorch with LAPACK support?

Jerry_Gill: <p>I am running DNN model on synthetic datasets to draw decision boundaries polytope in order to study the tropical geometrical perspective of synthetic adversarial attack. And getting the following runtime error:</p>
<blockquote>
<pre><code> Cell In[26], line 20
     17 sigma      = 0  #Sparsity Factor, CAUTION, this number should be less than epsilon1 otherwise, no   solution will be found
     19 print(f'The model was classifing the x_0 as class {model_new(x_0).unsqueeze(dim=0).argmax(1).item()}, The target class is {target.item()}')
---&gt; 20 delta_x,z,w,delta_a,eqq = str_atk_tg_for_syn(model_new,x_0,target,criterion,epsilon1,epsilon2,rho,eta,lamda,sigma,iterations)
     21 print(f'The attacked model is now classifying x_0 as class {model_new(x_0+delta_x).unsqueeze(dim=0).argmax(1).item()}')

Cell In[14], line 36, in str_atk_tg_for_syn(net, x, target, criterion, eps1, eps2, rho_0, eta, lamda, sigma, iterations)
     34 temp2   = 2*lamda*torch.mm(torch.mm(A.t(),delta_a),x.squeeze().reshape(-1,1))
     35 temp2   = temp2.reshape(z.shape) + rho*z.squeeze() - u.squeeze() # This is the    B in Ax = B
---&gt; 36 delta,_ = torch.linalg.solve(temp1, temp2.reshape(-1,1))
     37 delta   = delta.reshape(z.shape)
     39 #Updating w
</code></pre>
</blockquote>
<blockquote>
<pre><code>RuntimeError: Calling torch.linalg.lu_factor on a CPU tensor requires compiling PyTorch with LAPACK. Please use PyTorch built with LAPACK support.
</code></pre>
</blockquote>
<p>I installed LAPACK and related packages in anaconda environment using</p>
<blockquote>
<p>conda install -c “conda-forge/label/lapack_rc” lapack</p>
</blockquote>
<p>command but still getting this runtime error. How can I compile Pytorch with LAPACK support?</p>

Ed_Moman: <p>I am a newbie but my wild guess is that you are running the code on cpu what you need is mkl and mkl-include and, if you are using gpu, you need magma.</p>
<p>Then recompile.</p>
Segmentation fault with multiple devices

Ed_Moman: <p>I was training my model in two RTX4090 GPUS by setting the following:</p>
<pre><code class="lang-auto">os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0,1"
</code></pre>
<p>I am running the nightly version of PyTorch and it was working fine until a couple of days ago.</p>
<p>Now it produces segmentation fault (core dumped) errors or, occasionally bus errors.</p>
<p>Two things happened in between:</p>
<ul>
<li>
<p>I updated pytorch.</p>
</li>
<li>
<p>The driver was updated to version 535.98</p>
</li>
</ul>
<p>Cuda is 12.1.</p>
<p>Is anyone experiencing similar issues with either the new most recent development branches of pytorch or with the new driver?</p>
<p>Using either GPU works fine, but I can no longer use both simultaneously as I used to.</p>

Ed_Moman: <p>I have downgraded the driver to the last working version (535.86) but it is still impossible to use both GPUs.</p>

Ed_Moman: <p>The problem only occurs with gradient check-pointing.</p>
Flash Attention Fused Kernel

mathnow: <p>Hi Community,</p>
<p>I have a question regarding the flash attention implementation of Pytorch 2.0.<br>
Does PyTorch ‘only’ implement the fused kernel, i.e. all self-attention operations with a single read/write to HBM?</p>
<p>Or, does it also implement tiling and checkpointing/recomputation during backprop as in the Flash Attention paper? I’m not a hardware guy and this question might be simple, but it was not obvious to me when I read through the paper and the docs.</p>
<p>Another question that is more related to Flash Attention itself: Why is memory usage linear in sequence length during the forward pass? I was expecting sublinear due to tiling, but probably I’m missing something.</p>
<p>Thanks in advance</p>
Torch autocast's gradient

gusye1234: <p>Hi guys, I’m trying to understand how <code>torch.amp.autocast</code> works. The following is a minimal code example:</p>
<pre><code class="lang-auto">def see():
    model = torch.nn.Linear(10, 5).cuda()
    for p in model.parameters():
        p.data.fill_(0)
    X = torch.rand(3, 10).cuda()
    optimizer = torch.optim.SGD(model.parameters(), 0.001)
    scaler = torch.cuda.amp.GradScaler()
    with torch.cuda.amp.autocast():
        out = model(X)
        loss = out.mean()
        print(out.dtype, loss.dtype)
    scaler.scale(loss).backward()
   # output the gradient dtype
    print(next(model.parameters()).grad, next(model.parameters()).grad.dtype)
    scaler.step(optimizer)
    scaler.update()
</code></pre>
<p>and I noticed that the gradient’s precision is actually single-precision(FP32), which is weird. According to the paper “<em>Mixed precision training</em>”, shouldn’t it be FP16?</p>

ptrblck: <aside class="quote no-group" data-username="gusye1234" data-post="1" data-topic="169003">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gusye1234/48/34887_2.png" class="avatar"> gusye1234:</div>
<blockquote>
<p>and I noticed that the gradient’s precision is actually single-precision(FP32), which is weird.</p>
</blockquote>
</aside>
<p>That’s expected since the gradient <code>dtype</code> matches the parameter <code>dtype</code>, which is <code>float32</code> during mixed-precision training.</p>

gusye1234: <p>So is there an FP16 version of gradient during training? or FP32 all the time.</p>

fiona: <p>I have the same question. Do you have understood it?</p>
Use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor)

rgb1380: <p>Here is the froward() method of a loss class that calculates loss based on the elements of the confusion matrix. <code>c_loss</code> is float and is being converted to a tensor. This line results in the following warning:</p>
<p><code>UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).   loss = torch.tensor(c_loss, requires_grad=True)</code></p>
<p>The warning makes no sense as <code>c_loss</code> is not a tensor. What am I doing wrong here?</p>
<pre><code class="lang-auto">    def forward(self, inputs: torch.Tensor, targets: torch.Tensor):
        _, predictions = torch.max(inputs, 1)
        c = confusion_matrix(targets.cpu().numpy(), predictions.cpu().numpy())

        c_loss = c[0, 1] / (c[0, 1] + c[1, 1])
        # this line results in the warning
        loss = torch.tensor(c_loss, requires_grad=True)

        return loss
</code></pre>

ptrblck: <p>What does <code>type(c_loss)</code> show?</p>

rgb1380: <aside class="quote no-group" data-username="rgb1380" data-post="1" data-topic="186703">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/59ef9b/48.png" class="avatar"> rgb1380:</div>
<blockquote>
<p><code> construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).   loss = torch.tensor(c_loss, requires_grad=True)</code></p>
<p>The warning makes no sense as <code>c_loss</code> is not a tensor. What am I doing wrong here?</p>
</blockquote>
</aside>
<p>it is just float. That’s why the message does not make sense to me.</p>

ptrblck: <p>A <code>float</code> wouldn’t raise the warning so did you <code>print</code> the <code>type</code> as asked?</p>
((x - x_hat)**2).sum()/len(x) vs nn.functional.mse_loss(x_hat, x)

hwaseem04: <p>Consider the following training code for Autoencoders</p>
<pre><code class="lang-auto">def train(model, data, epoch, lr=0.001):
    opt = torch.optim.Adam(model.parameters(), lr)
    train_loss = [0] * epoch
    for i in range(epoch):
        for x, y in data:
            x = x.to(device)
            opt.zero_grad()
            x_hat = model(x)
            loss = nn.functional.mse_loss(x_hat, x)
#             loss = ((x - x_hat)**2).sum()/len(x)
            loss.backward()
            opt.step()
            train_loss[i] += (loss * len(x))
        train_loss[i] /= len(data.dataset)
        print(f'Epoch: {i+1}/{epoch}, loss: {train_loss[i]}')
</code></pre>
<p>Why do I get different value when I use  loss = ((x - x_hat)**2).sum()/len(x)  and loss =<br>
nn.functional.mse_loss(x_hat, x), isnt both the same?</p>

ptrblck: <p>No, these approaches are not the same depending on the tensor shape.<br>
As described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss">docs</a> <code>F.mse_loss</code> will calculate <code>mean(L)</code> by default, which would correspond to a division by the number of elements, not the batch size.</p>
<pre><code class="lang-python">x = torch.randn(10, 10)
x_hat = torch.randn(10, 10)

print(F.mse_loss(x_hat, x))
# tensor(1.7013)
print(((x - x_hat)**2).sum()/ len(x))
# tensor(17.0125)
print(((x - x_hat)**2).sum()/ x.nelement())
# tensor(1.7013)
print(((x - x_hat)**2).mean())
# tensor(1.7013)
</code></pre>
Pytorch and torchvision compatability error

Benny_Johnson: <p>Hi , I installed pytorch (1.12.0a0+2c916ef.nv22.3) and torch vision(0.13.0) on Jetpack 5.0.2.</p>
<p>getting below error while running yolov5</p>
<p>RuntimeError: Couldn’t load custom C++ ops. This can happen if your PyTorch and torchvision versions are incompatible, or if you had errors while compiling torchvision from source. For further information on the compatible versions, check <a href="https://github.com/pytorch/vision#installation" rel="noopener nofollow ugc">GitHub - pytorch/vision: Datasets, Transforms and Models specific to Computer Vision</a> for the compatibility matrix. Please check your PyTorch version with torch.<strong>version</strong> and your torchvision version with torchvision.<strong>version</strong> and verify if they are compatible, and if not please reinstall torchvision so that it matches your PyTorch install.</p>
<p>Kindly help to resolve<br>
thanks</p>

ptrblck: <p>Your <code>torchvision</code> version might be too new, so try to downgrade it to e.g. <code>0.12</code>.</p>

Izis_kanjarawy: <p>Please can u tell me how did u fix it …I am facing the same problem while running yolov7 …the version of torch is 1.8 and the version of torchvision is 0.9</p>

ptrblck: <p><code>torch==1.8.0</code> and <code>torchvision==0.9.0</code> were both released on March, 4th, 2021, and should thus be compatible.</p>
Does the pytorch has a tool to convert data from GPU to CPU and from CPU to GPU automatically when the GPU memory is not enough?

clockzhong: <p>Hi, all,</p>
<p>I’m facing a tedious problem when using pytorch tensor’s ops APIs, because I want to use GPU’s performance power to accelerate my data processing speed, but my GPU’s memory size is too small, so I need cut my operands tensors into smaller size and after getting the result, move it to the CPU main memory, and after getting all the small parts of the results, I combine them into one final results in the CPU’s main memory. It means I need manually do the operands tensors cutting work and combining work of the result, and moving those tensors between GPU memory and CPU memory, and for all different Operators in pytorch, I need manually do this work all again and again.</p>
<p>My question is: do we have any tool to do this data cutting and result combining work automatically currently? Then I could fully utilize my GPU’s calculating performance and my CPU memory’s size.</p>
<p>Thanks in advance!</p>
<p>Clock ZHONG</p>

ptrblck: <aside class="quote no-group" data-username="clockzhong" data-post="1" data-topic="186612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/clockzhong/48/39254_2.png" class="avatar"> clockzhong:</div>
<blockquote>
<p>Then I could fully utilize my GPU’s calculating performance and my CPU memory’s size.</p>
</blockquote>
</aside>
<p>For a potentially large performance penalty due to the additional data movement.</p>
<p>You could offload data to the CPU as described <a href="https://discuss.pytorch.org/t/modifying-forward-backward-pass/169687/2">here</a> or you could try to write a custom allocator with offloading capabilities using <a href="https://pytorch.org/docs/stable/generated/torch.cuda.CUDAPluggableAllocator.html#torch.cuda.CUDAPluggableAllocator"><code>torch.cuda.CUDAPluggableAllocator</code></a>.</p>

clockzhong: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your reply, I mean I want to us unified or similar tensor operator APIs which could calculate tensors operations in GPU, but the input and result tensors are stored in CPU, then I could use them as what I could use in pure CPU or GPU device, I mean I want to API’s framework to do the data moving work between GPU and CPU for me. I didn’t mention the model training yet, just use the basic pytorch tensor’s APIs, currently I do these data moving work manually case by case.</p>
<p>Thanks again.</p>
<p>Clock</p>

ptrblck: <p>Both suggestions should work for your use case, where the former (CPU offloading) is more explicit than the latter (custom allocator using unified memory).</p>
Can i compile pytorch to run on android platform using maui?

netif121: <p>Can i compile pytorch to run on android platform using maui. I know there are other solutions like moving the model from pytorch to onnx and running it on android or using pytorch mobile solution. I don’t have much knowledge in this field. That’s why I’m asking if it’s even possible ?</p>
<p>I tried to compile pytorch for ARM architecture, but I got several errors among others that it can’t see the identifier _addcarry_u64.</p>
Trying to backward through the graph a second time. Model composition

Bartosz_Brzoza: <p>Hi everyone,<br>
I have two models, say model_a, model_b.<br>
model_b takes as input the output of model_a as well as some other additional input.<br>
For each input to model_a there are several (large amount) of additional inputs to model_b.<br>
I want to make a gradient step for model_b’s weights for each of the additional inputs while accumulating gradients on model_a’s weights and only making a gradient step after all additional inputs for model_b have been forwarded.<br>
I want to achieve something like this:</p>
<pre><code class="lang-python">examples = 10
additional_input_per_example = 100
batch_size = 32
hidden_dim = 3

for i in range(examples):
  x1 = torch.randn(batch_size, hidden_dim)
  enc = model_a(x1) # Assume this takes 1 hour
  for j in range(additional_input_per_example):
    x2 = torch.randn(batch_size, hidden_dim)
    y = torch.randn(batch_size, hidden_dim)
    y_hat = model_b(enc, x2) # Assume this takes 5 seconds
    loss = criterion(y_hat, y)
    loss.backward()
    optimizer_b.step()
    optimizer_b.zero_grad()
  optimizer_a.step()
  optimizer_a.zero_grad()
</code></pre>
<p>If I move the calculation of “emb” into the inner loop it works, but is highly inefficient due to recomputation of “emb”.<br>
Is using the option retain_graph=True absolutelly essential?<br>
This code works when using this option, but in my actual case this increases the memory consumption several times over and seems unnecessary, since I only need to accumulate gradients in model_a weights and I am not using the graph for anything else.<br>
Thanks in advance for your help</p>

ptrblck: <p>In your current code snippet you are calculating the gradients w.r.t. <code>model_a</code>’s parameters multiple times inside the nested for loop. I.e. <code>loss.backward()</code> is attached to <code>model_b</code> and via <code>enc</code> to <code>model_a</code>. This will raise the error since intermediates in <code>model_a</code> will be freed in the first <code>backward</code> call.<br>
If you want to accumulate gradients in <code>model_a</code> you could use <code>retain_graph=True</code>. If not, you can <code>.detach()</code> <code>enc</code> before passing it to <code>model_b</code>.</p>

Bartosz_Brzoza: <p>Thanks Peter for your reply.<br>
Using <code>retain_graph=True</code> on the full graph seemed unnecessary, but your reply from a different thread helped me resolve the issue: <a href="https://discuss.pytorch.org/t/how-to-free-graph-manually-after-using-retain-graph-true/66258/2">https://discuss.pytorch.org/t/how-to-free-graph-manually-after-using-retain-graph-true/66258/2</a></p>
Torch::from_blob() has different value from its *data array

Kiki_Rizki_Arpiandi: <p>Please help, I try to create tensor from array using <code>torch::from_blob</code> but the tensor value is different than the array, here is my code:</p>
<pre><code class="lang-auto"> double cost_matrix_value[4][4] = {{9, 2, 7, 8},
                                      {6, 4, 3, 7},
                                      {6, 8, 1, 8},
                                      {7, 6, 9, 4}};
    auto cost_matrix = torch::from_blob(cost_matrix_value, {4, 4});
at::print(cost_matrix);
</code></pre>
<p>The result is</p>
<blockquote>
<p>0.0000  2.5312  0.0000  2.0000<br>
0.0000  2.4375  0.0000  2.5000<br>
0.0000  2.3750  0.0000  2.2500<br>
0.0000  2.1250  0.0000  2.4375<br>
[ Variable[CPUFloatType]{4,4} ]</p>
</blockquote>

hedinghong: <p>have you fixed this ?</p>
Is it possible to get two sorts of gradients in a single pass? If not what is the best way to go about it?

rao: <p>I want to do the following:</p>
<p>Say, I have neural network NN and inputs x.</p>
<ol>
<li>To train the network I will be passing gradients from a loss term:</li>
</ol>
<pre><code class="lang-auto">y = NN(x)
loss = (y-y_actual)**2
</code></pre>
<ol start="2">
<li>I also want to evaluate dy/dx, each pass, along with the usual d(loss)/d(weights of NN):</li>
</ol>
<pre><code class="lang-auto">y.backward() #Stores dy/dx in x (input)
loss.grad()  #For gradient descent
</code></pre>
<p>Is it possible in a single run? and if not what is the best way to evaluate each set of gradient optimally without needlessly generating dy/d(weights) or d(loss)/d(dx)</p>
Do i need to learn RNN and LSTMs prior to learn Transformer Model?

Teddy_Salas: <p>Do i need to learn RNN and LSTMs prior to learn Transformer Model?</p>

vdw: <p>Well, strictly speaking, an LSTM is special instance of an RNN. So getting a basic grasp about the ideas behind and inner workings of RNNs would be enough.</p>
<p>In my opinion, it’s probably not really needed to learn about RNNs, but I would still recommended it – again, at least the basic concepts. Firstly, the limitations of RNNs – mainly the problem long-term dependencies and the lack of parallelism – are one of the goals to address in the original <a href="https://arxiv.org/abs/1706.03762" rel="noopener nofollow ugc">Transformer paper</a>. And secondly, the concept of attention, as least as far as I know, has first been proposed (or at least popularized) by RNNs in <a href="https://arxiv.org/abs/1409.0473" rel="noopener nofollow ugc">this paper</a>. The Transformer paper generalized the idea of attention, but I think the concept is a more intuitive in the context of RNNs.</p>
<p>If you just want to train a Transformer model, just go for it. For some deeper understanding, it’s always good to learn about (closely) related ideas to put things into better context.</p>

Teddy_Salas: <p><a class="mention" href="/u/vdw">@vdw</a> Thank you for your suggestion</p>
Pytorch dynamic quantization issue

kysandeep176: <p>Hi - I am writing a script to quantize my .pth model ( universal image segmentation model) with dynamic quantization technique referred below.</p>
<p><a href="https://pytorch.org/docs/stable/quantization.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/quantization.html</a></p>
<p>my .pth file is built on mask2former architecture which has multiheadattention module in it. Here attached screenshot where this module is not supported for dynamic quantization. Can anyone confirm on this please!!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3ad56db2fe50954c098e0aeba3cd9fae6f69e95c.png" data-download-href="https://discuss.pytorch.org/uploads/default/3ad56db2fe50954c098e0aeba3cd9fae6f69e95c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3ad56db2fe50954c098e0aeba3cd9fae6f69e95c.png" alt="image" data-base62-sha1="8osXD1yxFEjLhxxmoo4xPTNaT5G" width="687" height="500" data-dominant-color="F6F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">701×510 19.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

kysandeep176: <p>Can anyone confirm on this please?</p>
How to reuse shm tensor in dataloader

PlusLight: <p>I use torch’s dataloader with <code>num_workers &gt; 0</code>. A single batch for my problem is very large (~100GB) and consumes a lot of CPU memory. These batches have the same size.</p>
<p>From my understanding, if I return my batch in my dataset’s <code>__getitem__()</code>, it will first be sent to shm by the loader worker process, then the handle is sent to the main process. However there are two problems:</p>
<ol>
<li>During sending the worker process local tensor to shm, the high watermark memory consumption is double the tensor size;</li>
<li>The worker has to allocate for each batch a large chunk of contiugous memory, then allocate another chunk in shm, then free the process local memory chunk, only to reallocate the same size for the next batch. And the shm memory chunk is eventually freed by the main process, and needs to be allocated by the worker process again.</li>
</ol>
<p>So my problem is, is there a way to reuse the shm tensor so that I can:</p>
<ol>
<li>populate directly the shm tensor in the worker process instead of allocating a worker process local tensor first;</li>
<li>reuse the shm tensor between worker process and main process instead of allocating and freeing it repeatedly.</li>
</ol>
<p>Thanks.</p>
Issues in nightly pytorch builds. Where to report them?

apivovarov: <p>Where can I post issues found in recent nightly pytorch builds (linux/cuda118)?<br>
Is there special github template / tag for such issues?<br>
Special discuss Category for such issues?<br>
Maybe special mail list?</p>

marksaroufim: <p>Github issue is probably a good place in general since the release team will triage those</p>

apivovarov: <p>Opened the issue - <a href="https://github.com/pytorch/pytorch/issues/107503" class="inline-onebox" rel="noopener nofollow ugc">[nightly][jit] bad constant exponent (e+38.f) in default_program fused_mul_div_add · Issue #107503 · pytorch/pytorch · GitHub</a></p>
Computer Vision sample code in PyTorch docs has super low accuracy

seberino: <p>I’ve been studying this nice sample PyTorch Computer Vision code here:</p>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>
<p>The accuracy it gives is only 54%!?  I’ve tried adding more than<br>
2 epochs but still haven’t figured out how to get it at a decent level.</p>
<p>Any ideas? It probably would be more confidence building for<br>
newcomers if their first PyTorch code gave a stellar accuracy! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Sincerely,</p>
<p>Chris</p>
Quantize Per-Trained model Using QLoRa or LoRa , PFET Technique

Youness_EL_BRAG: <p>i would like to ask how can I use QLoRa or  Parameter-Efficient Fine-Tuning  thin a model does not register at Hugging face instead is Based on <a href="https://github.com/OFA-Sys/OFA" rel="noopener nofollow ugc">OFA</a></p>
<p>Here the repo of the model: <a href="https://github.com/taokz/BiomedGPT/tree/main" class="inline-onebox" rel="noopener nofollow ugc">GitHub - taokz/BiomedGPT: BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks</a></p>
<p>i am trying to Quantize the Tiny version but I don’t know if I need to use Lora in which way to Parameter-Efficient Fine-Tuning</p>
Copy weights from pretrained ResNet-50 model to pruned model with different channels

King4819: <p>Hey guys, I want to ask if there’s a method to copy weights from pretrained ResNet-50 model to pruned model.</p>
<p>Now I have the prune cfg, which is the number of remain channels in each layer</p>
<p>e.g. pruned_cfg = [18, 23, 24, 25, 20, 30, 46, 46, 30, 39, 49, 71, 52, 59, 130,<br>
109, 62, 84, 73, 85, 86, 92, 112, 110, 105, 106, 24, 17, 41, 37, 37, 46]</p>
<p>And I have masks, which stores which channel are preserved in original models(“1”) and which is being pruned(“0”)</p>
<p>e.g. masks[0]=tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,<br>
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1.,<br>
0., 0., 1., 1., 1., 1., 1., 1., 0., 1., 0., 0., 0., 1., 1., 0., 0., 0.,<br>
1., 1., 1., 1., 1., 0., 1., 1., 1., 0.])</p>
<pre><code>    torch.sum(masks[0])=18 (remaining channels)
</code></pre>
<p>Now I want to create a new ResNet-50 model with pruned_cfg, and copy the corresponding channel weights from the original model (copy all 18 corresponding channel weights from the original model in layer1, …)</p>
<p>Is there an easy approach?<br>
Thanks very much!</p>

ptrblck: <p>I don’t know if there is an easy way to perform your desired loading and you would most likely need to load each weight separately:</p>
<pre><code class="lang-python">model = models.resnet18()

mask = torch.tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1.,
0., 0., 1., 1., 1., 1., 1., 1., 0., 1., 0., 0., 0., 1., 1., 0., 0., 0.,
1., 1., 1., 1., 1., 0., 1., 1., 1., 0.]).bool()

sd = models.resnet18(pretrained=True).state_dict()

with torch.no_grad():
    model.conv1 = nn.Conv2d(3, int(mask.sum().item()), 7, 2, 3, bias=False)
    model.conv1.weight.copy_(sd["conv1.weight"][mask])
</code></pre>
<p>Depending on the config and how you’ve created it, you might be able to iterate the <code>state_dict</code> keys etc., which could make the loading more convenient.</p>

King4819: <p>Sorry, I didn’t see your reply until now, thanks for you answer</p>
RuntimeError: reshape is not implemented for sparse tensors

jloverich: <p>Is there an efficient way to reshape a sparse tensor?  I’m using pytorch 1.7 and am trying to use a sparse tensor where I’ve been using a dense tensor (which is extremely sparse).  The code fails when it hits reshape as there is no implementation for a sparse tensor.  Are there any plans to implement this?</p>

shenshanf: <pre><code class="lang-auto">def sparse_reshape(sparse_tensor, size):
    mask = torch.sparse_coo_tensor(indices=sparse_tensor.indices(),
                                   values=torch.ones(sparse_tensor._nnz(),
                                                     device=sparse_tensor.device,
                                                     dtype=torch.bool),
                                   size=sparse_tensor.size())
    mask = mask.to_dense().reshape(size)

    sparse_tensor = torch.sparse_coo_tensor(indices=mask.to_sparse().indices(),
                                            values=sparse_tensor.values(),
                                            size=size)
    return sparse_tensor.coalesce()
    pass
</code></pre>
GPU becomes slower and the GPU utilization drops

Junseok: <p>I am using torch13.0+cu117 and have 4 A6000 GPUs.</p>
<p>When I tried to run a same code on several GPUs with different configs at the same time, the GPU utilization of each node drops (sometimes 0% and the training is paused for several minutes) and the it takes 2 to 4 times longer than using a single node. (See the node number 3, where it uses GPU memory but GPU is not working)<br>
Note that I am not using multiprocessing, just running the same code on different nodes for different experiments.<br>
I checked the CPU usage using “htop” command but it seems that it does not have too much burden.<br>
I don’t know how can I check where is the problem.<br>
Could you help me what can I try to fix this problem?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53f0b7ab7b171082dade9214a975e7b690e00eef.png" data-download-href="https://discuss.pytorch.org/uploads/default/53f0b7ab7b171082dade9214a975e7b690e00eef" title="스크린샷 2023-08-18 오후 4.24.43"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/53f0b7ab7b171082dade9214a975e7b690e00eef_2_690x102.png" alt="스크린샷 2023-08-18 오후 4.24.43" data-base62-sha1="bYzmCoBO7VTYJ36w2IIzlyZuCNp" width="690" height="102" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/53f0b7ab7b171082dade9214a975e7b690e00eef_2_690x102.png, https://discuss.pytorch.org/uploads/default/original/3X/5/3/53f0b7ab7b171082dade9214a975e7b690e00eef.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/3/53f0b7ab7b171082dade9214a975e7b690e00eef.png 2x" data-dominant-color="3E403D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2023-08-18 오후 4.24.43</span><span class="informations">1020×152 33.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m unsure how you are defining “node”, but it usually stands for a standalone server. However, in your case it seems you are talking about different GPUs in the same workstation?<br>
If so, you might run into a CPU bottleneck by launching multiple jobs. If the CPU is not fast enough to schedule work for all 4 GPUs, your GPUs will starve while the CPU tries to keep up with the scheduling.<br>
You should be able to see this effect in a visual profiler, e.g. Nsight Systems.</p>
Is torch.no_grad() making a difference in custom autograd functions?

CTZ: <p>Hi,<br>
I wondered if it makes a difference to specify torch.no_grad() inside a custom autograd class for the forward and backward functions.<br>
Thanks!</p>

ptrblck: <p>No, it shouldn’t make a difference since gradient computation is already disabled in custom <code>autograd.Function</code>s by default:</p>
<pre><code class="lang-python">class MyFun(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        print(torch.is_grad_enabled())
        ctx.save_for_backward(input)
        return input * 2
    
    @staticmethod
    def backward(ctx, grad_output):
        print(torch.is_grad_enabled())
        input, = ctx.saved_tensors
        return grad_output * input
    
f = MyFun().apply
x = torch.randn(1, 1, requires_grad=True)
out = f(x)
# False
out.mean().backward()
# False
</code></pre>
Formlist migration to newer version

Anil_Saklani: <p>I’m trying to update the legacy Python code below but I can’t seem to find the necessary method.</p>
<p>expansion_factor = 100<br>
for j in range(expansion_factor):<br>
for i in range(train_df.shape[0]):<br>
try:<br>
ex = data.Example.fromlist([train_df.qs[i], train_df.sol[i]], fields)<br>
train_example.append(ex)<br>
except:<br>
pass</p>
<p>for i in range(val_df.shape[0]):<br>
try:<br>
ex = data.Example.fromlist([val_df.qs[i], val_df.sol[i]], fields)<br>
val_example.append(ex)<br>
except:<br>
pass</p>
<hr>
<p>where  fields = [(‘Input’, Input),(‘Output’, Output)]<br>
and Input and Output are tokenized values</p>

ptrblck: <p>Your code snippet doesn’t seem to use any PyTorch-specific methods and it looks instead like <code>pandas</code> code. It’s also unclear what exactly doesn’t work.</p>

Anil_Saklani: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>  thank you!</p>
<p>Please see the more details below<br>
I am importing-</p>
<p>from torchtext.legacy.data import Field, BucketIterator, Iterator<br>
from torchtext.legacy import data</p>
<p>It is using torchtext.legacy for creating the formlist but I dont see similar package in latest version of pytorch , can you please suggest how I can replace <strong>data.Example.fromlist</strong> ,</p>
<p>I have tried pands datafame like (pd.DataFrame(list(zip(qs[i])  with list but it is not matching what earlier version of pytorch was doing</p>
<p>example<br>
qs[0] - what is the sum of 2+2<br>
sol[0]-  4<br>
I want to create a list where first element will be question and second element in the list is solution<br>
qs                                      sol<br>
what is the sum of 2+2          4</p>
<p>but if I use panda list it is creating char list<br>
w<br>
h<br>
a<br>
t</p>
Cannot import torch.func in M1 Pro Mac

ganesh1102: <p>I am trying to import <code>vmap</code> and <code>jacrev</code> from <code>torch.func</code> in my MacBook M1 Pro. Here’s what I am doing:</p>
<pre><code class="lang-auto">from torch.func import vmap, jacrev
</code></pre>
<p>Here’s the error I am getting:</p>
<pre><code class="lang-auto">ModuleNotFoundError: No module named 'torch.func'
</code></pre>
<p>I am using a conda environment with the latest version of PyTorch and Python 3.8.</p>
<p>How can I resolve this?</p>

ptrblck: <p>The <code>import</code> works for me using <code>2.1.0.dev20230817+cu121</code>.</p>

ganesh1102: <p>The import did not work for me in a Python or conda <code>virtualenv</code> in the VS code terminal, but works just fine in my MacOS terminal - strange. Thanks for your help.</p>

ptrblck: <p>I guess you are using different environments in VS Code vs. your terminal, where one might contain a PyTorch installation which is too old.</p>
Customize pytorch model export to ONNX

angelika: <p>I am trying to export pretrained Mask R-CNN model to ONNX format. Since this model in basic configuration has following structure (here I added <code>batch_size</code> as dynamic axes):</p>
<p><a href="https://i.stack.imgur.com/LIsSY.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/LIsSY.png" alt="enter image description here" width="392" height="500"></a></p>
<p>I want to customize my model and add <code>batch_size</code> to output (it means I need to add new dim to each of the outputs).</p>
<p>I wrote following code to make it possible:</p>
<pre><code>class MaskRCNNModel(torch.nn.Module):
  def __init__(self):
    super(MaskRCNNModel, self).__init__()
    self.model = torchvision.models.detection.maskrcnn_resnet50_fpn(weights='DEFAULT')
    in_features = self.model.roi_heads.box_predictor.cls_score.in_features
    self.model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes=7)
    self.model.load_state_dict(torch.load("saved_dict.torch"))

  def forward(self, input):
    outputs = self.model.forward(input)
    boxes = []
    labels = []
    scores = []
    masks = []
    for result in outputs:
        box, label, score, mask = result.values()
        boxes.append(box)
        labels.append(label)
        scores.append(score)
        masks.append(mask)
    
    return boxes, labels, scores, masks

maskrcnn_model = MaskRCNNModel()
maskrcnn_model.eval()
maskrcnn_model.to(device)

x = torch.rand(1, 3, 512, 512)
x = x.to(device)

maskrcnn_model(x)

torch.onnx.export(maskrcnn_model,
                  x,
                  "base_model_100_epochs.onnx",
                  opset_version=11,
                  input_names=["input"],
                  output_names=["boxes", "labels", "scores", "masks"])
</code></pre>
<p>but the code above doesn’t change any export parameters. The structure of output stays the same:</p>
<p><a href="https://i.stack.imgur.com/kpMfK.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/kpMfK.png" alt="enter image description here" width="413" height="500"></a></p>
<p>What should I do to customize <code>forward</code> method to be able to add <code>batch_size</code> into ONNX model output?</p>
Adding new nn.Parameter in Conv2d (that gets saved in state_dict)

epistoteles: <p>Conv2d contains two parameters that get saved in the state_dict: weight and bias. I want to add a new parameter of the same shape as bias (purpose is irrelevant here) that also gets saved in the state_dict. Currently, I am saving the parameter in the module itself in a nn.ParameterList of the same length as the number of Conv2d layers I have. However, I would prefer to have the parameter stored inside the Conv2d layer it belongs to.</p>
<p>How and where do I have to modify Conv2d to add my custom parameter?</p>

ptrblck: <p>You can directly assign new parameters to an <code>nn.Module</code>:</p>
<pre><code class="lang-python">conv = nn.Conv2d(3, 3, 3)
print(dict(conv.named_parameters()))

conv.my_param = nn.Parameter(torch.randn(1))
print(dict(conv.named_parameters()))
</code></pre>
<p>which will then show up in <code>.parameters()</code> as well as the <code>state_dict</code>.<br>
Note however, that <code>my_param</code> won’t be used as the <code>forward</code> method wasn’t overridden.</p>

epistoteles: <p>Thanks, this worked perfectly! I didn’t expect it to be this easy.</p>
Try to understand the VOC2012 Dataset for semantic segmentation using Pytorch

PytorchLearning: <p>Hi all,</p>
<p>I want to use PASCAL VOC 2012 dataset for semantic segmentation using Pytorch.</p>
<p>First, I’d like to understand this dataset. After that I want to build from scratch a model in pytorch for sementic segmentation to learn more.</p>
<p>I have just downloaded the VOC2012 Dataset, and in the folder JPEGImages in VOCdevkit/VOC2012/JPEGImages I have 17125 images.</p>
<p>But when I apply the code :</p>
<pre><code class="lang-auto">import os

def make_datapath_list(voc2012_path):
    img_dir = os.path.join(voc2012_path, "JPEGImages")
    anno_dir = os.path.join(voc2012_path, "Annotations")

    subsets = ["train", "val"]
    train_img_list, train_anno_list, val_img_list, val_anno_list = [], [], [], []

    for subset in subsets:
        subset_samples_path = os.path.join(voc2012_path, f"ImageSets/Main/{subset}.txt")
        with open(subset_samples_path, "r") as f:
            subset_samples = f.read().splitlines()

        for sample_name in subset_samples:
            img_path = os.path.join(img_dir, f"{sample_name}.jpg")
            anno_path = os.path.join(anno_dir, f"{sample_name}.xml")

            if subset == "train":
                train_img_list.append(img_path)
                train_anno_list.append(anno_path)
            elif subset == "val":
                val_img_list.append(img_path)
                val_anno_list.append(anno_path)

    return train_img_list, train_anno_list, val_img_list, val_anno_list

voc2012_path = "./VOCdevkit/VOC2012"
train_img_list, train_anno_list, val_img_list, val_anno_list = make_datapath_list(voc2012_path)
print("trainlist: ", len(train_img_list))
print("vallist: ", len(val_img_list))
print("train_anno_list: ", len(train_anno_list))
print("val_anno_list: ", len(val_anno_list))
</code></pre>
<p>I get as result</p>
<p>trainlist:  5717<br>
vallist:  5823<br>
train_anno_list:  5717<br>
val_anno_list:  5823</p>
<p>Why I get the sum 5717 + 5823 = 11540 instead of 17125</p>
<p>Why I didn’t get 17125?</p>
<p>Did I make a mistake or did I misunderstand the dataset ?</p>
<p>Thanks for your help.</p>

PytorchLearning: <p>Hi All,</p>
<p>For those who are interested, I found the solution to my question on page 6 of this website.</p>
<p><a href="http://host.robots.ox.ac.uk/pascal/VOC/voc2012/devkit_doc.pdf" class="onebox" target="_blank" rel="noopener nofollow ugc">http://host.robots.ox.ac.uk/pascal/VOC/voc2012/devkit_doc.pdf</a></p>
Updating MaskRCNN loss function for class imbalance?

johnny69: <p>I’m creating an instance segmentation model with MaskRCNN. There are four classes A, B, C, and D. The problem is, there are only about 100 samples each of A and B, but 1000 samples each of C and D. This leads to a significant class imbalance.</p>
<p>Given the nature of the data (medical stuff) I cannot easily gather more data. I’d like to modify the loss function to address the class imbalance instead, but I’m not sure how to do this.</p>
<p>This is the loss function that my MaskRCNN model uses (I’m just using the <a href="https://pytorch.org/vision/main/models/mask_rcnn.html" rel="noopener nofollow ugc">pre-trained model created by PyTorch</a>):</p>
<pre><code class="lang-auto">def maskrcnn_loss(mask_logits, proposals, gt_masks, gt_labels, mask_matched_idxs): 
     # type: (Tensor, List[Tensor], List[Tensor], List[Tensor], List[Tensor]) 
     """ 
     Arguments: 
         proposals (list[BoxList]) 
         mask_logits (Tensor) 
         targets (list[BoxList]) 
  
     Return: 
         mask_loss (Tensor): scalar tensor containing the loss 
     """ 
  
     discretization_size = mask_logits.shape[-1] 
     labels = [l[idxs] for l, idxs in zip(gt_labels, mask_matched_idxs)] 
     mask_targets = [ 
         project_masks_on_boxes(m, p, i, discretization_size) 
         for m, p, i in zip(gt_masks, proposals, mask_matched_idxs) 
     ] 
  
     labels = torch.cat(labels, dim=0) 
     mask_targets = torch.cat(mask_targets, dim=0) 
  
     # torch.mean (in binary_cross_entropy_with_logits) doesn't 
     # accept empty tensors, so handle it separately 
     if mask_targets.numel() == 0: 
         return mask_logits.sum() * 0 
  
     mask_loss = F.binary_cross_entropy_with_logits( 
         mask_logits[torch.arange(labels.shape[0], device=labels.device), labels], mask_targets 
     ) 
     return mask_loss 
</code></pre>
<p>My question is: how can I modify this function to consider class imbalance and potentially add a weight for each class? (I didn’t write this code and it doesn’t make much sense to me). Any advice would be appreciated!</p>
How can I load a PyTorch (.pth.tar) model with simultaneous parallelization across my GPUs

molokanov: <p>I have a clear understanding how to load a PyTorch model from a <code>.pth.tar</code> file to a single prescribed GPU, but in my situation, the entire model doesn’t fit on a single GPU. So I need to load a pipeline-parallelized model (from a <code>.pth.tar</code> file) to several GPUs at once.<br>
Is it possible in PyTorch without involvement of additional software? If yes, how to implement this?<br>
Looking for a response.<br>
Thanks.</p>
Torch DDP: Memory leak when loading datasets to memory

Christopher_Hendra: <p>I have a huge numpy array of size 2.5 million x 5200 that I can load into memory for training. I am trying to train a model with 4 GPU with torch DDP but I realize that loading this naively will result in 4 duplicates of such an array being loaded into the memory. I tried to split the array into 4 equally sized smaller arrays and assigned each process to load the smaller array based on their rank number but the memory consumption rose up so quickly, way more than what would happen if I were to load the original array once and I had to terminate the training script before my machine ran out of memory.</p>
<p>I suspected that internally the data is being duplicated across the different processes. Does anybody know if there is any workaround to this?</p>
<p>Thank you!</p>

rvarm1: <p>Hi, if you have indeed assigned each process to only load a subset of the data, then memory consumption should definitely be less than having each process load the entire dataset into CPU memory.</p>
<p>Are you using PyTorch APIs such as DataSet / DataPipes / DistributedSampler in order to do this? A code snippet with the higher than expected memory usage would be valuable.</p>

Christopher_Hendra: <p>Hi <a class="mention" href="/u/rvarm1">@rvarm1</a></p>
<p>Thanks for your reply. I am using TensorDataset but did not use distributed sampler since each process should see a different subset of the data. Each file here is named after the process that should access them, i.e chunks_1.npy will be loaded by the process with rank 1 (utils.get_rank() will return the rank of the process)</p>
<pre><code>train_signals = torch.Tensor(np.load(os.path.join(root_dir, "chunks_{}.npy"\
       .format(utils.get_rank()))))
train_seqs = torch.LongTensor(np.load(os.path.join(root_dir, 
                                      "references_{}.npy"\
       .format(utils.get_rank()))).astype('int64'))
train_lengths = torch.LongTensor(np.load(os.path.join(root_dir, 
                                         "reference_lengths_{}.npy"\
       .format(utils.get_rank()))).astype('int64'))
    
train_ds = TensorDataset(train_signals, train_seqs, train_lengths)
train_dl = DataLoader(train_ds, batch_size=batch_size_per_gpu, 
                      num_workers=args.num_workers, pin_memory=True,
                      worker_init_fn=random_fn)
</code></pre>
<p>I have also checked that each chunks_{}.npy.format(utils.get_rank()) is the correct partitions of the original, larger array</p>
<p>Thanks!</p>

LBlaabjerg: <p>Did anyone figure out a solution here? Thanks!</p>
Runtime segmentation fault at torch unloading ( llvm::cl::Options() )

jlquetin: <p>Hi there !</p>
<p>I am facing an issue when running my app built from C++ code. It is basically an inference sample to detect objects in an image. I use libtorch to do load my model and do the inference.<br>
The error arises only at the end of the program, with a Segmentation Fault that is thrown after the destruction of all my objects.<br>
The debugger points to a llvm destructor, as described below :</p>
<p><code>Invalid free() / delete / delete[] / realloc() ==2964293==    at 0x483CA3F: free (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==2964293==    by 0xAA67B3E: llvm::cl::Option::~Option() (in /home/jlquetin/Documents/tools/libtorch/lib/libtorch_cpu.so) ==2964293==    by 0x1E9C8FDD: __cxa_finalize (cxa_finalize.c:83) ==2964293==    by 0x66F23B2: ??? (in /home/jlquetin/Documents/tools/libtorch/lib/libtorch_cpu.so) ==2964293==    by 0x4011F6A: _dl_fini (dl-fini.c:138) ==2964293==    by 0x1E9C88A6: __run_exit_handlers (exit.c:108) ==2964293==    by 0x1E9C8A5F: exit (exit.c:139) ==2964293==    by 0x1E9A6089: (below main) (libc-start.c:342) ==2964293==  Address 0x1 is not stack'd, malloc'd or (recently) free'd</code></p>
<p>Please note that the error doesn’t occur when I don’t use my specific UI (built internally) which uses OpenGL, GLE and glfw3. I have read some topics about library incompatibility but nothing clear to solve this issue. I am working on Ubuntu 20.04 and building my project with gcc 9.4.0 and libtorch 1.13.1.<br>
Does anyone have a clue about the problem here ?</p>