akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Engine.py error

Lane: <p>This error shows me in<code> engine.py</code> in function <code>train_one_epoch()</code>, but I think that I don’t need to change this script. Because of function <code>_get_image_size()</code> in <code>transforms.py</code> I was downgrade torchvision to 0.10.0 and torch = 1.10.0 but now this error shows me and I don’t know if the problem is because of versions or something else ?</p>
<p>I followed this tutorial <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#torchvision-object-detection-finetuning-tutorial" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a> and I’m wondering which versions are used here ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54f723aba04779fce9c8371e87740ab112e1a039.png" data-download-href="https://discuss.pytorch.org/uploads/default/54f723aba04779fce9c8371e87740ab112e1a039" title="molim"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54f723aba04779fce9c8371e87740ab112e1a039_2_690x388.png" alt="molim" data-base62-sha1="c7DBgEuG76hhnPIFAvLMAuL2SpP" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54f723aba04779fce9c8371e87740ab112e1a039_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54f723aba04779fce9c8371e87740ab112e1a039_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54f723aba04779fce9c8371e87740ab112e1a039_2_1380x776.png 2x" data-dominant-color="2A2C2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">molim</span><span class="informations">1920×1080 405 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/landeka/Nezz/Model1.py", line 271, in &lt;module&gt;
    main()
  File "/home/landeka/Nezz/Model1.py", line 262, in main
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
  File "/home/landeka/Nezz/vision/references/detection/engine.py", line 28, in train_one_epoch
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
  File "/home/landeka/Nezz/vision/references/detection/engine.py", line 28, in &lt;listcomp&gt;
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
  File "/home/landeka/Nezz/vision/references/detection/engine.py", line 28, in &lt;dictcomp&gt;
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
AttributeError: 'int' object has no attribute 'to'
</code></pre>
Autocast from DataParallel module

tsp_t: <p>I’m trying to reduce memory usage by using <code>autocast</code> at method <code>forward</code> only for some modules inside the model.</p>
<p>However, in a situation where the model is Dataparalleled, doesn’t performance decrease occur even if I do not specify that autocast is performed on a specific device like <code>torch.autocast("cuda", torch.float16)</code> ?</p>
<p>I want to know if I can get better (memory or speed) performance when using something like <code>torch.autocast(input.device, torch.float16)</code></p>

ptrblck: <p>No, the <code>device_type</code> argument in <code>autocast</code> doesn’t care about the actual device with its ID (so <code>cuda:0</code>, <code>cuda:1</code> etc.) but only about the type (<code>cuda</code> vs. <code>cpu</code>).</p>
Why is the .to method faster than passing device=cuda:0

yoterel: <p>Hello,<br>
recently I profiled some piece of code training a NN using torch, and discovered the .to method takes a significant amount of time out of the running time.</p>
<p>when digging in a bit further, I indeed allocate tensors in the forward pass in this manner:</p>
<p><code>torch.randn((x, y)).to("cuda:0")</code></p>
<p>This didn’t seem like a problem initially, however this probably constructs the tensor on CPU and then passes it to GPU, and creates a bottleneck. I tried bench-marking what would happen if I do:</p>
<p><code>torch.randn((x, y), device="cuda:0")</code></p>
<p>But unexpectedly, it performs worse? I created this in a small example which also shows the same behavior:</p>
<pre><code class="lang-auto">import torch
import time

cur_time = time.time()
for i in range(5000):
    torch.randn(5, device="cuda:0")
print("device=device took: {}".format(time.time() - cur_time))
cur_time = time.time()
for i in range(5000):
    torch.randn(5).to("cuda:0")
print(".to took: {}".format(time.time() - cur_time))
</code></pre>
<p>Why is this the case?<br>
How can I speed up the creation of tensors by allocating them directly on GPU?</p>
<p>the example reproduces for different functions such as torch.zeros and torch.ones. I am using windows 10 with python 3.9.12, and torch 1.12.1 (GPU=NVIDIA GeForce RTX 2060).</p>

ptrblck: <p>CUDA operations are executed asynchronously so you would need to synchronize the code before starting and stopping the timers via <code>torch.cuda.synchronize()</code>. Adding it gives:</p>
<pre><code class="lang-python">device=device took: 0.023486614227294922
.to took: 0.0527348518371582
</code></pre>
<p>on my setup.</p>
Sampling word from a probability distribution

Zephyrus: <p>I’m trying to build a probabilistic model using PyTorch. I want to sample a batch of words from a categorical distribution (like the corresponding steps in LDA). But In my model, the multinomial distribution that is used to generate the topic assignment for each word has 2 batch dimensions, the shape of which is (I, D, K), I and D are batch dimensions and K is the topic-related dimension. Each element on the topic-related dimension represents the count of the corresponding topic. Currently I sample document-word using a loop that iterates through 3 dimensions, but it’s very slow. How can I speed up this step by vectorizing or whatever?</p>
<p>Here is my <a href="https://github.com/hjnnjh/Dynamic-User-Interests/blob/master/code/simulation.py" rel="noopener nofollow ugc">code</a>. You can run it directly.</p>
Need to recognize digits

Alexander_Alexandrov: <p>I want to recognize digits on images, but no luck for now. Although images seem to be quite recognizable NN does not seem to do any better than random guess.<br>
Notebook:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1zaqQ3tFG7zv4-dv88PM1utVmLCQsF3rZ?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220912-060049-RC01_473840151" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1zaqQ3tFG7zv4-dv88PM1utVmLCQsF3rZ?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1zaqQ3tFG7zv4-dv88PM1utVmLCQsF3rZ?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Images:<br>
<a href="https://drive.google.com/drive/folders/12nhWJo73gHUXWa0vg38cCOlz_aKHesKN?usp=sharing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://drive.google.com/drive/folders/12nhWJo73gHUXWa0vg38cCOlz_aKHesKN?usp=sharing</a></p>
<p>Any help is appreciated.<br>
Thank for your time!</p>

ptrblck: <p>Your model uses:</p>
<pre><code class="lang-python">nn.Sigmoid(),
nn.Softmax()
</code></pre>
<p>as the last layers which is wrong.<br>
<code>nn.CrossEntropyLoss</code> expects raw logits so remove both layers and rerun your script.</p>
Multithreading on a single GPU using Single Model for Inferencing

Salahuddin: <p>Hello,</p>
<p>I want to load the <code>PyTorch</code> model on a single GPU and perform multiple inferences through multi-threading on a single CUDA device. Is it possible, and if so, could you guide me on how to do so? I want to return inference results as well. The reason is I have noticed that not complete GPU power is utilized in single inference, so I thought through multithreading I would be able to 100% utilize my CUDA device</p>
How can i totally uninstall pytorch ? And why the output of torch.__file__ is None

bulbyang: <p>I  want to uninstall pytorch by using</p>
<pre><code class="lang-auto">conda uninstall pytorch
</code></pre>
<p>but after I has done this ,torch can be imort correctlly.<br>
I use torch.<strong>file</strong> to see where is the torch,however the outcome is ‘None’.</p>
<p>just as<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4ecc7906770f2d8f445779dbc6710c851b35253c.png" alt="屏幕截图 2022-09-13 234056" data-base62-sha1="bf5k3JuVkC0yYZmqfeukxEflEW8" width="358" height="100"></p>

ptrblck: <p>Is your <code>wdir</code> set to the pytorch source directory or are you in a folder containing a <code>torch.py</code> file?<br>
Also, does <code>pip list</code> show any PyTorch installs?</p>

bulbyang: <p>Thank you very much.<br>
There aren’t any torch.py files in the folder.When i try <code>pip list</code> ,it indicates that no torch or pytorch were installed in my computer.I will try to find anthor ways.<br>
Thanks again.</p>

ptrblck: <p>That’s really strange. What does <code>print(dir(torch))</code> return after importing it?</p>
Pytorch model sometimes trains and sometimes does not learn

Manish_Shukla: <p>hi, i have a simple feed forward NN for doing forecasting on tabular time series data. i transformed the data and feed it through the network in following way:</p>
<ol>
<li>if the tabular data has n fields. and i want to use t times to predict T future, i transform the t rows to t * n features in first layer. than i ahve 10 cells in next layer and have T cells in next layer.</li>
</ol>
<p>this simple model works very well sometimes and understands and fits the data really well sometimes. but most of the time it does not learn at all. it does not diverge but loss seem to get stuck and not reduce further. this does not happen always.</p>
<p>any ideas why it would not behave in-consistently. (mostly bad and sometimes really learns well)</p>
<p>thanks much,<br>
Manish</p>
Is there an advantage to DataLoader num_workers > 1 if the data is first loaded onto each CPU?

Sam_Lerman: <p>As the question asks, if I’m loading data directly onto CPU, and my <code>Dataset</code>’s <code>__getitem__</code> only indexes an initialized array — after having loaded it from disk in the <code>__init__</code> method — is there still an advantage to using more than one worker with <code>num_workers &gt; 1</code>?</p>

ptrblck: <p>I would claim it depends on your system, but would not expect to see a huge speedup if at all. Using too many workers could also slow down your data loading due to the simple indexing operation and the overhead from multiprocessing. In the end, you should profile different approaches for your system and use case, and check which config works best.</p>
Error training images for AI art generation (create a .ckpt file)

dhmiller: <p>I did not write the software that threw the error. I am trying to train some images for use as a model with AI generative art using an A6000 under Win 10 Pro. The developer of the software does not know how to handle this error. Hoping to get some info to pass along to him from this forum. Thanks for any help. The error I got while running the training function is:     RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 4096, 77]], which is output 0 of SliceBackward, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>

JuanFMontesinos: <p>You are probably slicing a tensor and modiying it overiding the data directly.<br>
It’s diff to help u without the source code.</p>
<p>Best<br>
J</p>

dhmiller: <p>Thanks much. Afraid I can’t get my hands on the code.</p>
PyTorch is joining the Linux Foundation

zetyquickly: <p>Hello,</p>
<p>There’s a lot of people talking about recent news about PyTorch in Twitter. But could anybody tell why it is good (for the framework) that PyTorch joins foundation and will or could change in short term and the long term?</p>
<p>What I can think about is more backends (GPUs, TPUs) will be supported. But was there any obstacles to do this thing under the Meta’s roof with all the support of NVidia, Microsoft, Quansight contributors?</p>
Can not move weights to GPU

acs: <p>Hello all,</p>
<p>This is my first question and I am a torch newbie and feel free to educate me if I am doing anything wrong with this question topic here.</p>
<p>I basically want to run my existing torch code on GPU.</p>
<p>I do following:</p>
<pre><code class="lang-auto">    if torch.cuda.is_available():
        dev = "cuda:0"
    else:
        dev = "cpu"
    device = torch.device(dev)

    model = model.to(device)

    model.train()

    net_input = net_input.to(device)

    net_output = model(net_input)

</code></pre>
<p>Yet I am getting the following error:</p>
<pre><code class="lang-auto">     return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same
</code></pre>
<p>I thought moving the model and the tensors to GPU device should be enough. What am I missing here? Where should I start debugging this issue?<br>
Any tips appreciated.</p>
<p>Thanks!</p>

acs: <p>Alright, I found out there were some vanilla <code>list()</code> usages instead of <code>nn.ModuleList()</code> which were causing certain weights to stay on CPU. I wish this was explicitly stated in <a href="https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Quickstart — PyTorch Tutorials 1.12.1+cu102 documentation</a> or some other parts of the documentation more explicitly.</p>
Oscillating in loss and accuracy during fine tuning

Zeyu_Yan: <p>Hey guys:</p>
<p>I noticed that in this tutorial:<br>
<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a><br>
The training loss and accuracy keeped oscillating. Is this a supposed behavior or we should expect the training loss to keep dropping and training accuracy to keep increasing after some time?</p>
<p>Please let me know what you guys think. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Thx a lot.</p>
What is the difference between calling torch.ao.quantize and torch.quantize

Rami_Ismael: <p>I am new to PyTorch Quantization some documentation use torch.ao.quantize, and another uses torch. quantization.</p>

jerryzh168: <p>please use <code>torch.ao.quantization.quantize</code> since <code>torch.quantization.quantize</code> is going to be deprecated</p>

Rami_Ismael: <p>What did pytorch quantization become deprecated? And, is the torch.ao is the replacement for PyTorch.quantization?</p>

jerryzh168: <p>yes, <code>torch.quantization</code> is going to be replaced by <code>torch.ao.quantization</code> migration is in progress, we’ll make an announcement when it is done</p>

Rami_Ismael: <p>What does ao mean in torch.ao ?</p>

jerryzh168: <p>it means architecture optimization, it will include quantization, sparsity and pruning and other ao techniques</p>
Dataset Wrapper Class for Parallel Reads of HDF5 via Multiprocessing

carsonmclean: <p>I am needing to manage a large amount of physiological waveform data, like ECGs, and so far have found HDF5 to be the best for compatibility with Python, PyTorch, Pandas, etc. The ability to slice/query/read only certain rows of a dataset is particularly appealing. However, I am struggling to develop a stable wrapper class which allows for simple yet reliable parallel reads from many <code>multiprocessing</code> workers, such as the case with PyTorch <code>dataset</code>/<code>dataloader</code>. I am fine with single threaded writes as I only have to ETL my source data into the HDF5 once, but lacking parallel reads really hurts run times during data analysis, PyTorch model runs, etc. I have tried both PyTables via Pandas and <code>h5py</code> with the following scenarios:</p>
<ul>
<li>One large HDF5 file
<ul>
<li>Could only get single process reads stable. <code>mp</code> pretty quickly corrupted the entire file.</li>
</ul>
</li>
<li>One HDF5 file per time series interval
<ul>
<li>ie, hundreds of millions of files. Still having collisions leading to corrupted files even when opening read only with a context manager. Also means backing up or transferring so many files is very slow due to the high IO cost.</li>
</ul>
</li>
</ul>
<p>Apologies for no code samples, but I’ve tried dozens of approaches over the last few months (each was hundreds of lines) and eventually always ran into stability issues. Is there any good example code of how to build a dataset wrapper class backed by HDF5? AFAIK, <code>h5py</code>'s MPI-based “Parallel HDF5” is not relevant since this is on one machine, and “Single Writer Multiple Reader” is not necessary as I am fine writing single-threaded before any reading. Ideally, I’d hope that whenever I need data in my project, I could just do similar to the following:</p>
<pre><code class="lang-auto">import dataset  # my HDF5 dataset wrapper class
import multiprocessing as mp

def dataloader(idxs):
  temp = []
  ds = dataset.Dataset()
  for _, idx in idxs.iterrows():
    df = ds.get_data(idx)
    # do some data wrangling in parallel
    temp.append(df)
  return temp

pool = mp.Pool(6)
data = pool.map(dataloader, indices)
</code></pre>
<p>Many thanks!</p>

Connor_Anderson: <p>I’m also running into the same problem. Has anyone ever gotten something like this to work? My experience is that a dataloader with multiple workers can simultaneously read from a large HDF5 file and process batches, but as soon as I move to a distributed data parallel setting the file gets corrupted or something, because I start getting errors saying that certain keys don’t exist even though they do.</p>

rkube: <p>I would also be interested in how people approached or even solved this problem.</p>
How to detect matches of data?

zenon: <p>Hello, I am quite new with AI. I would like to develop an AI that calculates the probability of matching 2 data records.</p>
<p>Example:<br>
I have 2 person profiles with different data, like name, first name, alias, date of birth, city of residence, email address, address, hobbies, etc . Not all records contain all data, but some details are of course more meaningful for a comparison than others.</p>
<p>Is there a possibility to design an algorithm that takes 2 of these records and then tells me what is the probability (or similarity) that it is the same person?</p>
<p>Can you guys give me a hint as to the best place to start?</p>
<p>Thank you all so much.</p>
The Inaccuracy of PyTorch Conv Calculation

ao_xu1: <p>After I run this code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

ta = torch.Tensor([[[[2., 4.], [0., 1.]]]])
up = nn.ConvTranspose2d(in_channels=1, out_channels=1, kernel_size=2, stride=1, padding=0)
with torch.no_grad():
    up.weight.data = torch.Tensor([[[[3., 1.], [1., 5.]]]])
    tb = up(ta)
print(tb)
</code></pre>
<p>I get the result:</p>
<blockquote>
<p>tensor([[[[ 5.8427, 13.8427,  3.8427],<br>
[ 1.8427, 16.8427, 20.8427],<br>
[-0.1573,  0.8427,  4.8427]]]], grad_fn=)</p>
</blockquote>
<p>Thematically, the answer should be:</p>
<blockquote>
<p>tensor([[[[ 6., 14.,  4.],<br>
[ 2., 17., 21.],<br>
[0.,  1.,  5.]]]], grad_fn=)</p>
</blockquote>
<p>Why the inaccuracy of PyTotorch Calculation will happen?</p>

KFrank: <p>Hi Ao!</p>
<aside class="quote no-group" data-username="ao_xu1" data-post="1" data-topic="161203" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ao_xu1/48/53145_2.png" class="avatar"> ao_xu1:</div>
<blockquote>
<pre><code class="lang-auto">up = nn.ConvTranspose2d(in_channels=1, out_channels=1, kernel_size=2, stride=1, padding=0)
</code></pre>
</blockquote>
</aside>
<p>You’re ignoring the <code>bias</code> term in <code>ConvTranspose2d</code>.  Try:</p>
<pre data-code-wrap="python"><code class="lang-python">up = nn.ConvTranspose2d (in_channels=1, out_channels=1, kernel_size=2, stride=1, padding=0, bias = False)
</code></pre>
<p>You will then obtain your expected result.  (Or you could set <code>up.bias</code> to a<br>
known quantity just as you do with <code>up.weight</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

ao_xu1: <p>I didn’t notice the bias of the Conv module before. Thanks a lot!</p>
MultiGPU training

exponential: <p>I have 3 GPUs (80GB each) on my virtual machine, however for some reason pytorch only uses one of the GPUs.<br>
Please see minimal reproduction of my code</p>
<pre><code class="lang-auto">import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"  
os.environ["CUDA_VISIBLE_DEVICES"]="0, 1, 2"
from torch import nn

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Network(**kwargs)
model = nn.DataParallel(model)
model.to(device)

trainloader = DataLoader(ImagesDataset('/ResizedImages', image_size), batch_size=batch_size,shuffle=True, num_workers=int(num_workers))

for i in range(EPOCHS):
        for img in trainloader:
            img_ = img.to(device)
            loss = model(img_)
            opt.zero_grad()
            loss.backward()
            opt.step() 
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/0573843ca9c9418716fe035e7f068cc63d2735d9.png" alt="Screenshot 2022-08-20 014333" data-base62-sha1="MdSAkSkGLnfoHz6yj20HfVq8NX" width="637" height="330"></p>
<p>When I run <code>nvidia-smi</code> after training, I get the output screenshot- meaning my training only uses 1 GPU. Is this due to the size of my network being minimal and thus training does not require that large memory (80GB x 3) or I am getting something wrong?</p>
<p>Many thanks!</p>

ptrblck: <p><code>nn.DataParallel</code> creates an imbalance on the GPUs and while your seem to be quite large, the other devices could still be used.<br>
You can add debug print statements to the <code>forward</code> method of the model and check the current <code>device</code> of the input tensor via <code>print(x.device)</code> to check which GPU the inputs use.<br>
Generally we recommend to use <code>DistributedDataParallel</code> for the best performance and to avoid the memory imbalance.</p>

exponential: <p>Hi ptrblck, thanks<br>
You were right about the imbalance created by <code>nn.DataParallel</code><br>
I later had to resort to using ddp.</p>
A error from FasterRCNN

MartinZhang: <p>Hi,</p>
<p>today,  I want save fasterrcnn with torch.jit.trace</p>
<p>like this:</p>
<pre><code class="lang-auto">import torch

import torchvision

model = torchvision.models.detection.fasterrcnn_mobilenet_v3_large_320_fpn(weights=FasterRCNN_MobileNet_V3_Large_320_FPN_Weights.DEFAULT)

data = (torch.rand(3, 300, 400), torch.rand(3, 500, 400))

torchscript_fasterrcnn = torch.jit.trace(model, data)

torch.jit.save(torchscript_fasterrcnn, 'model.pt')
</code></pre>
<p>But, there is a error about:</p>
<pre><code class="lang-auto">AttributeError: 'Tensor' object has no attribute 'items'
</code></pre>
<p>how to make it work? thanks.</p>
Efficient implementation of softmax/attention and resnet updates

tonyr: <p>I’m reasonably up-to-speed with NN architectures but I could do with some pointers to efficient implementations for softmax and resnet.</p>
<p>Let’s say I have a million vectors and I want to propagate these a thousand time steps into the future with attention selection and then backprop the lot.  At each step I compute a distance from each of a million keys then do a big softmax to get my attention weighting, a[t,i].   I now update my vectors with v[t+1,i] = (1 - a[t,i]) * v[t,i] + a[t,i] * d[t,i], and go on to do this another 999 times.</p>
<p>Now most a[t,i] will be zero, so all of this is really, really inefficient.    Let’s not worry about the computation of a[t,i], let’s assume that SVD or something similar can give me a shortlist and I’m happy pruning out those that are nearly zero.   Let’s say that this gives me ten non-zero a[t,i] that I care about, the rest I’m happy to treat as zero.</p>
<p>What I’m thinking is that I store the time step, n[t,i] in which the last non-zero update occured.  I can then write v[t+1,i] = (1 - a[t,i]) * v[n[t,i],i] + a[t,i] * d[t,i] and only do this for non-zero a[t,i].   But I don’t know if that’s the best way to do it in PyTorch, it’s certainly not very matrix freindly.</p>
<p>Softmax/attention is designed to be sparse, transformers are dominating our modelling, surely this is a problem that many people have had.   So, am I thinking along the right lines, and if so can someone point me to some papers/implementations that with speed my PyTorch coding?   Thanks!</p>
_model_URL import error

ArshadIram: <pre><code class="lang-auto">ImportError: cannot import name '_MODEL_URLS' from 'torchvision.models.mnasnet'
</code></pre>
<p>I am getting import error while import this:</p>
<pre><code class="lang-auto">from torchvision.models.mnasnet import _MODEL_URLS as URLs 
</code></pre>
<p>Any help would be highly appreciated.</p>

ptrblck: <p>The <code>_MODEL_URLS</code> <code>dict</code> was removed in <a href="https://github.com/pytorch/vision/pull/5618">this PR</a>. I don’t know what exactly you want to do with these URLs, but look at the PR to see how to use the introduced <code>weights</code> object to load the <code>state_dict</code>s.</p>

ArshadIram: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I updated the torch version, and this issue is resolved. However, I am getting an unusual error while downloading datasets on pycharm IDE.</p>
<pre><code class="lang-auto">urllib.error.URLError: &lt;urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:997)&gt;
</code></pre>
<p>my torch version is:</p>
<pre><code class="lang-auto">torch.__version__
'1.12.1'
</code></pre>

ArshadIram: <p>This issue is also fixed by importing these commands on download file:</p>
<pre><code class="lang-auto">import ssl
ssl._create_default_https_context = ssl._create_unverified_context
</code></pre>
Sudden Decrease in Training Accuracy

Kapil_Rana: <p>I am using logsoftmax and NLLLoss. for my classification problem (50 classes)<br>
My training accuracy decreased after 88 epoch.<br>
In 88 epoch: Accuracy is 51 %<br>
Maximum in 84 epoch : 71 %</p>
<p>Suddenly in 89 epoch accuracy is 1.34 %<br>
stuck at 1.34 after that (&gt;89 epoch)</p>
<p>learning_rate = 1e-4<br>
optimizer = optim.SGD(model.parameters(), lr=0.0015, momentum=0.9)</p>
<p>Please suggest, why It is like thar</p>

tlim: <p>mind sharing your code for the training loop and accuracy function?</p>

Kapil_Rana: <p>my imageloader getitem()<br>
img1 = self.loader(os.path.join(self.root1, path1))<br>
img1 = np.asarray(img1)<br>
img1 = torch.from_numpy((img1)).float()<br>
target = torch.eye(50)[target]</p>
<p>return img1, target</p>
<p>critiria = nn.NLLLoss()<br>
learning_rate = 1e-4<br>
optimizer = optim.SGD(model.parameters(), lr=0.0015, momentum=0.9)</p>
<p>for batch_idx, (imgs1,labels1) in enumerate(train_loader):</p>
<pre><code>    img_org,target = imgs1.to(device,dtype=torch.float), labels1.to(device)
    img_org = img_org.permute(0, 3, 1, 2)
    optimizer.zero_grad()
    output = model(img_org)
    loss = critiria(output, target)
    loss.backward()

    _, actual = torch.max(target.data, 1)
    _, predicted = torch.max(output.data, 1)
    total = float(len(target))
    correct = (predicted == actual).sum()
    total_correct_test = total_correct_test + np.float(correct.cpu().numpy())


beta =  (len(train_loader.dataset))/ batchsize   
print(train_loss)
train_loss /= beta
acc= 100. * total_correct_train / len(train_loader.dataset)
np_acc=np.around((100* total_correct_train / len(train_loader.dataset)),decimals=2)
</code></pre>
<p>print(“Accuracy==”,np_acc)</p>

tlim: <p>from what I can see, your main loop is accumulating <code>total_correct_test</code> but you are printing <code>total_correct_train</code> → <code>np.around(acc,decimals2)</code>, not too sure what is happening here.</p>

Kapil_Rana: <p>total_correct_test is total number of images classified correctly.</p>
<p>acc is just % of the total_correct_test<br>
np.acc is rounding of total accuracy of decimal = 2</p>

Siavash: <p>I have similiar issue.<br>
Is this problem solved?</p>
Output different size from input in unet

moonhiyoun: <p>I want to get output of different size from input in unet network.</p>
<p>What I am doing now is using unet to remove speech noise.<br>
The input is a <strong>noisy speech</strong> and the target is a <strong>clean speech</strong>.</p>
<p>The input size is <strong>(4, 256, 1)</strong> where 4 is the number of frames.<br>
I want to remove the noise of the current frame using the information of 4 frames including the past.<br>
So I want the <strong>output to be (1, 256, 1) or (256, 1)</strong>.<br>
However, I can change the number of channels, but I’m having trouble getting an output size other than (4, 256, 1).</p>
<p>The model is nothing special.<br>
Please tell me what processing should be done to the output.</p>
<pre><code class="lang-auto">def UNet(input_size=(FRAME, 256, 1)):
    activation_func = 'relu'
    filter_size = 5

    inputs = Input(input_size)

    # Encoder
    conv1 = Conv2D(32, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(inputs)
    conv1 = Conv2D(32, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(conv1)
    pool1 = MaxPooling2D(pool_size=(2, 2), padding='same')(conv1)

    conv2 = Conv2D(64, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(pool1)
    conv2 = Conv2D(64, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(conv2)
    pool2 = MaxPooling2D(pool_size=(2, 2), padding='same')(conv2)

    conv3 = Conv2D(128, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(pool2)
    conv3 = Conv2D(128, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(conv3)
    drop3 = Dropout(0.2)(conv3)

    # Decoder
    up4 = Conv2D(64, 2, activation=activation_func, padding='same', kernel_initializer='he_normal')(
        UpSampling2D(size=(2, 2))(drop3))
    merge4 = concatenate([conv2, up4], axis=3)
    conv4 = Conv2D(64, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(merge4)
    conv4 = Conv2D(64, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(conv4)

    up5 = Conv2D(32, 2, activation=activation_func, padding='same', kernel_initializer='he_normal')(
        UpSampling2D(size=(2, 2))(conv4))
    merge5 = concatenate([conv1, up5], axis=3)
    conv5 = Conv2D(32, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(merge5)
    conv5 = Conv2D(32, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(conv5)

    conv5 = Conv2D(2, filter_size, activation=activation_func, padding='same', kernel_initializer='he_normal')(conv5)

    conv6 = Conv2D(1, 1, activation='relu')(conv5)

    model = Model(inputs = inputs, outputs=[conv6])

    return model
</code></pre>

moonhiyoun: <p>This has been resolved. Until now, I thought that only the square kernel size was possible.</p>
Concat Features from two pretrained models and add a classification head then do Ensemble Learning

Haalum: <p>Hi,</p>
<p>I have two pre-trained classification models. I want to create a third model by</p>
<ol>
<li>removing the classification layers from the pre-trained models and freezing all the other layers, then</li>
<li>concatenating the features from these two models and finally</li>
<li>adding a new classification head [fine-tune this layer only].</li>
</ol>
<p>So I’m going to end up with 3 models. Then I want to do ensemble learning to combine the predictions from these 3 models by voting.</p>
<p>Does anyone know the best way to do this? Thanks in advance.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">This code snippet</a> might be useful as it shows a simple example for your use case.</p>

Haalum: <p>Thank you for the reply. Appreciate it.</p>
<p>A quick question. Since I have two pre-trained models (let’s say modelA &amp; modelB), and I want to create modelC using the pre-trained models and also want to use all three of them for final predictions.</p>
<p>Do I make two copies of modelA (let’s say - modelA_1, modelA_2) and modelB (modelB_1, modelB_2). Keep first copies as they are and take the second copies (modelA_2, modelB_2) remove the classification heads and freeze all other layers, then concat the features and then add a classification head?</p>
<p>The short form of my question is, do I make two copies of each pre-trained model?</p>
<p>My goal is to have three predictions in total from modelA_1, modelB_1, &amp; modelC. I hope I’m making sense. Thanks!</p>

ptrblck: <p>You could create copies, but this will of course increase the memory usage and you would need to execute each model separately, so it’s quite wasteful.<br>
The better approach would be to use e.g. forward hook to get the desired forward activation from the original model during a single forward pass, and to pass this activation to the new classifier heads.</p>
Get the mean of 1D tensor for every value

Harry-KIT: <p>For example, I have 2 tensors below:<br>
a = tensor([[ 4.0155,  3.4703,  4.1574,  3.6536,  3.0531]])<br>
b = tensor([[ 1.0568,  3.8943, -0.1209, 0.6889, 0.3717]])</p>
<p>result:</p>
<p>c = tensor([[ mean(4.0155, 1.0568) ,  mean(3.4703, 3.8943), mean(4.1574,  -0.1209),  mean(3.6536, 0.6889),  mean(3.0531,0.3717)]])</p>

ptrblck: <p>This should work:</p>
<pre><code class="lang-python">a = torch.tensor([[ 4.0155, 3.4703, 4.1574, 3.6536, 3.0531]])
b = torch.tensor([[ 1.0568, 3.8943, -0.1209, 0.6889, 0.3717]])

c = torch.cat((a, b), dim=0)
c.mean(dim=0)
# tensor([2.5361, 3.6823, 2.0182, 2.1712, 1.7124])
</code></pre>
Profiling GPU/CPU usage on a EC2 instance when training a PyTorch Lightning model

dylan: <p>Hi, interested in profiling the GPU/CPU usage when training my model. Specifically, I’m interested in profiling my PyTorch lightning data module and lightning model. Would like to figure out what variables are taking up most space, and what functions can be optimized.</p>
<ol>
<li>I know that vanilla PyTorch profilers enabled DataModule monitoring. Does the same apply to Lightning?</li>
<li>Are there other things I can look into to track memory usage?</li>
<li>How do I visualize profiler activity on Tensorboard?</li>
</ol>
Unable To Save Libtorch Optimizer That Isn't Declared Locally

AidanShipperley: <p>I have an Adam optimizer defined in my class “MyPolicy” that I use to do all of the training on my policy.</p>
<p>In MyPolicy.h I declare a shared pointer to the optimizer:</p>
<pre><code class="lang-cpp">std::shared_ptr&lt;torch::optim::Adam&gt; m_AdamOptimizer;
</code></pre>
<p>and in the constructor for MyPolicy.cpp:</p>
<pre><code class="lang-cpp">m_learning_rate = 0.0003;
m_agent = std::make_shared&lt;Agent&gt;();
m_AdamOptimizer = std::make_shared&lt;torch::optim::Adam&gt;(
    m_agent-&gt;parameters(), torch::optim::AdamOptions(m_learning_rate).eps(1e-5)
);
</code></pre>
<p>This works perfectly, and I’ve been able to train my policy as expected.</p>
<p>However, I can’t save the adam optimizer, because when I include the line of code to save it:</p>
<pre><code class="lang-cpp">std::string optimizerFileName = "myOptimizer.pt";
torch::save(m_policy.m_optimizer, optimizerFileName);
</code></pre>
<p>I get this error:<br>
<code>error C2679: binary '&lt;&lt;': no operator found which takes a right-hand operand of type 'const Value' (or there is no acceptable conversion)</code></p>
<p>I understand that the “proper” way to initialize the optimizer would be without a shared pointer to it, such as with this:</p>
<pre><code class="lang-cpp">torch::optim::Adam optimizer(
    m_agent-&gt;parameters(), torch::optim::AdamOptions(m_learning_rate).eps(1e-5)
);
</code></pre>
<p>But this variable “optimizer” is only declared locally, so I can’t reference it outside of where it’s declared.<br>
I can’t create a new local optimizer before I save and set it equal to m_policy.m_optimizer because you get a <code> attempting to reference a deleted function</code> error, and I can’t declare the optimizer regularly (not as a shared pointer) in the h file because there isn’t a default constructor for torch::optim::Adam. Is there a way to declare an optimizer in the h file and still be able to save it?</p>
<p>Thank you!</p>

AidanShipperley: <p>I figured this out, for some reason you have to dereference the optimizer before it is saved and loaded. This is extremely confusing since you don’t need to, and in fact shouldn’t dereference any nn module when saving or loading.</p>
<pre><code class="lang-cpp">torch::save(m_policy.m_agent, agentFileName);
torch::save(*m_policy.m_optimizer, optimizerFileName);
</code></pre>
<p>Hopefully this helps somebody.</p>
Faster RCNN finetuning on custom dataset

basingse: <p>Hello,</p>
<p>I am new to object detection, so apologies if this question was asked before. I tried to look on internet and couldn’t find convincing answer.</p>
<p>In classification, if someone wants to finetune on custom dataset, the recommended way is</p>
<blockquote>
<p>Take the pretrained model (any architecture of your choice) on image-net. Replace the <code>cls</code> layer with newly initialized layer and finetune the model on your dataset.</p>
</blockquote>
<p>As per myself, I feel the above procedure works in most of cases. I am interested to know how someone can do object detection finetuning for faster-rcnn</p>
<ol>
<li>Should I take pretrained model on COCO dataset and replace the <code>cls</code> and <code>reg</code> layer of detection network ?.</li>
<li>Should I take <strong>only the backbone trained on COCO dataset</strong> and then train whole backbone, RPN and Fast-RCNN on my dataset ?</li>
<li>Should I take <strong>only the backbone trained on image-net</strong> and then train whole backbone, RPN and Fast-RCNN on my dataset ?</li>
<li>Is it advisable to train RPN on custom dataset (as number of examples will be less. for example my dataset has only 2500 images) ?</li>
</ol>
<p>If you think that questions sound really vauge and un-clear and want to know more details, please let me know.</p>
Why is my RNN erroring out at loss storage container?

Jordan_Howell: <p>Hello,</p>
<p>I’m getting the below error and I’m not quite sure why.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "korea_lstm.py", line 134, in &lt;module&gt;
    y_test)
  File "korea_lstm.py", line 105, in full_gd
    train_losses = np.zeros(epochs)
ValueError: maximum supported dimension for an ndarray is 32, found 373482
</code></pre>
<p>I thought it may be a GPU thing but when I take the GPU off as an option, I get the same error.</p>
<p>The below is my model:</p>
<pre><code class="lang-auto">def full_gd(model,
            X_train,
            y_train,
            X_test, 
            y_test,
            epochs=200):
    
    train_losses = np.zeros(epochs)
    test_losses = np.zeros(epochs)
    
    for it in range(epochs):
        optimzer.zero_grad()
        
        outputs = model(X_train)
        loss = mape_loss(outputs, y_train)
        
        loss.backwards()
        optimizer.step()
        
        train_losses[it] =loss.item()
        
        test_outputs = model(X_test)
        test_loss = mape_loss(test_outputs, y_test)
        test_losses[it] = test_loss.item()
        
        if (it+1) % 5 == 0:
            print(f"Epoch {it+1}/{epochs}, Train Loss: {loss.item():.4f},Test Loss: {test_loss.item():.4f}")
            
        return train_loss, test_loss
</code></pre>
<p>I’m nots sure why it’s getting held up at the <code>np.zeros</code> command.  Before trying an RNN, I ran this as an auto regressive model and had no issues.  Is there anything that stands out as glaringly wrong?</p>
How to Quantize CNN into 4-bits?

Kai123: <p>Hi all…I am new to PyTorch as well as Quantization. I want to quantize a CNN model to custom bitwidhts. Could anyone provide me a link to source code so that I can get some idea…Thank you all…</p>

jerryzh168: <p>we do not support 4 bit currently, but contributions are welcome, do you just want to try quantization aware training or do you want to run 4 bit kernels etc.?</p>

Kai123: <p>Thank you for your time Jerry…I want to perform quantize aware training for a cnn model to lower bit precision than int8. I want to know the exact procedure. I found some articles on the internet. But they are mostly telling about how to calculate the scale, zero point and how to quantize and dequantize…I want to know how perform quantization aware training…</p>

seungjun: <p>Hi <a class="mention" href="/u/kai123">@Kai123</a>,</p>
<p>You can check this <a href="https://discuss.pytorch.org/t/expending-pytorch-with-lower-than-8-bit-quantization/80343">thread</a>.</p>
<p>Currently, there is <a href="https://docs.nvidia.com/deeplearning/tensorrt/pytorch-quantization-toolkit/docs/userguide.html" rel="noopener nofollow ugc">pytorch-quantization</a> by NVIDIA.<br>
You can change the number of bits.</p>

jerryzh168: <aside class="quote no-group" data-username="Kai123" data-post="3" data-topic="132677" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kai123/48/42706_2.png" class="avatar"> Kai123:</div>
<blockquote>
<p>Thank you for your time Jerry…I want to perform quantize aware training for a cnn model to lower bit precision than int8. I want to know the exact procedure. I found some articles on the internet. But they are mostly telling about how to calculate the scale, zero point and how to quantize and dequantize…I want to know how perform quantization aware training…</p>
</blockquote>
</aside>
<p>If you just need to do QAT then you can try setting <code>quant_min</code>, <code>quant_max</code> in <code>FakeQuantize</code> module I think</p>
<p>you can find the way we configure <code>FakeQuantize</code> here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/qconfig.py#L129" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/qconfig.py#L129</a>, we just need to configure FakeQuantize with <code>quant_min</code>, <code>quant_max</code> for 4 bit, e.g. -8, 7 and then define the qconfig based on that.</p>

christophezei: <p>Hello I did that, but now How can I simulate model size as well, since sometimes I simulate weights and activations with different bit width for exp: weights using 4 bits and activations: 8bits</p>

jerryzh168: <p>I think you’d probably need to estimate the model size since to support this in tensor you’d have to modify pytorch core right now, which is not easy to do. We might move quantization out of core and make extensions easier in the future</p>
It is necessary to use CPU to do QAT

Nanton: <p>Hi,</p>
<p>I was wondering if it is possible to do QAT with GPU. According to this tutorial ( <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.12.1+cu102 documentation</a>), we need to use CPU.</p>
<p>However, according to this blog( <a href="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/#device-and-operator-support" rel="noopener nofollow ugc">Introduction to Quantization on PyTorch | PyTorch</a>), we can use either CPU or GPU.</p>
<p>Thanks for your attention</p>

Vasiliy_Kuznetsov: <p>Yes, QAT training works on GPU.</p>

suraj.pt: <p>If you scroll all the way down to the QAT cell in this <a href="https://github.com/fbsamples/pytorch-quantization-workshop/blob/main/Quant_Workflow.ipynb">notebook</a> there is an example of QAT with GPU</p>

Nanton: <p>Thanks for all replies here. It seem QAT is supported by GPU. Maybe this tutorial (<a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#quantization-aware-training" class="inline-onebox" rel="noopener nofollow ugc">(beta) Static Quantization with Eager Mode in PyTorch — PyTorch Tutorials 1.12.1+cu102 documentation</a>) has to be changed a little?</p>
<p>Since at the beginning, it says “Note that quantization is currently only supported for CPUs, so we will not be utilizing GPUs / CUDA in this tutorial.”</p>
<p>Thanks <a class="mention" href="/u/jerryzh168">@jerryzh168</a></p>

jerryzh168: <p>QAT supports GPU in the beginning, I think that line means it’s not supported in GPU for inference. We had been working on supporting GPU inference through TensorRT and cudnn, but haven’t tried officially releasing them</p>
Applying grid search to Resnet18

RAFAIL_MAHAMMADLI: <p>Hello,<br>
I would like to apply grid search to Resnet18 model.Can anyone help me about it?</p>

ptrblck: <p><code>skorch</code> is a scikit-learn compatible PyTorch wrapper and provides examples on how to perform <a href="https://skorch.readthedocs.io/en/stable/user/quickstart.html#grid-search">Grid search</a>.</p>

RAFAIL_MAHAMMADLI: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your response. I am wondering how can change this code for implementing Resnet 18 ?</p>
<p>class MyModule(nn.Module):<br>
def <strong>init</strong>(self, num_units=10, nonlin=F.relu):<br>
super(MyModule, self).<strong>init</strong>()</p>
<pre><code>    self.dense0 = nn.Linear(20, num_units)
    self.nonlin = nonlin
    self.dropout = nn.Dropout(0.5)
    self.dense1 = nn.Linear(num_units, 10)
    self.output = nn.Linear(10, 2)

def forward(self, X, **kwargs):
    X = self.nonlin(self.dense0(X))
    X = self.dropout(X)
    X = F.relu(self.dense1(X))
    X = F.softmax(self.output(X))
    return X
</code></pre>

ptrblck: <p>You would have to change the entire model architecture and either implement the resnet from scratch or you could use other implementations for a resnet from e.g. <code>torchvision.models</code>.</p>
Will anyone point me to a sales forecasting tutorial?

Jordan_Howell: <p>Hello,</p>
<p>I need to develop a multi-item/multi-location sales forecasting model to replace a current 20 year old model in production.  I’m seeing a lot of tutorials on stock price prediction / single item foreacasting prediction, which is great, but my model is taking all my items and locations and assuming their all the same.</p>
<p>Is there something out there that can help guide me on how to do this?</p>