akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pycharm does not reconize my pytorch

username: Osama_Harmouche
cooked: <p>Hello,</p>
<p>I built my venv by using pip in my command prompt.</p>
<p>when I tried easyocr and pytorch on pycharm detects their imports as errors, while if I run my script on the same venv on cmd or jupyter notebook it works fine.</p>
<p>even when I installed the package using quick fix pycharm did not recognize cuda for easyocr although on the same script I am using tensorflow-gpu, I tried to reinstall pytorch and easy ocr with pycharm terminal and shows that they were install but still pycharm could not recogize them.</p>
<p>thank you for your time.</p>
Resizing semantic segmentation labels

username: drydenwiebe
cooked: <p>Hello,</p>
<p>I have per pixel labels of the shape [1, H, W] that is of type torch.long (or torch.int64) that are integers representing classes for a semantic segmentation task.</p>
<p>When I try to resize it with torch.nn.functional.interpolate it says that it is not implemented for torch.long.</p>
<p>I was trying to convert the tensor to uint8, then resize, then convert to torch.long, the only issue with this is that it adds NANs to the mask somehow.</p>
<p>Any ideas? What is the standard way to do this?</p>
<p>Thanks!</p>

username: Dazitu616
cooked: <p>See <a href="https://discuss.pytorch.org/t/using-transforms-on-mask-in-multi-class-semantic-segmentation-increases-the-number-of-class/80558/2">this</a> if you are doing resize in data loading. Otherwise I often convert the segmentation mask to <code>float</code> and then <code>torch.nn.functional.interpolate(mask, $TARGET_SHAPE, mode='nearest')</code> and finally convert it back to <code>long</code>.</p>

username: drydenwiebe
cooked: <p>Sweet! That works! I ended up using uint8 instead of floats because interpolate is implemented for that.</p>
Calculating IOU for semantic segmentation

username: drydenwiebe
cooked: <p>Hello!</p>
<p>I want to calculate the mean Intersection over Union (mIoU) of my predicted vs ground truth semantic segmentation labels.</p>
<p>My prediction is of shape [B, 1, H, W] where B is the batch size, H is the height and W is the width. Each pixel has an integer that represents the class (eg: 1-30). The labels are the same shape, including the integer class.</p>
<p>I would like a way to calculate and report the mIoU based on these two tensors. Is there a library or implementation somewhere that I can refer to?</p>
<p>Thank you!</p>
Backpropagation after torch.flip

username: PiTorch
cooked: <p>Hi,</p>
<p>Suppose you have a convolution layer (C), max pooling layer (MP), and dense layers (Ds) in an architecture that follows:</p>
<p>Input → C1 → C2 → MP → Ds → output</p>
<p>Your input is a 10x2 matrix (rows x columns). Your C1 and C2 kernel is a 3x1. So after the C1, you get a 8x2 matrix, after C2 a 6x2. The MP is a (1,2), so you get a 6x1 which is then fed into Ds to get an output. Back propagation works fine.</p>
<p>What about this scenario where you have a flip operation (FLIP) which flips only one column of the previous output (PO).</p>
<p>PO[:1] = torch.flip(PO[:1], dims = 1)</p>
<p>For example, if your output was [[1,4], [2,5], [3,6]], FLIP output: [[1,6], [2,5], [3,4]].</p>
<p>Input → C1 → C2 → FLIP → MP → Ds → output</p>
<p>Here now after your C2 which outputs 6x2, you FLIP then carry on with the rest of the network. So sometimes the max pooling picks up values from the 1st column, sometimes the 2nd (where the position is changed).</p>
<p>Does back prop follow this flip operation? As in, will the C1 and C2 weights adjust accordingly. Or maybe a better question: after the torch.flip operation (which creates a copy from the original input tensor), will back propagation find the right targets to update?</p>
<p>I hope I didn’t confuse you too much. Thank you for your patience and consideration of this question</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="PiTorch" data-post="1" data-topic="143928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/dec6dc/48.png" class="avatar"> PiTorch:</div>
<blockquote>
<p>Does back prop follow this flip operation?</p>
</blockquote>
</aside>
<p>Yes, Autograd will track the <code>flip</code> operation as seen in this small example:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(2, 2, requires_grad=True)
print(x)
&gt; tensor([[-0.6456,  1.6832],
          [ 0.3877,  1.0371]], requires_grad=True)

grad = torch.tensor([[1., 0.],
                     [1., 0.]])
x.backward(grad)
print(x.grad)
&gt; tensor([[1., 0.],
          [1., 0.]])

x.grad = None
y = torch.flip(x, dims=(1,))
print(y)
&gt; tensor([[ 1.6832, -0.6456],
          [ 1.0371,  0.3877]], grad_fn=&lt;FlipBackward0&gt;)

y.backward(grad)
print(x.grad)
&gt; tensor([[0., 1.],
          [0., 1.]])
</code></pre>

username: PiTorch
cooked: <p>Great thank you!</p>
<p>Follow up if you can:<br>
We add another C (C3) after the max pool. Why does a ReLU activation after C3 allow this operation but a Tanh brings up an error (an inplace error) which then requires a clone?</p>

username: ptrblck
cooked: <p>Inplace operations on inputs to layers are disallowed if this layer needs the inputs to calculate the gradients, which seems to be the case for <code>tanh</code> but not for <code>relu</code>.</p>

username: PiTorch
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> . And thank you for the countless other answers you’ve helped me (indirectly from other peoples posts) with over the past year</p>
how to test a batch of images through a pytorch model

username: Mohammad_sAdrA_Ghayo
cooked: <p>I have a pytorch model and I can test a sample image by following scripts. I want to know how can I send a batch of images to this model. I am pretty noob at pytorch and tensorflow.</p>
<h1>
<a name="load-one-image-1" class="anchor" href="#load-one-image-1"></a>load one image</h1>
<pre><code class="lang-auto">    img_trans = test_transform( img ) 
    img_trans = torch.unsqueeze(img_trans, dim=0)
    img_var = Variable(img_trans).cuda()
    score = model(img_var).data.cpu().numpy()
</code></pre>
<p>Also I want to use gpu for this task as well.</p>

username: ptrblck
cooked: <p>You could load a few images and create a batch of these tensors using e.g. <code>torch.stack</code>.<br>
E.g. assuming each image has a shape of <code>[3, 224, 224]</code> and you are loading 5 images, the code would look similar to this:</p>
<pre><code class="lang-python">x = []
# load 5 images
for _ in range(5):
    img = torch.randn(3, 224, 224) # load your actual image here and transform it to a tensor
    x.append(img)
x = torch.stack(x)
print(x.shape)
# &gt; torch.Size([5, 3, 224, 224])

out = model(x)
</code></pre>
<p>Of course you could also use a <code>Dataset</code> etc. so I would generally recommend to take a look at the <a href="https://pytorch.org/tutorials/">tutorials</a>.</p>
<p>Also, remove the <code>Variable</code> usage, as it’s deprecated since PyTorch <code>0.4</code>.</p>
Same platform diff docker (runtime/devel) cant read GPU

username: jaffe-fly
cooked: <p>all test in the same platform(centos, GPU driver is 416.xxx, )<br>
in dockerhub<br>
docker image pytorch/pytorch:1.8.1-cuda11.1-cudnn8-devel is ok, it can read gpu(torch.cuda.is_available() is true) but its too big, so i try pytorch/pytorch:1.8.1-cuda11.1-cudnn8-rutime cant read gpu ,why? where i can see the docker file? or how can i solve use samll docker image can read gpu question</p>

username: marksaroufim
cooked: <p>What do you mean by too big? If you’re interested in studying what the pytorch docker images look like you can take a look at <a href="https://github.com/pytorch/pytorch/tree/master/docker" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/tree/master/docker</a></p>
Loading PyTorch model in multiprocessing Process flashes cmd

username: Tomas_Trdla
cooked: <p>Hi since I haven’t got any responses on s.o., I will try it here… Its my first post on this forum so please let me know if I I need to change something.</p>
<aside class="onebox stackexchange">
  <header class="source">

      <a href="https://stackoverflow.com/questions/66722935/loading-pytorch-model-in-multiprocessing-process-flashes-cmd" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/5990202/tomas-trdla" target="_blank" rel="noopener nofollow ugc">
    <img alt="Tomas Trdla" src="https://www.gravatar.com/avatar/55965af3474bf5853ed8ee77baeee250?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/66722935/loading-pytorch-model-in-multiprocessing-process-flashes-cmd" target="_blank" rel="noopener nofollow ugc">Loading PyTorch model in multiprocessing Process flashes cmd</a>
</h4>

<div class="tags">
  <strong>python, command-line, multiprocessing, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/5990202/tomas-trdla" target="_blank" rel="noopener nofollow ugc">
    Tomas Trdla
  </a>
  on <a href="https://stackoverflow.com/questions/66722935/loading-pytorch-model-in-multiprocessing-process-flashes-cmd" target="_blank" rel="noopener nofollow ugc">02:48PM - 20 Mar 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: marksaroufim
cooked: <p>Looks like issue was resolved on SO thread by updating PyTorch</p>
There is an issue with pytorch conda channel

username: anikitin
cooked: <p>Hi,</p>
<p>there is an issue with pytorch conda channel:<br>
<a href="https://conda.anaconda.org/pytorch/linux-64/repodata.json" class="onebox" target="_blank" rel="noopener nofollow ugc">https://conda.anaconda.org/pytorch/linux-64/repodata.json</a></p>
<p>on the website it says:</p>
<blockquote>
<p>A message has been sent to our webmaster and we will work on fixing this bug.</p>
</blockquote>
<p>unable to install pytorch via conda, any idea how to fix this?</p>

username: marksaroufim
cooked: <p>Thank you for flagging this seems to have been fixed</p>
Infer multiple torch models on a single GPU

username: Tuyen_Vo_Quang
cooked: <ul>
<li>I am currently trying to infer 2 torch models on the same GPU, but my observation is that if 2 of them run at the same time in 2 different threads, the inference time is much larger than running them individually.</li>
<li>Is there any way to make use of single GPU for running multiple models in parallel?</li>
</ul>
<p>Reference:</p><aside class="quote quote-modified" data-post="1" data-topic="83559">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/d78d45/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/inference-multiple-models-on-a-single-gpu/83559">Inference Multiple Models on a Single GPU</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    I am trying to build a system and i need to do inference on 60 segmentation models at same time ( Same models but different inputs). I wonder if this is possible to do on pytorch? 
I am not sure about what kinda system i need to use for this, i am planning to use 4X RTX8000 and if it is not enough i can use two system with 4X RTX8000 each or with a better gpu. 
Would i lose too much performance because of using multiple models? How many models i can put on a gpu, what the performance of it depen…
  </blockquote>
</aside>


username: marksaroufim
cooked: <p>The best approach here is to use an inferencing framework which handles this problem for you pytorch/serve is an example I work and can help answer any questions</p>
How to make pytorch recognize AMD GPU

username: jonkatz6
cooked: <p>I am using an AMD R9 390.<br>
I Installed pytorch given the instructions from the following suggestions:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ffde978ab63fce5e1334351bca6a44a08a0a437e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e_2_690x311.png" alt="image" data-base62-sha1="AvwEFOIod0vOQf8QCsGi8o4tS9w" width="690" height="311" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e_2_690x311.png, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffde978ab63fce5e1334351bca6a44a08a0a437e.png 2x" data-dominant-color="F4E0DD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">843×380 33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However in python <code>torch.cuda.is_availible()</code> returns false. As I understood, this ROCm version is meant to use the amd gpu as cuda.</p>
<p>If there are additional steps I need to take to utilize the GPU, please let me know. Thank you.</p>

username: marksaroufim
cooked: <p>It’s not the most intuitive but <code>.device("cuda")</code> on a AMD GPU actually translates to ROCM to keep semantics  for old code.  <a href="https://github.com/pytorch/pytorch/issues/10670#issuecomment-726420223" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/10670#issuecomment-726420223</a></p>
Using smaller PyTorch version for only using softmax/BCELoss calculations

username: mburaksayici
cooked: <p>Hi Everyone,<br>
If i would like to use PyTorch only for calculation of loss, softmax,sigmoid, and these kind of unheavy jobs, which I won’t train any models and I’ll just receive PyTorch arrays, is there any smaller version of PyTorch that I can use only for  softmax/sigmoid/loss kind of things?  I don’t want to fully install PyTorch. I’ll receive PyTorch model output arrays. I can write softmax/sigmoid functions by myself on smaller package such as numpy, but i would like to keep it consistent and work with all PyTorch functions, but current PyTorch is heavy. PyTorch-cpu is not that heavy, I’m aware, but, is there any alternative that I couldn’t see/think of to use PyTorch functions on metrics?</p>

username: marksaroufim
cooked: <p>Does torchscript meet your requirements? <a href="https://pytorch.org/docs/stable/jit.html" class="inline-onebox" rel="noopener nofollow ugc">TorchScript — PyTorch 1.10 documentation</a></p>
<p>You could also instead just copy out the implementation of those functions in your mini library or leverage something like <a href="https://minitorch.github.io/index.html" class="inline-onebox" rel="noopener nofollow ugc">module0 — MiniTorch 0.1 documentation</a></p>
Real time inference on an SG3 pytorch GAN

username: prismspecs
cooked: <p>Currently developing a flask server in Python to serve generated images in real time. I know this is possible because RunwayML does so at about 90 milliseconds per frame for a 1024x1024 generator. Right now I am running some tests and noticed that it is very slow (like 1000 ms). Is this just because I am saving the image and that is causing the bottleneck? Is it something else? What is the fastest way to pass the image to another application running on the same machine?</p>
<pre><code># blah blah, if Flask receives a jquery post containing an array I use for Z ...
    img = G(z, c, trunc)                           # NCHW, float32, dynamic range [-1, +1], no truncation
    img = (img.permute(0, 2, 3, 1) * 127.5 + 128).clamp(0, 255).to(torch.uint8)
    PIL.Image.fromarray(img[0].cpu().numpy(), 'RGB').save('newtest.png')
</code></pre>
<p>It looks like Runway’s C++ lib receives the image as raw bytes and then re-encodes as an image</p>
<pre><code class="lang-auto">||// Save the incoming pixels to a buffer using JPG compression.|
|---|---|
||ofBuffer compressedPixels;|
||ofSaveImage(pixelsToReceive, compressedPixels, (type == OFX_RUNWAY_JPG)?OF_IMAGE_FORMAT_JPEG:OF_IMAGE_FORMAT_PNG);|
|||
||// Encode the compressed pixels in base64.|
||ofxIO::ByteBuffer base64CompressedPixelsIn;|
||ofxIO::Base64Encoding base64Encoder (false, false, true);|
||base64Encoder.encode(ofxIO::ByteBuffer(compressedPixels.getData(), compressedPixels.size()), base64CompressedPixelsIn);|
</code></pre>

username: marksaroufim
cooked: <p>It sounds like you’re trying to build an inferencing framework from scratch - have you considered using something like <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">https://github.com/pytorch/serve</a></p>
How to reduce the deployment footprint

username: Vishal_Ahuja
cooked: <p>Hi All,</p>
<p>This is the first time I am going to deploy a model in production(AWS), so need some help. The AWS server will not have a GPU and our inference code will be in python - we are not worried about speed right now. The main requirement is to keep the size of the instance minimal. Considering this, can someone please tell me how to install pytorch (CPU only) without having to install all the extra packages that come with anaconda? Any other tips will be very appreciated; I am new to this space.</p>
<p>Sincerely,<br>
Vishal</p>

username: marksaroufim
cooked: <p>If you’re interested you can use torchserve which will allow you to deploy a pytorch model without needing to install pytorch <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">https://github.com/pytorch/serve</a></p>
<p>It’s also the default way in which sagemaker models are served so if you’re already using that things should be easy</p>
<p>An alternative would be loading a torchscript model in c++ like so <a href="https://pytorch.org/tutorials/recipes/torchscript_inference.html#loading-torchscript-models-in-c" class="inline-onebox" rel="noopener nofollow ugc">TorchScript for Deployment — PyTorch Tutorials 1.10.1+cu102 documentation</a></p>
How to obtain the bounding box

username: color_less
cooked: <p>How do I make a Region of Interest image with a bounding box like the one in the third image? both images are in Nifti format.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebd4fd19d4a529a35949005728bdb561bc2720d3.png" alt="ni" data-base62-sha1="xEgwXO8CXPUyHvVrDtjnySLv36r" width="399" height="188"></p>
Fuse Conv+BN layers in pretrained model and save it as new model

username: pmousoul
cooked: <p>Hello,</p>
<p>I would like to load a pretrained model, fuse the Conv+BN layers, and save it as a new model.</p>
<p>I have found functions that perform the merging but it seems that I miss the workflow which is pytorch specific, e.g.</p>
<ol>
<li>add fusion in the model constructor</li>
<li>load_state_dict → complains with error because it expects different layer configuration (not fused)</li>
</ol>
<p>It seems that I should also have modify the model.state_dict but I could not find how this is done.</p>
<p>Thanks in advance,<br>
Panagiotis</p>

username: marksaroufim
cooked: <p><code>optimize_for_inference()</code> does this for you <a href="https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.html" class="inline-onebox" rel="noopener nofollow ugc">torch.jit.optimize_for_inference — PyTorch 1.10 documentation</a></p>
CUDA runtime error for Mixed precision teaining

username: chinmay5
cooked: <p>I am running into a strange issue when using the mixed precision, grad_scaling function. The Pytorch version is 1.9, CUDA is 11.1 and I am using <code>torchio</code> for augmentation. I am not exactly sure what this error indicates and what should be done for debugging. If I remove the mixed precision training, I do not see the error. Any insights would be really helpful.</p>
<p>Here is the error message when using CUDA_LAUNCH_BLOCKING=1</p>
<pre><code class="lang-auto">File "/home/chinmayp/setup/pytorch/lib/python3.6/site-packages/torch/autograd/__init__.py", line 149, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>Thanks,<br>
Chinmay Prabhakar</p>

username: ptrblck
cooked: <p>Could you update to the latest stable or nightly release and check, if you are still hitting this issue?<br>
If so, could you post a minimal, executable code snippet to reproduce the error?</p>

username: chinmay5
cooked: <p>Updating the Cuda version solved the issue. Thank you so much. But, what should be the general steps to follow in such debugging scenarios? There are scenarios, when bumping up cuda version is not desirable.</p>

username: ptrblck
cooked: <p>The general steps for debugging an illegal memory access would be:</p>
<ul>
<li>update PyTorch to the nightly version and check, if this might have been a known and already fixed issue</li>
<li>if that doesn’t help: rerun the code with <code>CUDA_LAUNCH_BLOCKING=1</code> and check which operation is failing to narrow down if it’s coming from the framework or e.g. a custom extension</li>
<li>if you get stuck, create and issue and ping me on it</li>
</ul>
Add batch dimension to this multiplication

username: magicaltoast
cooked: <p>Hi, I have two tensors with shapes (20,10) and (Batch, 10) and I want to multiply them to get out the shape (Batch, 20, 10). I can easily do it without the batch dimension like this <code>(torch.rand(20, 10) * torch.rand(10))</code> but I can’t make it work with batch</p>

username: KFrank
cooked: <p>Hi Damian!</p>
<aside class="quote no-group" data-username="magicaltoast" data-post="1" data-topic="143930" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/magicaltoast/48/46586_2.png" class="avatar"> magicaltoast:</div>
<blockquote>
<p>I want to multiply them to get out the shape (Batch, 20, 10).</p>
</blockquote>
</aside>
<p>Use <code>unsqueeze()</code> to line up the dimensions appropriately, and then let<br>
pytorch’s <a href="https://pytorch.org/docs/1.10.0/notes/broadcasting.html" rel="noopener nofollow ugc">broadcasting</a> handle your batch dimension:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; (torch.rand (20, 10) * torch.rand (6, 10).unsqueeze (1)).shape
torch.Size([6, 20, 10])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Vocab + Vectors: questions on simultaneous usage in pipeline

username: vprokopev
cooked: <ul>
<li>What is a format for a file with cached pre-trained vectors for the Vectors object? (I will do pre-training myself using fasttext). Is it the same as fasttext uses?</li>
</ul>
<p>Example from fasttext docs:</p>
<pre><code class="lang-auto">218316 100
the -0.10363 -0.063669 0.032436 -0.040798 0.53749 0.00097867 0.10083 0.24829 ...
of -0.0083724 0.0059414 -0.046618 -0.072735 0.83007 0.038895 -0.13634 0.60063 ...
one 0.32731 0.044409 -0.46484 0.14716 0.7431 0.24684 -0.11301 0.51721 0.73262 ...
</code></pre>
<ul>
<li>If the format is as shown above, should I just skip the vocab stage and put tokens (after tokenization) straight trough Vocab object to get token vectors? Or should I turn tokens to ints before the tasttext pre-training and then use ints instead of tokens in a vectors file? That way I will make use of the Vocab object.</li>
</ul>
<p>What is the intended way to do it? What was the intended use case for Vectors objects? is it to skip vocab entierly, or do you guys expext to have Vectors objects with tokens being ints?</p>
<p>Thank you!</p>
CUDA Running out of memory after a few batches in an epoch

username: DebadityaPal
cooked: <p>This is my training function:</p>
<pre><code class="lang-auto">def train():

  device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
  model.to(device)
  model.train()
  optim = torch.optim.AdamW(model.parameters(), lr=5e-5)

  for epoch in range(10):
      with tqdm(dataloader, unit=" batch", leave=True, position=0) as tepoch:
        for i, data in enumerate(tepoch):
            inputs = tokenizer(data['text'], padding=True, truncation=True, return_tensors='pt')
            lm_labels = inputs['input_ids'].to(device)
            sarcasm_labels = data['label'].to(device)
            masked_tokens = mask_inputs(inputs['input_ids']).to(device)
            attention_mask = inputs['attention_mask'].to(device)
            
            outputs = model(masked_tokens, attention_mask=attention_mask, labels=lm_labels, next_sentence_label=sarcasm_labels)

            optim.zero_grad()
            # extract loss
            loss = outputs.loss
            # calculate loss for every parameter that needs grad update
            loss.backward()
            # update parameters
            optim.step()
            # print relevant info to progress bar
            tepoch.set_description(f'Epoch {epoch}')
            tepoch.set_postfix(loss=loss.detach().item())
            torch.cuda.empty_cache()
</code></pre>
<p>The training begins fine, however after a few batches in the first epoch itself, I get the Runtime Error of Cuda being out of memory.<br>
The model is a Huggingface BertForPretraining Model.<br>
Can someone help me with this?</p>
<p>Edit: For some reason, if I reduce the size of my Dataset, the problem is solved, but I cannot figure out why this is the case.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="DebadityaPal" data-post="1" data-topic="143914">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/debadityapal/48/46579_2.png" class="avatar"> DebadityaPal:</div>
<blockquote>
<p>For some reason, if I reduce the size of my Dataset, the problem is solved, but I cannot figure out why this is the case.</p>
</blockquote>
</aside>
<p>This might point to a memory increase in each iteration, which might not be causing the OOM anymore, if you are reducing the number of iterations.<br>
Check the memory usage in your code e.g. via <code>torch.cuda.memory_summary()</code> or <code>torch.cuda.memory_allocated()</code> inside the training iterations and try to narrow down where the increase happens (you should also see that e.g. <code>loss.backward()</code> reduces the memory usage).</p>
Masking without breaking graph

username: asura
cooked: <p>Hi,</p>
<p>Say, I have a tensor <code>a</code> : <code>tensor([0.6511, 0.6983, 0.2852, 0.2281], requires_grad=True)</code>. Now, I convert it to 1 if <code>value &gt; 0.5</code> else change it to 0. Now, the new tensor is <code>tensor([1, 1, 0, 0], requires_grad=True)</code>. I use this tensor to perform a masking operation. However, when I backpropagate, the gradients of <code>a</code> become 0.</p>
<p>Another approach might be to use <code>a.detach()</code>, however, now <code>a</code> is not included in the computation graph, and gradients are not computed.</p>
<p>How do I approach this without breaking the graph, and without computing grad value as 0.</p>
<p>Small Snippet:</p>
<pre><code class="lang-python">a = regularizer(torch.mul(per_step_task_embedding, layer_out)) # tensor([0.6511, 0.6983, 0.2852, 0.2281], requires_grad=True)
b = a.round() # # tensor([1, 1, 0, 0], requires_grad=True)
for i, (p, g) in enumerate(zip(params, grad)):
    p.update = - g * b[i] # masking
</code></pre>
<p>Thank you!</p>

username: asura
cooked: <p>Ok, so I have made a hack around this to make it work. Writing what worked for me here, if anyone comes across a similar problem.<br>
Firstly, the regularizer I used had Sigmoid() in the end. I removed it.<br>
So, now I have:</p>
<pre><code class="lang-python">layer_pred = regularizer(torch.mul(per_step_task_embedding, layer_out)) 
sigm = torch.nn.Sigmoid()
temp_pred = layer_pred.detach()
cond = sigm(temp_pred).round() # [0 or 1] 
layer_pred.data = cond.data # replace data in layer_pred without changing graph
masking_module(layer_pred, params, gradients)
layer_pred.data = temp_pred.data # replace data with original data again without changing graph
</code></pre>

username: AlphaBetaGamma96
cooked: <p>Just replying to let you know that you shouldn’t use <code>.data</code> field as it’s only there for backwards compatibility and can lead to incorrect calculations. You can read more <a href="https://discuss.pytorch.org/t/the-difference-between-torch-tensor-data-and-torch-tensor/25995/2">here</a>, try using <code>.copy_()</code> instead.</p>

username: asura
cooked: <p>Thanks a lot, <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> !<br>
Replacing <code>.data</code> with <code>.copy_()</code></p>
<pre><code class="lang-python">temp_pred = layer_pred.clone()
cond = sigm(temp_pred).round() # [0 or 1] 
layer_pred.copy_(cond)
masking_module(layer_pred, params, gradients)
layer_pred.copy_(temp_pred)
</code></pre>
Torch.load OSError: [Errno 22] Invalid argument

username: Arij-Aladel
cooked: <p><span class="mention">@ptrblck_de</span> It is is a very normal loading for a model and it was working very fine in my code, Today using torch.load('path_to_my_model ')  through an error <code>OSError: [Errno 22] Invalid argument</code>   Any insights about what is going on?</p>

username: ptrblck
cooked: <p>Check if the path is containing any Unicode characters (e.g. by copying them from another IDE), if forward/backward slashes are mixed etc.</p>

username: Arij-Aladel
cooked: <p>ok thanks it is working now</p>
Fine-tuning SSD Light

username: mladen.korunoski
cooked: <p>Hello everyone,</p>
<p>I want to fine-tune an object detector in PyTorch. For that, I was using this tutorial:</p>
<p><a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>However, FastRCNN model is not suitable for my use case so instead, I fine-tuned SSDLight. I wrote this code to set a new classification head:</p>
<pre><code class="lang-auto">    from functools import partial
    from torchvision.models.detection import _utils as det_utils
    from torchvision.models.detection.ssdlite import SSDLiteClassificationHead
    
    model = torchvision.models.detection.ssdlite320_mobilenet_v3_large(pretrained=True)

    in_channels = det_utils.retrieve_out_channels(model.backbone, (320, 320))
    num_anchors = model.anchor_generator.num_anchors_per_location()
    norm_layer  = partial(nn.BatchNorm2d, eps=0.001, momentum=0.03)

    model.head.classification_head = SSDLiteClassificationHead(in_channels, num_anchors, num_classes, norm_layer)
</code></pre>
<p>Since my model is not performing well, I want to ask the community if the above code is correct.</p>
<p>Thanks in advance.</p>

username: Enji
cooked: <p>Hello, I’m having the same issue, did you manage to find a solution?</p>
Gradient checkpointing in validation and testing

username: PabloVD
cooked: <p>Hi,<br>
I’m using gradient checkpoints to save memory training a model with Pytorch Geometric. There is no problem during the training procedure. However, during the evaluation time (validation and testing), I get the following error:</p>
<blockquote>
<p>UserWarning: None of the inputs have requires_grad=True. Gradients will be None</p>
</blockquote>
<p>I understand that I get this because during evaluation I do not compute gradients, using <code>@torch.no_grad()</code>. I tried to mark the inputs of the model as if they would require gradients, with <code>input.requires_grad_()</code>, but the warning message still appears.</p>
<p>Is there a problem with that message error? Could it affect the validation results? ¿Or it just would affect the gradients, which I’m not computing during evaluation?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I think you could ignore the warning, since (as you’ve already said) you are not interested in calculating the gradients during validation/testing.</p>

username: PabloVD
cooked: <p>Great, thank you for confirming it!</p>
Having trouble calculating gradients locally inside nn.Module

username: MStypulkowski
cooked: <p>Hi,</p>
<p>I am using <code>torchdiffeq</code>'s (<a href="https://github.com/rtqichen/torchdiffeq" rel="noopener nofollow ugc">https://github.com/rtqichen/torchdiffeq</a>) neural ODE solvers and trying to calculate gradients “locally” inside of the module required by the method (<code>ODEfunc</code>). I am not using any optimizer since I have no trainable parameters.</p>
<p>Here is my simplified code:</p>
<pre><code class="lang-auto">import numpy as np
import torch
# !pip install torchdiffeq # uncomment to install
from torchdiffeq import odeint_adjoint as odeint

def loss_fun(x):
    return x[0]**2 + x[1]**2


class ODEfunc(nn.Module):
    def __init__(self):
        super(ODEfunc, self).__init__()

    def forward(self, t, w):
        _w = w.clone().detach().requires_grad_(True)
        print(_w)
        loss_val = loss_fun(_w)
        print(loss_val)
        loss_val.backward()
        grad = _w.grad
        return grad

w0 = torch.tensor([-0.5, 2.5])
odefun = ODEfunc()
path = odeint(odefun, w0, torch.tensor([0, 1]).float(), rtol=1e-3, atol=1e-3)
</code></pre>
<p>I get the following output:</p>
<pre><code class="lang-auto">tensor([-0.5000,  2.5000], requires_grad=True)
tensor(6.5000)
</code></pre>
<p>and the error on the <code>loss_val.backward()</code> line:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>However, if I run the code for getting the gradients outside of the <code>ODEfunc</code>:</p>
<pre><code class="lang-auto">_w = w0.clone().requires_grad_(True)
print(_w)
loss_val = loss_fun(_w)
print(loss_val)
loss_val.backward()
grad = _w.grad
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">tensor([-0.5000,  2.5000], requires_grad=True)
tensor(6.5000, grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>no errors and correct values of the gradient.</p>
<p>The difference is in the second part my loss function has <code>grad_fn</code>. I don’t understand why wrapping the code in another nn.Module breaks the calculation of the gradients. Any help and workaround will be great!</p>

username: MStypulkowski
cooked: <p>I turnes out there is the <code>torch.no_grad()</code> inside the <code>odeint</code>'s forward pass. I fixed my problem using <code>torch.enable_grad()</code> inside <code>forward</code> in <code>ODEfunc</code>.</p>
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

username: Poorva_Rane
cooked: <p>I’m getting the above error even though I’m not using multiprocessing.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Are you using a dataloader with multiple workers? If so, you might want to have your dataloader work with cpu tensors (and pinned memory) and send the tensors to the gpu in your training loop. Or not use multiple workers to avoid any issue with multiprocessing,</p>

username: Poorva_Rane
cooked: <p>Yes, I was using multiple workers. Not using them in my Dataloader fixed the issue. Thanks!</p>

username: mcskwayrd
cooked: <p>For me, using multiple workers is necessary, and it used to work (in July), but after an update to my PyTorch distro I started getting this error.</p>
<p>Setting num_workers=1 didn’t fix this issue.</p>
<p>Opened new thread <a href="https://discuss.pytorch.org/t/not-using-multiprocessing-but-getting-cuda-error-re-forked-subprocess/54610">here</a>.</p>

username: Gautam_Kumar
cooked: <p>May be I am missing something but can’t I use dataloader and tensors on GPU ?</p>

username: albanD
cooked: <p>It is tricky because CUDA does not allow you to easily share data across processes. And so the transfert from the process that loads the sample to the main one won’t be optimal.<br>
You want to get a Tensor from pinned memory and send it to the GPU in the main process to avoid such issues.</p>

username: Gautam_Kumar
cooked: <p>I shared my code <a href="https://github.com/pytorch/pytorch/issues/1494#issuecomment-647789060" rel="nofollow noopener">here</a> , I followed the workaround and added <code>torch.multiprocessing.set_start_method('spawn')</code> before my DummyDataSet class but now its throwing error</p>
<pre><code class="lang-auto">  File "/usr/local/lib/python3.6/multiprocessing/context.py", line 242, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set
.....
RuntimeError: DataLoader worker (pid 1095) exited unexpectedly with exit code 1. Details are lost due to multiprocessing. Rerunning with num_workers=0 may give better error trace.

</code></pre>

username: albanD
cooked: <p>Do you really need to have the Tensors on the GPU when you load them? Sending them back in pinned memory does not work for you?</p>

username: haowxu
cooked: <p>After using multiple workers I noticed that the only time consumed about data is transferring data from CPU to GPU (in the training loop), so I wonder if it is possible to move the transferring step into the data set class to save training time. Then I got the same error as the author got. So I guess we can not do this, right?</p>
Does torch.squeeze break the computation graph?

username: meshghi
cooked: <p>Hi, for calculating my loss function, I’m using torch.squeeze to change the dimension of the output from [N,1,H,W] to [N,H,W]. I was wondering if this action breaks the computation graoh and cause failure of autograd?</p>

username: ptrblck
cooked: <p>No, <code>squeeze</code> won’t detach the tensor and Autograd will properly calculate the gradients for the original tensor as seen here:</p>
<pre><code class="lang-python">x = torch.randn(2, 1, 2, 2, requires_grad=True)
y = x.squeeze(1)
y.mean().backward()
print(x.grad)
</code></pre>

username: meshghi
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
Adding layers between layers of a pretrained model

username: zhenkai
cooked: <blockquote>
<p>import torchvision.models as models<br>
base_model = models.densenet161(pretrained=True).features</p>
</blockquote>
<p>How can I insert additional layers into the pre-trained model above? In densenet, the input will go through conv0 layer, followed by denseblock1, …, denseblock4 before outputting 2208 channels feature maps. I would like to insert convolution layers right after conv0 and denseblock1 layers, any idea how i can do it?</p>

username: ptrblck
cooked: <p>You could try to manipulate the model initialization <a href="https://github.com/pytorch/vision/blob/f2f490b10bfb4e498478d4cad25300db955bcfcd/torchvision/models/densenet.py#L154-L212">here</a> and add your custom modules. However, this would of course disallow loading the pretrained <code>state_dict</code> directly, and you could either use the <code>strict=False</code> approach (I wouldn’t recommend it without a proper verification as it could easily break if you are not careful), manipulate the <code>state_dict</code> to match the new keys in your model, or manually load the pretrained parameters.</p>

username: zhenkai
cooked: <p>Thanks for the answer! I am wondering if I could add the layers after loading the pre-trained model. I tried adding my custom layer inside the pre-trained densenet like codes below, does it make sense? or i must manipulate the model initialization ?</p>
<blockquote>
<p>class encoder(nn.Module):</p>
<pre><code>def __init__(self, params):

    super(encoder, self).__init__()

    self.params = params

    import torchvision.models as models

   

    self.base_model = models.densenet161(pretrained=True).features

    print("---------------- adding custom layers ------------------------")

    self.base_model.conv0.add_module("custom1", CUSTOM_LAYER(96, 96, para=(1,2,3,4)))

    self.base_model.denseblock1.denselayer6.add_module("custom2", CUSTOM_LAYER(384, 384, para=(1,2,3,4)))

    self.base_model.denseblock2.denselayer12.add_module("custom3", CUSTOM_LAYER(768, 768, para=(1,2,3,4)))

def forward(self, x):

    feature = x

    for k, v in self.base_model._modules.items():

        feature = v(feature)

    return feature
</code></pre>
</blockquote>

username: ptrblck
cooked: <p>Yes, your approach also looks valid and might be even the easiest way. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Batchnorm: how to stop tracking the running stats but use the stats of a batch?

username: torchimport
cooked: <p>Hi,</p>
<p>I have a question about controlling the batchnorm.</p>
<p>As I know, if we apply eval() mode,<br>
the batchnorm stops tracking the running stats and uses the tracked running stats that is registered in the buffer.</p>
<p>However, I would like to stop tracking running stats but use the stats of a given batch in the forward pass.</p>
<p>Is there a method for this?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>If you don’t want to use the running stats at all, use <code>track_running_stats=False</code> during the initialization of the batchnorm layer and the input stats will always be used.</p>

username: torchimport
cooked: <p>Thank you for your quick response!</p>
<p>I am fine-tuning the pretrained model that already has the tracked running stats,<br>
and I want to make sure that the tracked running stats do not change at this time.</p>

username: ptrblck
cooked: <p>I don’t know if you still want to use the running stats later or not. In the former case, call <code>model.bn.train()</code> to use the input activation stats and reload the <code>state_dict</code> on these layers afterwards (or manually reset the running stats). In the latter case, you should be able to replace the pretrained batchnorm layers with new ones using <code>track_running_stats=False</code> and load the affine parameters only (if they are used in the pretrained model).</p>

username: torchimport
cooked: <p>Thanks for your kind explanation.<br>
As your solution, I will manually adjust the batchnorm params or running stats.</p>
<p>Thank you!</p>
Custom loss implementation (function vs class)

username: gonzrubio
cooked: <p>What would make one implement a custom loss as a class instead of as a regular function?</p>
<p>For instance, this implementation of the <a href="https://github.com/aladdinpersson/Machine-Learning-Collection/blob/master/ML/Pytorch/object_detection/YOLO/loss.py" rel="noopener nofollow ugc">YOLOv1</a> loss function is done by expanding <code>nn.Module</code>. Whereas this <a href="https://discuss.pytorch.org/t/how-to-implement-weighted-mean-square-error/2547/2">MSE loss</a> is implemented via a function.</p>
<p>I guess my question is, why is <a href="https://pytorch.org/docs/stable/nn.html#loss-functions" rel="noopener nofollow ugc">every loss</a> implemented as a class in PyTorch if one can simply define a function as explained <a href="https://discuss.pytorch.org/t/custom-loss-functions/29387/2">here</a>?</p>

username: ptrblck
cooked: <p>The advantage of using classes (as <code>nn.Module</code> in this case) is that they can store internal states without explicitly passing all arguments. E.g. if you want to get the unreduced loss you could use the module via:</p>
<pre><code class="lang-python">x = torch.randn(10, 10)
y = torch.randn(10, 10)

# module
criterion = nn.MSELoss(reduction='none')
loss = criterion(x, y)

# functional API
loss = F.mse_loss(x, y, reduction='none')
</code></pre>
<p>so it might be convenient to create the criterion once and just reuse it.<br>
The same applies to a weighted loss or any module.<br>
E.g. for a linear layer you could use both APIs as well:</p>
<pre><code class="lang-python">x = torch.randn(10, 10)

# module
lin = nn.Linear(10, 10)
out = lin(x)

# functional API
weight = nn.Parameter(torch.randn(10, 10))
bias = nn.Parameter(torch.randn(10))
out = F.linear(x, weight, bias)
</code></pre>
<p>Depending on the flexibility you need and your actual workload, pick what fits your use case the best.</p>

username: gonzrubio
cooked: <p>Thank you very much for answering the question. As a follow-up, If I am doing an implementation from scratch of a paper for learning purposes where they propose a new loss (i.e. <a href="https://arxiv.org/abs/1708.02002" rel="noopener nofollow ugc">RetinaNet</a> or <a href="https://arxiv.org/abs/1506.02640" rel="noopener nofollow ugc">YOLO</a>), am I better off implementing them as a function and putting them inside a utils/metrics file or should they have their own dedicated class that implements them? Knowing that I am only going to call them once and use them only in that particular repository.</p>

username: ptrblck
cooked: <p>I would probably implement the new loss function as an <code>nn.Module</code> and try to provide the same or similar arguments as the default loss functions in PyTorch (i.e. the <code>reduction</code> parameter at least, maybe a <code>weight</code> etc.).</p>
<p>EDIT: internally inside the <code>nn.Module</code> you could (and probably should) use the functional form, so that users could in fact pick which approach they want to use (PyTorch uses the same approach of calling the functional API inside the <code>nn.Module</code>s).</p>
How sparse transformer reduces memory complexity

username: hyunsik_kim
cooked: <p>I’m trying to implement the model name “sparse transformer” with pytorch.</p>
<p>As far as I check with <a href="https://github.com/pytorch/fairseq/blob/main/fairseq/modules/sparse_multihead_attention.py#L140" rel="noopener nofollow ugc">fairseq open source</a> for sparse attention mechanism, they simply added the mask matrix with original QK dot product matrix (trg_seq_len ,src_seq_len).</p>
<p>I wonder that above operation is able to reduce memory complexity even though it still calculate n**2 dot product? then could you let me know how?</p>