akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
FSDP: size_based_auto_wrap_policy import error?

Ziyu_Huang: <p>I am using</p>
<pre><code class="lang-auto">python 3.8
torch 1.12
</code></pre>
<p>And I meet this:<br>
ImportError: cannot import name ‘size_based_auto_wrap_policy’ from ‘torch.distributed.fsdp.wrap’<br>
Could anyone provide some suggestion? Thank you!!</p>
<p>In my win10, I have pytorch                   1.12.1          py3.9_cuda11.6_cudnn8_0    pytorch<br>
In my linux server, I have torch                     1.12.0a0+bd13bc6          pypi_0    pypi<br>
my win10 can find the size_based… but server can not!! Why?</p>

ptrblck: <p>That’s expected as your server build is too old.<br>
<a href="https://github.com/pytorch/pytorch/commit/bd13bc6"><code>bd13bc6</code></a> was merged on March 10th, while <code>size_based_auto_wrap_policy</code> was merged on May 10th in <a href="https://github.com/pytorch/pytorch/pull/76858">#76858</a>.</p>

Ziyu_Huang: <p>Sorry, I am still a learner…I am not sure what is ‘server build’? Is that someting I can change? Or a server hardware that I can not change? (Previously I thought I can update pytorch to solve this…)<br>
Thank you!!!</p>

ptrblck: <p>“Server build” refers to the PyTorch version built from source on your “linux server”:</p>
<aside class="quote no-group" data-username="Ziyu_Huang" data-post="1" data-topic="161580">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziyu_huang/48/14158_2.png" class="avatar"> Ziyu_Huang:</div>
<blockquote>
<p>In my linux server, I have torch 1.12.0a0+bd13bc6 pypi_0 pypi</p>
</blockquote>
</aside>
<p>Update it from the latest binaries or rebuilt ir from the current master and it should work.</p>
Keep the gradient function when changing the data type of tensor

sgmcook2: <p>I’m losing the grad fn when changing the data type in a tensor. Is it possible to keep the grad fn when doing this?</p>
<p>e.g. I have a float tensor called my_tensor with a grad fn</p>
<p>new_tensor = my_tensor.type(torch.uint8)</p>
<p>Now the tensor has no grad fn.</p>

srishti-git1110: <p>Hi, please see the following code -</p>
<pre><code class="lang-auto">x = torch.tensor([1.0, 2], requires_grad=True)
x = x+5 # x now has a grad_fn
y = x.type(torch.double)
z = x.type(torch.uint8)

print(y)
print(z)
</code></pre>
<p>out -</p>
<pre><code class="lang-auto">tensor([6., 7.], dtype=torch.float64, grad_fn=&lt;ToCopyBackward0&gt;)
tensor([6, 7], dtype=torch.uint8)
</code></pre>
<p>The explanation is <code>int</code> type tensors cannot require gradients as integral valued functions are not differentiable meaningfully.</p>
<p>And so, the tensor converted to double - <code>y</code> has the <code>grad_fn</code> attribute but <code>z</code> which is an integral tensor has its <code>grad_fn</code> as None .</p>
OSError when importing torchaudio

Omer80: <p>I am new to PyTorch, and trying to learn to work with torchaudio using this tutorial - <a href="https://pytorch.org/tutorials/intermediate/speech_command_classification_with_torchaudio_tutorial.html" rel="noopener nofollow ugc">link</a>. After installing all of the required packages I’ve received an OSError when trying to import torchaudio:</p>
<pre><code class="lang-auto">OSError: dlopen(.../opt/anaconda3/envs/pt/lib/python3.9/site-packages/torchaudio/lib/libtorchaudio.so, 0x0006): Symbol not found: __ZN3c1015SmallVectorBaseIjE8grow_podEPvmm
Referenced from: ..../opt/anaconda3/envs/pt/lib/python3.9/site-packages/torchaudio/lib/libtorchaudio.so
  Expected in: .../opt/anaconda3/envs/pt/lib/python3.9/site-packages/torch/lib/libc10.dylib
</code></pre>
Spectral norm does not work with convolution layers?

Andrew_Legend: <p>I am not sure if I am doing something wrong but when I do the exact thing that the documentation does for linear layers when it comes to spectral normalization, the norm isn’t equal to 1.</p>
<pre><code class="lang-auto">snm = nn.utils.parametrizations.spectral_norm(nn.Conv2d(1, 3, 3))

print(torch.linalg.matrix_norm(snm.weight, 2))
</code></pre>
<p>Am I doing it wrong? if so what can I do to fix it?</p>
Is the suitable model complexity more decided by the data volume or the size of input/output data?

SouthTorch: <p>For example, I would like to find a suitable model to get a value from an image.<br>
Does the mode complexity is decided more by how many data entries are there? or How large the image is (width, height, channel), and how complex the value is (a single value, or an array of values)?</p>
<p>Is it possible that the model can be well trained based on 100,000 data entries, but cannot be well trained  on 1,000,000 data entries?</p>
<p>Based on your experience, what is your opinion? Thanks.</p>
Torch sparse activation function

Max_Unhold: <p>Hello,</p>
<p>in my model, I would like to apply a non-linear function such as the <a href="https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html" rel="noopener nofollow ugc">ReLU</a>  module or the <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.relu.html" rel="noopener nofollow ugc">torch.nn.functional.relu</a> function to a <a href="https://pytorch.org/docs/stable/sparse.html" rel="noopener nofollow ugc">sparse</a> tensor but get a<br>
<code>NotImplementedError</code>.<br>
Are there activation functions that I can apply to a sparse tensor?</p>
Is it possible to generate a single .so file with LibTorch?

Fei_Liu: <p>I want to ship a <code>.so</code> file for client usage (same platform). The client does not have LibTorch installed. Can I somehow pack all the depended libraries into one single library for client usages? Thanks!</p>

Fei_Liu: <p>Bumping up the thread.</p>

ptrblck: <p>I guess you might want to ship your library and statically link every thing else to it.</p>

Fei_Liu: <p>I don’t understand. How does this solve the problem? For my case it’s OK to use either static library or dynamic library. The point is that the libtorch libraries are mostly dynamic and I cannot pack those into one single object (either static or dynamic). And the question is whether LibTorch can give a compiling option to generate a single object file (instead of a light object with some other deps on <code>libtorch/lib</code>)</p>

ptrblck: <p>In case you want to avoid installing <code>libtorch</code> dependencies on the target system, you might want to rebuild <code>libtorch</code> and also statically link everything to it.</p>

Fei_Liu: <p>Ah I see. Let me know if my understanding is correct (my CMake / build knowledge is somewhat limited):</p>
<p>The source code in the libtorch distribution should be enough to build the whole package, including all the libraries under <code>libtorch/lib</code>. So if I want to rebuild a single library, I can just discard all the existing <code>lib/...</code> stuff and build from scratch. This is doable since the source code is in whole (nothing hidden or library file only).</p>
<p>Question for me to get started: for the above “build from scratch” step, what do I need to do? Do I need to learn some CMake basics and use/alter <code>libtorch/share/cmake/...</code> accordingly to give me a single static library (and statically linked with my model code) ? Or do I do something else (what should I learn to get started) ? The way I’m building my package currently is to run some simple g++ command (which  <code>-L ... -l ... </code> to link to the right shared objects e.g. <code>-ltorch_cpu</code>). I did go through the minimal example in the official <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">installation page</a> as well.</p>

ptrblck: <p>I think the first step is to check which libs you exactly want and which one should be statically linked. If I understand your use case correctly, you don’t want to ask the user to install e.g. MKL, but want to provide it. Besides shipping the library in your installation (and dynamically linking to it), you could thus also directly statically link MKL to your build by setting <code>USE_STATIC_MKL=1</code> from <a href="https://github.com/pytorch/pytorch/blob/master/setup.py#L53">here</a>.<br>
I don’t know which libraries should be treated in a similar way, but be aware of the common 2GB linker limitation, which could break your build by default if you want to link libs larger than 2GB (there are specific linker args to avoid these issues).</p>

Fei_Liu: <p>I’ll give it a try. Thanks for your help!</p>
Is there has some best nlp framework process text data suitable with cpp libtorch ?

_Veggie: <p>HI Dear Libtorch Team:<br>
when I first use cpp libtorch to process the text data feel difficult ,I don’t know any nlp framework in cpp writen and suitable with libtorch cpp, I know the opencv cpp is suitable with libtorch in visual domain. but nlp cpp framework I don’t know ,also for the recommend system cpp, do we have some sklearn framework like skorch in cpp version ?<a href="https://github.com/skorch-dev/skorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - skorch-dev/skorch: A scikit-learn compatible neural network library that wraps PyTorch</a></p>
<p>thanks</p>
<h3>
<a name="versions-1" class="anchor" href="#versions-1"></a>Versions</h3>
<p>libtorch 1.12.1<br>
macos<br>
cpp 17</p>
DDP model hangs on torch.save()

asivap: <p>When I save my DDP model with torch.save it hangs indefinitely. After playing around with it some, I noticed that <strong>accessing any value in model.state_dict()</strong> will hang.</p>
<p>For example, state_dict.keys() works fine but state_dict.values() will hang. Upon further inspection I can assign one of the values e.g. val = state_dict[key1] and I can print properties like val.device. However, printing val will hang indefinitely.</p>
<p>Essentially accessing this memory will hang. This feels like very bizarre behavior and I’m not sure how to proceed. Would love ideas!</p>

zoonup: <p>Hi asivap,</p>
<p>Did you ever figure this out? I’m having the same issue.</p>

JamesDickens: <p>Are you loading the model in a DDP setup as well?</p>

TomChen: <p>Hi,<br>
I’m having the same issue. The funny thing is, all of the ckpts can be saved successfully except the last epoch’s ckpt.</p>
Torchvision deformable Conv Layer Memory leak

Shashikant_Verma: <p>While using a network with normal nn.Conv2d, Everyrhing is fine.<br>
But,<br>
Replacing it with torchvisions deformable layer, Cuda memory continuously bumps up every iteration.</p>
<p>Tried with mmcv’s Deformable pack too. Same happens.<br>
What is the issue?</p>

ptrblck: <p>Could you post a minimal, executable code snippet which would reproduce this issue, please?</p>

Shashikant_Verma: <p>I had registered forward hooks to access offsets in deformable conv for visualization.<br>
The hooks were unattended and causing the leak.</p>
<p>Thanks for response.<br>
Issue is resolved.</p>
Does amp.autocast convert all ops to FP16 type?

haoran-hash: <p>After i read the Pytorch docs, i think it’s not.<br>
But what about this situation ?<br>
<code>op1</code> output a Tensor <code>output1</code> (dtype=torch.float16). The next operation <code>op2</code> is FP32 type, so it needs <code>torch.float32</code> input. Now, does <code>op2</code> need to convert <code>output1</code> dtype to <code>torch.float32</code> ?</p>

ptrblck: <p>No, <code>autocast</code> does not convert everything to <code>float16</code>, as it’s numerically not stable enough for a lot of use cases. You could perform if via directly calling <code>model.half()</code> and wouldn’t need <code>autocast</code> for it.</p>
<p>Yes, the transformations are done when needed.</p>

haoran-hash: <p>Similar to <code>Tensor.to(device=torch.float32)</code> ?<br>
In this case, I think it will take a lot of time to switch between <code>torch.float16</code> and <code>torch.float32</code>, because there are many ops with different FP type.</p>

ptrblck: <p>Yes, you are right that transformations are not free, but eventually you would still see an end2end speedup using amp and you should profile your models to check it.</p>

haoran-hash: <p>I get it, thank you very much.</p>
I want to share my Quantize Aware Training Resnet Notebook

Rami_Ismael: <p>I want to improve the documentation of PyTorch quantization. The best thing I can do is to share my Notebook that shows step-by-step PyTorch Quantization Aware Training works and update the date with PyTorch.ao.  Finally, it has a method that shows the size of the model.  I was wondering how I could share my notebook.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1O9qRw7s6W4uCNK7iDZxRy2N7uukqaL3d?usp=sharing">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220915-060050-RC00_474511507" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1O9qRw7s6W4uCNK7iDZxRy2N7uukqaL3d?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1O9qRw7s6W4uCNK7iDZxRy2N7uukqaL3d?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>The <a href="https://github.com/pytorch/tutorials">tutorials repo</a> might be a good fit for your example. CC <a class="mention" href="/u/jerryzh168">@jerryzh168</a> what do you think?</p>

jerryzh168: <p>yeah opening a PR in the tutorials repo would be good, we don’t have time to work on a QAT tutorial yet, so contributions are welcome. our eager mode quantization tutorials are in this folder: <a href="https://github.com/pytorch/tutorials/tree/master/advanced_source" class="inline-onebox" rel="noopener nofollow ugc">tutorials/advanced_source at master · pytorch/tutorials · GitHub</a></p>
Calculating certain blocks of the output matrix

teshnizi: <p>Hi everyone!</p>
<p>I was wondering if I could only calculate specific entries of a matrix multiplication output. Specifically, I have matrices Q and K (both <code>N x D</code>), and I want to calculate a block diagonal sub-matrix of <code>Q K^T</code> (The colored part in the figure below):</p>
<p><strong><img src="https://lh6.googleusercontent.com/w8EI699bcmIe-RFz8hGuyP757ZenGapD6JkW8DukuypwflnPElkc0JfEhprN5bX50w_pTVtOK2ISxYLg9TsvsXxmuNGh6wh23W4IQ2MHiUiHF5aJ4BJxpf-0o-5QhDqyF6tRBbDxrhkbCIs6CwRyIgniljkERD6dnAgGOWq8KK6syMIPw0WLjEPkDU0j=s2048" alt="|618px;x190px;" width="690" height="211"></strong></p>
<p>It’s in fact for calculating attention for two (or more) data entries with different number of input features.</p>
Only 1 loss and multi-branch training

Marshall_stan: <p>My network have <strong>1 input</strong> fed into <strong>3 branches</strong> processed differently, and only <strong>1 loss function</strong> to constraint the output and GT, but I found that the <strong>gradient</strong> of the weights and the biases in the 1st branch are normal,  however in the <strong>2nd and 3rd</strong> branches, <strong>the gradient of the weights are always zero while the gradient of biases are normal</strong>.What’s happening?</p>
<p>Thanks,<br>
Stan</p>
How to combine/concatenate two resnet50 (one pre-trained and one untrained) in parallel manner to predict image memorability?

Amit_Kumar_Mandal: <p>Respected everyone,</p>
<p>I have two models: resnet50 (trained) and resnet50 (untrained). I need to combine these two models to predict image memorability. Can anyone tell me how to achieve this using python?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e67d98d6f90c12dd64bffe6a7b9ea6feb2b2fec2.jpeg" alt="model" data-base62-sha1="wT0UkEHCwBdezyigqPUlakmFJ1U" width="467" height="168"></p>
Transfer weights from classifier encoder for image segmentation task

Mukeyii: <p>Hello,</p>
<p>for a task, I have implemented a segmentation model using Unet from scratch that can successfully segment five features (e.g. A, B, C, D, E). Using the library segmentation models I also tried to compare different backbones with Imagenet weights.</p>
<p>Now, matching this domain, I found a large (150k) and public Kaggle dataset that does not label the above classes at pixel level, but classifies the image data into different classes (for classification tasks). So instead of Imagenet, I would also like to try to transfer the weights of this dataset.</p>
<p>Could I somehow store the encoder or the weights of the encoder (e.g. ResNet) from a simple classifier to load them into my Unet and test if I get better results using this tranfer learning approach? Code examples are very welcome.</p>
<p>Kind regards</p>

ptrblck: <p>It depends on your UNet implementation, but I guess you could reuse the encoder for another classification model (you might already do the opposite of reusing a classification model in your UNet).<br>
To do so, reuse the encoder from the UNet, add a classification head to the model, and train it using the large classification dataset. Afterwards, load the pretrained <code>state_dict</code> of the <code>encoder</code> only to your UNet <code>encoder</code>.</p>
"Start locally" installs wrong version (1.12.0 instead of 1.12.1)

andrekomet: <p>I tried different cuda version with Linux-&gt;Pip-&gt;Python .<br>
It always installs 1.12.0 and not 1.12.1 .<br>
I’d like to use functorch, but it’s not possible with 1.12.0 to install functorch with pip.</p>

ptrblck: <p>Could you post the install command you were using so that we could check if we can reproduce it, please?</p>

andrekomet: <pre><code class="lang-auto">pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
</code></pre>
<p>I also tried the other cuda versions; all with the same result: torch 1.12.0</p>

ptrblck: <p>I cannot reproduce it on my system (Ubuntu + Python 3.8):</p>
<pre><code class="lang-auto">pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com, https://download.pytorch.org/whl/cu116
Collecting torch
  Downloading https://download.pytorch.org/whl/cu116/torch-1.12.1%2Bcu116-cp38-cp38-linux_x86_64.whl (1904.8 MB)
</code></pre>
<p>and <code>1.12.1</code> is found and installed.</p>
How can i check the model device (libtorch)

Seungsu: <p>Hi.</p>
<p>I have two questions about libtorch.</p>
<p>First, i want to know my module’s location in libtorch like cpu or cuda:0.</p>
<p>Is there any api in libtorch?</p>
<pre><code class="lang-auto">torch::jit::module model = torch::jit::load(model_path);

model.??? &lt;- ??? is the api that prints the model location (device)
</code></pre>
<p>Second, Can i directly copy the module from gpu:0 to gpu:1, using NVLinks?</p>
<pre><code class="lang-auto">torch::Device device = torch::Device(torch::kCUDA, 0);
torch::jit::module model_a = torch::jit::load(model_path);
model_a.to(device)

torch::jit::module model_b = ??? &lt;- ??? is the api that copies the model directly using NVLinks. do not across the host dram.
</code></pre>
<p>Thanks for reading.</p>

ptrblck: <p>A model itself does not belong to any specific device as its parameters and buffers can be located on different devices. Assuming you’ve moved the parameters and buffers to a single device, you could grab e.g. the first parameter and check its <code>device</code> attribute instead.</p>
<p>NVLink will be automatically used when detected and you won’t be able to enable or disable it from your SW stack.</p>

Seungsu: <p>Thanks.</p>
<p>Can i get an answer about one more question?</p>
<p>If i want to copy my module (actually create independent Module), what api should i call?</p>
<p>I found apis in torch/jit/api/module.h, that are clone and deepcopy.</p>
<p>What is difference between them?</p>
<p>Actually, i want to copy a module from cuda:0 to cuda:1.</p>

ptrblck: <p>You should be able to <code>clone</code> the module to create independent parameters and push the cloned one to the other device afterwards:</p>
<pre><code class="lang-auto">auto module2 = module-&gt;clone();
</code></pre>
Distributed Training Across Multiple Cloud Providers

Jacob_Blindenbach: <p>I’m trying to train a model with instances from AWS and Google Cloud. Unfortunately neither GLOO or NCCL seem to work with multiple cloud providers. I tried using an Algo VPN to run all the instances in a local setting, but that failed to work as well. Does Pytorch support training across multiple cloud providers?</p>

pbelevich: <p>Although I don’t think it’s a good idea but PyTorch/Gloo/NCCL doesn’t know anything about “cloud providers” so the problem must be only in the network configuration(nodes must be visible to each other) and network latency/throughput</p>
Make NN moves convolutionally over an image and produce multiple outputs

randomnd_random: <p>Assume I trained a NN based on 32x32 input images. I want to pass an image with size 512x512 to the trained NN and I would like the trained NN to move convolutionally over the 512x512 images and for each 32x32 patch, the NN will produce an output. The current behavior is that the NN will move convolutionally over the image, but it will produce one output. Is there a way to modify the default behavior of the NN when passing a testing image?</p>

Karthik_Ganesan: <p>You can just loop over the 512x512 image to get each 32x32 ‘image’ and classify each image. That’s probably the easiest way.</p>
Suggestion for doing transfer learning on a very small dataset

wi-sh: <p>I have a very small private-dataset {500 samples} of a skin disease{BinaryClassificationTask}, I have tried resnet18, densenet121 and efficientNetB0 so far . Just used some basic augmentation{HorizontalFlip, RandomRotation, GaussianNoise, GaussianBlur} and trained for 50 epoch for each. The surprising thing is  I am getting best test-set accuracy on resnet18 {85%} and getting around <code>70%</code> on the rest of the models. Does this mean I have to fine-tune more on my bigger models? Or is it just the lack of data? Currenty, not using any learning rate scheduler and just using Adam as optimizer.</p>
"Key" bias in pytorch multi-head attention is redundant

jmandreoli: <p>The pytorch doc does not really indicate how they implement biases in <code>torch.nn.MultiheadAttention</code>. It only refers to the original paper (Vaswani et al)  which does not use biases. A bit of common sense, and reverse engineering, allows to understand how biases are inserted: instead of applying a linear transform to the key and query inputs, they use an affine transform (i.e. linear + bias).</p>
<p>Now writing down the formula shows that the “key” bias is redundant, because it adds the same term to each cell of a vector (actually each row of a matrix) which is then passed to a softmax (and it is well known that softmax is invariant by such an operation).</p>
<p>To confirm, I ran a pytorch mha twice with the same input, changing only the “key” bias from one run to the next, and observed no difference in the result. As a sanity check, changing the “query” or “value” biases instead did yield a difference.</p>
<p>Snippet at</p><aside class="onebox githubblob" data-onebox-src="https://github.com/jmandreoli/Public/blob/main/mha.py">
  <header class="source">

      <a href="https://github.com/jmandreoli/Public/blob/main/mha.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/jmandreoli/Public/blob/main/mha.py" target="_blank" rel="noopener nofollow ugc">jmandreoli/Public/blob/main/mha.py</a></h4>


      <pre><code class="lang-py"># The purpose of this snippet is to show that the "key" bias in pytorch multi-head attention is redundant
# Consequences:
# * memory space for that bias is wasted
# * torch spends time computing operations involving that bias and propagating null gradients back
# Probably not significant, but still could be avoided

import torch
MHA = torch.nn.MultiheadAttention(embed_dim=12,num_heads=4,batch_first=True)
# variant with different key and value dimensions:
#MHA = torch.nn.MultiheadAttention(embed_dim=12,num_heads=4,batch_first=True,kdim=17,vdim=19)
B = 13; M = 10; N = 8 # batch size, input size, output size
output_shape = (B,N,MHA.embed_dim) # expected shape of the result
biases = dict(zip(('query','key','value'),torch.chunk(MHA.in_proj_bias.data,3)))
def test(bias_name='key'): # either 'key', 'query' or 'value'
  query_input = torch.rand(B,N,MHA.embed_dim) # query
  key_input = torch.rand(B,M,MHA.kdim) # key
  value_input =  torch.rand(B,M,MHA.vdim) # value
  with torch.no_grad():
    # Retrieve the bias
    bias = biases[bias_name]
</code></pre>



  This file has been truncated. <a href="https://github.com/jmandreoli/Public/blob/main/mha.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Consequences:</p>
<ul>
<li>waste of memory to store the useless bias</li>
<li>waste of computation time in forward (adding the bias) and backward (propagating null gradients)</li>
</ul>
<p>It is probably not significant in most applications, but could easily be removed.</p>
Labeling components in a binary tensor

david.bernstein: <p>Hi Everyone, I have a binary image which is the output of a segmentation net. As part of the post processing I have to find all the connected components in this image, which I do with the openCV connectedComponents function. However I was wondering if there is a pytorch equivalent so I can do this operation while the tensor is still on the GPU. Does anyone have or know of something equivalent or similar?</p>
<p>Thanks,<br>
Dave</p>

KFrank: <p>Hi David!</p>
<aside class="quote no-group" data-username="david.bernstein" data-post="1" data-topic="78109" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ecd19e/48.png" class="avatar"> david.bernstein:</div>
<blockquote>
<p>which I do with the openCV connectedComponents function. However I was wondering if there is a pytorch equivalent so I can do this operation while the tensor is still on the GPU.</p>
</blockquote>
</aside>
<p>I don’t know one way or the other whether there is a pytorch / gpu<br>
connected-components function.  (I sort of doubt it.)  But, in any<br>
event, a connected-components algorithm would be unlikely to<br>
benefit much from a gpu’s pipelined architecture.</p>
<p>I suppose that if after constructing the connected components<br>
you needed to move the result back to the gpu, avoiding the<br>
gpu-cpu-gpu memory transfers could be a benefit of running<br>
connected components on the gpu.</p>
<p>Best.</p>
<p>K. Frank</p>

david.bernstein: <p>Hi K, thanks for your reply. I haven’t yet found anything similar in pytorch so perhaps you’re right. Will keep looking though.</p>
<p>Dave</p>

khatrishubham: <p>A GPU connected component implementation <a href="https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9111-a-new-direct-connected-component-labeling-and-analysis-algorithm-for-gpus.pdf" rel="noopener nofollow ugc">https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9111-a-new-direct-connected-component-labeling-and-analysis-algorithm-for-gpus.pdf</a></p>
<p>I think this can easily be taken in.</p>
Index the tensor by float point value

Marshall_stan: <p>Can we index a tensor in PyTorch by float point value to keep the floatTensor’s gradient?</p>

ptrblck: <p>No, you cannot use floating point type as indices.<br>
Depending on your use case you might want to interpolate the original tensor.</p>

Marshall_stan: <p>What actually I want to is that: <strong>Initializing an all-one mask, divide the mask into two parts with high along the radius direction, one part keeps all one and the other part is set to zero,</strong> and <strong>my code</strong> is as followed:<br>
h = nn.Parameter(h, requires_grad=True)</p>
<pre><code>    t_w, t_h = torch.tensor(float(w)), torch.tensor(float(h))
    max_radius = torch.sqrt(t_w * t_w + t_h * t_h)
    high = h * max_radius
    X, Y = torch.meshgrid(torch.linspace(0, h - 1, h), torch.linspace(0, w - 1, w))
    **D = torch.sqrt(X * X + Y * Y)**

    mask = torch.ones((64, 64))
    **mask[(D &lt; high)] = 0**
</code></pre>
<p>and I want to get the mask mentioned above, but it seems that <strong>the grad from mask to high is truncated</strong>, how can I solve this issue? THX!</p>
Pytorch randomresizedcrop vs zoom_range in datagenerator

Ali_Aghababaei: <p>Hi everyone,</p>
<p>I have a problem augmenting my image dataset via zooming in or out. I have read that RandomResizedCrop in torchvision.transforms could be an alternative for the zoom_range in keras data generator. However, I am not able to understand how exactly they are different. For example, in RandomResizedCrop we first crop a “random” portion of the image, but I think that the keras data generator does not act on a random portion of the image.  Can anyone help me in this regard?</p>

ptrblck: <p>I don’t fully understand how the Keras implementation works as <code>zoom_range</code> seems to be only an argument to a generator, not a transformation.<br>
In any case, if you only want to zoom into the image center, you could use a <code>CenterCrop</code> and <code>Resize</code> transformation sequentially.</p>

Ali_Aghababaei: <p>Thank you very much for your response.<br>
I want to exactly understand the difference between Keras and Pytorch implementations for zooming in or out through an example:<br>
If I consider that my images are 256 * 256 and the zoom_range in the Keras data generator is 0.2, for example, I think the exact alternative in Pytorch could be as follows:<br>
First, I should CenterCrop the image to a size of (256 * 0.8  = 205, 256 * 0.8 = 205) and then resize it to the original size of 256 * 256.<br>
Is that right?</p>

ptrblck: <p>No, I don’t think this would match what Keras does, as it seems to be a random operation in Keras as seen <a href="https://faroit.com/keras-docs/1.2.2/preprocessing/image/">in the docs</a>:</p>
<blockquote>
<p><strong>zoom_range</strong>: Float or [lower, upper]. Range for random zoom. If a float, <code>[lower, upper] = [1-zoom_range, 1+zoom_range]</code>.</p>
</blockquote>
<p>If you specify <code>zoom_range=0.2</code> I would thus assume the Keras implementation will randomly zoom into the image with a factor randomly samples between <code>[0.8, 1.2]</code>.</p>
Call of overloaded ‘OperatorMap(<brace-enclosed initializer list>)’ is ambiguous

santisy: <p>Hello, I am trying to build libtorch from source and faced the following error.</p>
<pre><code class="lang-auto">pytorch/torch/csrc/jit/passes/device_type_analysis.cpp: In member function ‘void torch::jit::{anonymous}::DeviceTypePropagationPass::buildRuleRegistry()’:
pytorch/torch/csrc/jit/passes/device_type_analysis.cpp:240:5: error: call of overloaded ‘OperatorMap(&lt;brace-enclosed initializer list&gt;)’ is ambiguous
     };
     ^
In file included from pytorch/torch/csrc/jit/passes/device_type_analysis.cpp:6:0:
pytorch/torch/csrc/jit/ir/ir.h:1651:12: note: candidate: torch::jit::OperatorMap&lt;T&gt;::OperatorMap(std::initializer_list&lt;std::pair&lt;const char*, T&gt; &gt;) [with T = std::function&lt;bool(torch::jit::Node*)&gt;]
   explicit OperatorMap(std::initializer_list&lt;std::pair&lt;const char*, T&gt;&gt; init) {
            ^~~~~~~~~~~
pytorch/torch/csrc/jit/ir/ir.h:1646:12: note: candidate: torch::jit::OperatorMap&lt;T&gt;::OperatorMap(std::initializer_list&lt;std::pair&lt;std::shared_ptr&lt;torch::jit::Operator&gt;, T&gt; &gt;) [with T = std::function&lt;bool(torch::jit::Node*)&gt;]
   explicit OperatorMap(
            ^~~~~~~~~~~‘OperatorMap(&lt;brace-enclosed initializer list&gt;)’ is ambiguous
</code></pre>
<p>I cloned <code>v1.11.0</code> and my compile setting is:</p>
<pre><code class="lang-auto">cmake -DBUILD_SHARED_LIBS:BOOL=ON -DUSE_MPI=OFF -DCMAKE_BUILD_TYPE:STRING=Release -DPYTHON_EXECUTABLE:PATH=`which python3` -DUSE_CUDA=OFF -DCMAKE_INSTALL_PREFIX:PATH=../pytorch-install ../pytorch
</code></pre>
<p>Other information:</p>
<pre><code class="lang-auto">Debian 9
g++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
cmake version 3.18.4
</code></pre>

arizoneer: <p>I have the same issue. Have you had it resolved?</p>
Why are the model output results of training code and demo code different?

ysleer: <p>hello.<br>
No matter how much I think about it, I ask a question because there is something I don’t understand.</p>
<p>We trained with the training code and saved the weights.<br>
When I called the stored weights as demo code and looked at the output, the result was completely different.<br>
I don’t know why.</p>
<p>So as an additional experiment, when I load the weights stored in the training code and look at the model output, it comes out normally, but when I load the weights with the demo code and look at the model output, the results are very strange.</p>
<p>As a result of additional checking, we confirmed that there is no problem with input and weight. Are there any other problems???</p>
<p>I use ddp for training and dp for running demo, could this be a problem??</p>

ptrblck: <p>I would recommend to use a static input (e.g. <code>torch.ones</code>) and compare the outputs of both approaches.<br>
If these outputs are already different, the model’s parameters and/or buffer might not have been loaded correctly or the model has some randomness in its operations (e.g. dropout is used when you’ve forgotten to call <code>model.eval()</code>).<br>
On the other hand, if the outputs of both models for a static input are equal, the issue would most likely be in the data loading and processing.</p>

ysleer: <p>thank you<br>
Thanks, I solved the problem.<br>
The problem was the difference between model.train() and model.eval().<br>
When running the demo, it was confirmed that the same result was obtained when proceeding with model.train().<br>
However, when running the demo, I know that I have to use model.eval(). If so, am I storing the weights in the wrong way? Or did I load it wrong when I load it?<br>
If not both, is the model the problem??</p>
<p>Additionally, the only difference between model.eval() and model.train() is whether to use drop out or batchnorm?? Are there any other differences?</p>
<p>I’m not sure which part to fix.</p>

ptrblck: <p>I don’t know how to interpret your post completely, but it seems you’ve narrowed down the difference of the output to the <code>model.train()</code> mode?<br>
If so, call <code>model.eval()</code> in both scripts and compare the outputs again.</p>
<aside class="quote no-group" data-username="ysleer" data-post="3" data-topic="161440">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ysleer/48/34715_2.png" class="avatar"> ysleer:</div>
<blockquote>
<p>Additionally, the only difference between model.eval() and model.train() is whether to use drop out or batchnorm?? Are there any other differences?</p>
</blockquote>
</aside>
<p>Dropout and batchnorm layers are both using the internal <code>self.training</code> flag, but generally every layer can use it, so it’s best practice to call <code>model.eval()</code> always during the validation run even if these two layers are not used.</p>
How to manually initialize optimizer state?

Mehran_Shakerinava: <p>Hi.</p>
<p>It seems like the optimizer states are created lazily on the first call of <code>step</code>.<br>
I have written a multi-threaded code that share an optimizer, and on the first call to step some threads jump ahead while the state dictionary of a parameter is being created and try to use keys that don’t yet exist and cause a <code>KeyError</code>.<br>
Currently I have fixed this by created the threads with bigger time delays in between, so they don’t catch up with the thread that’s creating the states.</p>
<p>How can I manually force the states to be created for an optimizer like <code>rms_prop</code>?</p>
<p>Thanks.</p>

SimonW: <p>Run <code>step</code> once and then fork threads?</p>

Mehran_Shakerinava: <p>I’ve already tried to run step once, but it seems like step doesn’t do anything if there are no gradients.<br>
Everything seems to be based on laziness in Pytorch <img src="//discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"></p>

SimonW: <p>Yeah, I meant running one iteration. Sorry for not being clear. Alternatively you can also set <code>.grad</code> of all parameters to zeros and then <code>step</code> once. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Let me know if that works</p>

Mehran_Shakerinava: <p>How can I set grads to zero? simply param.grad = 0 or maybe param.grad = Variable(0)? Could you provide a short code?</p>

SimonW: <p><code>param.grad = Variable(para.data.new(para.size()).zero_())</code></p>

seungjun: <p>Thanks <a class="mention" href="/u/simonw">@SimonW</a><br>
The following snippet worked for me.</p>
<pre><code class="lang-python">for group in optimizer.param_groups:
    for p in group['params']:
        p.grad = p.data.new(p.size()).zero_()
        p.grad.requires_grad_(False)
optimizer.step()
</code></pre>
Problem about pytorch-->onnx-->tensorrt-->DS

yangyi_hpu: <p>Dear professor, this is a hurry problem for me, it makes me crazy server days.</p>
<p>I use the pytorch to load ResNet50 in torchvision. I hope to train it by my own dataset, and translate it by tensorrt for deepstream. However, it is a crushing problem for me.</p>
<p>(1) My model is:</p>
<p>class ResNet50_with_softmax(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(ResNet50_with_softmax, self).<strong>init</strong>()<br>
<a href="http://self.net" rel="noopener nofollow ugc">self.net</a> = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)<br>
fc_features = self.net.fc.in_features<br>
self.net.fc = nn.Linear(fc_features, 3)<br>
self.softmax = nn.Softmax(dim=1)</p>
<pre><code>def forward(self, x):
    x = self.net(x)
    x = self.softmax(x)
    return x
</code></pre>
<p>(2) My onnx transform as below:</p>
<p>import torch<br>
from myModel import ResNet50_with_softmax<br>
input_names = [“input”]<br>
output_names = [“predictions/Softmax”]<br>
input_data = torch.randn((2, 3, 200, 200))<br>
model = ResNet50_with_softmax()</p>
<h1>
<a name="param_dic-torchloadresnet50_with_softmax_gpu_dictpth-1" class="anchor" href="#param_dic-torchloadresnet50_with_softmax_gpu_dictpth-1"></a>param_dic = torch.load(“ResNet50_with_softmax_GPU_dict.pth”)</h1>
<h1>
<a name="modelload_state_dictparam_dic-2" class="anchor" href="#modelload_state_dictparam_dic-2"></a>model.load_state_dict(param_dic)</h1>
<p>model.eval()<br>
output = model(input_data)<br>
print(output)<br>
torch.onnx.export(model,<br>
input_data,<br>
‘ResNet18_with_softmax_GPU_dict.onnx’,<br>
export_params=True,<br>
do_constant_folding=True,<br>
input_names=input_names,<br>
output_names=output_names,<br>
dynamic_axes={‘input’: {0: ‘batch_size’, 2:‘width’, 3:‘height’},<br>
‘predictions/Softmax’: {0: ‘batch_size’}})</p>
<p>(3) My command for translating the onnx into tensorrt as below</p>
<p>/usr/src/tensorrt/bin/trtexec --onnx=ResNet18_with_softmax_GPU_dict.onnx<br>
–minShapes=input:1x3x300x200<br>
–optShapes=input:8x3x300x200<br>
–maxShapes=input:16x3x300x200<br>
–workspace=14096<br>
–saveEngine=SGIE_with_softmax_D.engine --fp16</p>
<p>my problem is: a few times I can get the result from the engine by Deepstream,but many times I can not, despite the code and processes are the same,<br>
It makes me crazy. Please kindly help me.</p>
<p>I tried many times,<br>
(1) If I download the pre-trained RestNet18 from pytorch automatically, and don’t train it by my dataset, deepstream can get the result.<br>
(2) Based on (1), I train RestNet18 on my dataset, Deepstream can get the result.<br>
But!!<br>
(3) Go on, I change RestNet18 to Resnet50,Deepstream CAN Not get the result.<br>
(4) I retry ResNet18, it can not work<br>
(5) I reboot the computer, It can not work both ResNet50 and ResNet18.</p>
<p>The code and the process are the same!<br>
what the matter?</p>
<p>Thank you very much! someone kindly help me.</p>
Three questions about accessor

haoran-hash: <ol>
<li>Why use accessor instead of torch::Tensor in cuda kernel function ?</li>
<li>What’s the difference between accessor and packed accessor ?</li>
<li>What’s the difference between packed_accessor32 and packed_accessor64 ?</li>
</ol>
<p>I have read <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#using-accessors" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html#using-accessors</a>, but I am still not clear about these questions. I hope someone can help me. Thanks a lot.</p>

eqy: <p>My understanding is that the accessors are generally used for convenience (and potentially safety), as they allow you to index tensors in kernels in more of a “Python” style, rather than having to deal with understanding the underlying strides for each of the axes—as the main alternative is to access tensors in the “flat” way and to do the indexing arithmetic manually. <code>PackedAccessors</code> according to the docs are the equivalent for CUDA kernels, and I believe 32 vs. 64 is referring to the bit width of the underlying flat index (e.g., if you have large tensors with potentially more than <code>INT_MAX</code> elements, the 64-bit variant should be used).</p>

haoran-hash: <p>I think you have answered question 1 and 3 very well.<br>
But for question 2, I know <code>PackedAccessors</code> is about CUDA. What I want to ask is what this sentence mean in Pytorch docs. I didn’t make it clear before, I’m sorry.</p>
<blockquote>
<p>The fundamental difference with Accessor is that a Packed Accessor copies size and stride data inside of its structure instead of pointing to it. It allows us to pass it to a CUDA kernel function and use its interface inside it.</p>
</blockquote>

tom: <p>To do indexing into the tensors, the CUDA kernel (the function running on the GPU) needs to have the stride and size information on the GPU. The PackedAccessor has fields that contain these, so if you use them as kernel arguments, the information will be transferred. This is in contrast to the Accessor, which only has a reference to stride and size but in exchange is much more efficient to create because you don’t copy the arrays. But if you pass those as a kernel argument, it will transfer the pointer to CPU memory to the GPU, so it’ll lead to the GPU doing invalid memory accesses.</p>
<p>Best regards</p>
<p>Thomas</p>

haoran-hash: <p>Thanks for your detailed answer. I have fully understood it ! <img src="https://discuss.pytorch.org/images/emoji/apple/clap.png?v=12" title=":clap:" class="emoji" alt=":clap:" loading="lazy" width="20" height="20"></p>
When iterater over one huge data file, how to load next huge data file?

Ken_Zheng: <p>Hello, I have ten python pickle files, such 1.pkl ~ 10.pkl, every pickle file larger than 10GB,<br>
whth the RAM limited, How can I:<br>
Load 1.pkl in dataset，training complete, then<br>
Load 2.pkl in dataset, continue training<br>
…<br>
Load 10.pkl in dataset, continue training<br>
Epoch end</p>
<p>How can I achieve this? Thank you</p>

ptrblck: <p>You could implement exactly the logic you’ve described:</p>
<ul>
<li>create a custom <code>Dataset</code> accepting a path to the <code>pkl</code> file</li>
<li>use a loop iterating all 10 paths</li>
<li>create a new <code>Dataset</code> using the current path</li>
<li>train your model inside the loop</li>
</ul>

Ken_Zheng: <p>OK, Thank U very much, I will try it</p>
RuntimeError: mat1 and mat2 shapes cannot be multiplied (8192x1 and 2048x625)

BobKim: <p>Hello everyone!<br>
I’m studying CNN model, and i want to build a CNN with the following structure</p>
<p>CNN<br>
Conv2D(in_channels=1, out_channels=32, kernel_size = 3, stride=1)<br>
ReLU<br>
MaxPool2D(kernel=2, stride=2)<br>
Conv2D(in_channels=32, out_channels=64, kernel_size = 3, stride=1)<br>
ReLU<br>
MaxPool2D(kernel=2, stride=2)<br>
Conv2D(in_channels=64, out_channels=128, kernel_size = 3, stride=1)<br>
ReLU<br>
MaxPool2D(kernel=2, stride=2)<br>
Linear(in_features=2048, out_features=625)<br>
ReLU<br>
Dropout<br>
Linear(in_features=625, out_features=10)</p>
<p>And then, the script i wrote is:</p>
<p>class CNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(CNN, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, stride=1)<br>
self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=1)<br>
self.conv3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1)</p>
<pre><code>    self.dropout1 = nn.Dropout(0.25)
    self.dropout2 = nn.Dropout(0.5)

    self.fc1 = nn.Linear(in_features=2048, out_features=625)
    self.fc2 = nn.Linear(in_features=625, out_features=10)

def forward(self, x):
    x = self.conv1(x)
    x = F.relu(x)
    x = F.max_pool2d(x, 2)

    x = self.conv2(x)
    x = F.relu(x)
    x = F.max_pool2d(x, 2)

    x = self.conv3(x)
    x = F.relu(x)
    x = F.max_pool2d(x, 2)
    x = self.fc1(x)
    x = F.relu(x)
    x = self.dropout1(x)
    x = self.fc2(x)
    output = F.log_softmax(x, dim=1)
    return output
</code></pre>
<p><strong>but i get RuntimeError!</strong><br>
<strong>RuntimeError: mat1 and mat2 shapes cannot be multiplied (8192x1 and 2048x625)</strong></p>
<p>I would really appreciate it if you could let me know what’s wrong with my example.</p>

nassim: <p>Hi<br>
what is the shape of your input?<br>
you are likely getting this error because the shape of x when applying fc1 does not match.<br>
you probably need to Flatten your tensor and check if its dimension is [batch_size,2048],or maybe you need to change the value of stride or maybe add some maxpool layers to make it match</p>

BobKim: <p>Thanks! nassim!<br>
I use MNIST dataset, so the shape of input is [1, 1, 28, 28]<br>
So is <strong>in_features=2048</strong> an invalid value in <strong>Linear(in_features=2048, out_features=625)</strong>?</p>

nassim: <p>Yes in_features=2048 is an invalid value in this case.<br>
You start with x of shape [1,1,28,28] after your first conv layer, x will be of shape [1,32,26,26],<br>
after the max_pool layer ,the shape is [1,32,13,13],after conv2 it is [1,64,11,11],after the next max_pool it is [1,64,5,5], after conv3 it is [1,128,3,3],then after the last max_pool[1,128,1,1], so your fc1 need to be of type (in_features=128,out_features=x), but before applying a fully connected layer your data needs to be of shape [batch_size,in_features] so you need to transform your data from [x,128,1,1] to [x,128].<br>
You can do that by adding x=torch.squeeze(x) before applying fc1.</p>