akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ONNX custom operator runtime error

Buzz-Beater: <p>I have a simple custom operator that inherits from torch.autograd.Function.</p>
<pre><code class="lang-auto">import torch.onnx
import torchvision

from torch import nn
from torch.autograd import Function

class MyReLUFunction(Function):
    @staticmethod
    def symbolic(g, input):
        return g.op('MyReLU', input)

    @staticmethod
    def forward(ctx, input):
        ctx.input = ctx
        return input.clamp(0)

    @staticmethod
    def backward(ctx, grad_output):
        grad_input = grad_output.clone()
        grad_input.masked_fill_(ctx.input &lt; 0, 0)
        return grad_input

class MyReLU(nn.Module):
    def forward(self, input):
        return MyReLUFunction.apply(input)

model = nn.Sequential(
        nn.Conv2d(1, 1, 3),
        MyReLU(),
        )
dummy_input = torch.randn(10, 1, 3, 3)
torch.onnx.export(model, dummy_input, "model.onnx", verbose=True)
</code></pre>
<p>Following the instruction from the documentation, I added <code>symbolic(g, input)</code> for within the class, but when I conduct onnx export, it is still giving me</p>
<blockquote>
<p>RuntimeError: No Op registered for MyReLU with domain_version of 9</p>
</blockquote>
<p>So I was wondering what is the correct way to register a customized function when exporting to onnx.</p>

sksenthilkumar: <p>Is this error solved?</p>

sksenthilkumar: <p>I had the same issue. Then found out that the given example will work in pytorch 1.1, 1.2 but it is throwing erros with the latest version like pytorch 1.5 and 1.7. When I digged into the pytorch onnx documentation I found that we have to pass as an additional argument <code>operator_export_type</code> to <code>torch.onnx.export</code> function(ref: <a href="https://pytorch.org/docs/stable/onnx.html#functions" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/onnx.html#functions</a>).</p>
<p>In my understanding of the documentation in order to support the custom lyaers with symbolic links, the additional argument <code>operator_export_type</code> should be assigned a value of <code>OperatorExportTypes.ONNX_ATEN_FALLBACK</code>.</p>
<p>So your line</p>
<pre><code class="lang-auto">torch.onnx.export(model, dummy_input, "model.onnx", verbose=True)
</code></pre>
<p>Should be replaced with</p>
<pre><code class="lang-auto">torch.onnx.export(model, dummy_input, "model.onnx", verbose=True, operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK)
</code></pre>
<p>Then this error should be solved.</p>
Constant subtraction operation takes a lot of time

Wangaroo: <p>When I was using Insightface, I found that sometimes the function <code>sample</code> took up a lot of time, resulting in low GPU utilization. I used line_profiler to check the time of each line, and found that this line of code runs very slowly</p>
<p><a href="https://github.com/deepinsight/insightface/blob/master/recognition/partial_fc/pytorch/partial_classifier.py#L56" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/deepinsight/insightface/blob/master/recognition/partial_fc/pytorch/partial_classifier.py#L56</a></p>
<p>unreasonable result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30743b384e1af4e4a1748062517e8342d6845c21.png" data-download-href="https://discuss.pytorch.org/uploads/default/30743b384e1af4e4a1748062517e8342d6845c21" title="无标题"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30743b384e1af4e4a1748062517e8342d6845c21.png" alt="无标题" data-base62-sha1="6UDWt5Xh0IYlMXxQwCZiJu7HWnf" width="690" height="482" data-dominant-color="5C5C5C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">无标题</span><span class="informations">1037×725 29.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>reasonable result (I split this line into two lines):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b7b735114bf6cb08d8bc7c5eac406a12d44c129.png" data-download-href="https://discuss.pytorch.org/uploads/default/6b7b735114bf6cb08d8bc7c5eac406a12d44c129" title="rBAoL1_J_KWAdMRvAAEu6vsp8vQ635"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/b/6b7b735114bf6cb08d8bc7c5eac406a12d44c129.png" alt="rBAoL1_J_KWAdMRvAAEu6vsp8vQ635" data-base62-sha1="fkPA0hRhm1nLOb8q2nd8cFlLa5j" width="690" height="498" data-dominant-color="565656"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rBAoL1_J_KWAdMRvAAEu6vsp8vQ635</span><span class="informations">1004×725 29.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can I solve this problem? Thanks.</p>

Wangaroo: <p><code>total_label</code> is distributed in multi gpus, and its shape is (batches * ngpus, embedding_size)</p>
How to control parallel when compiling pytorch source code?

wangxiang2713: <p>Hi, i want to compile pytorch source code, so i follow: <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch#from-source</a><br>
However, when i run: /usr/local/conda/bin/python setup.py install, i get:</p>
<pre><code class="lang-auto">Building wheel torch-1.7.0a0
-- Building version 1.7.0a0
cmake --build . --target install --config Release -- -j 64
...
nvcc error   : 'cicc' died due to signal 9 (Kill signal)
</code></pre>
<p>I only have 16 kernels in my docker, i think it failed because of -j 64. I run /usr/local/conda/bin/python setup.py install --help but don’t find the parameter. So, how can i change it?</p>

ptrblck: <p>You can use the <code>MAX_JOBS</code> env variable via:</p>
<pre><code class="lang-auto">MAX_JOBS=1 python setup.py install
</code></pre>

wangxiang2713: <p>Get, thank you very much!</p>
Install error cant detect cuda

Nian_Shi: <p>torch.cuda.is_avaliable()<br>
return False</p>
<p>Collecting environment information…<br>
PyTorch version: 1.2.0<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.0.130<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Ubuntu 18.04.5 LTS (x86_64)<br>
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0<br>
Clang version: Could not collect<br>
CMake version: Could not collect</p>
<p>Python version: 3.6 (64-bit runtime)<br>
Is CUDA available: False<br>
CUDA runtime version: 10.0.130<br>
GPU models and configuration:<br>
GPU 0: TITAN RTX<br>
GPU 1: TITAN RTX<br>
GPU 2: TITAN RTX<br>
GPU 3: TITAN RTX<br>
GPU 4: TITAN RTX<br>
GPU 5: TITAN RTX<br>
GPU 6: TITAN RTX<br>
GPU 7: TITAN RTX</p>
<p>Nvidia driver version: 450.80.02<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.19.4<br>
[pip3] torch==1.2.0<br>
[pip3] torchvision==0.4.0a0<br>
[conda] _pytorch_select           0.2                       gpu_0<br>
[conda] blas                      1.0                         mkl<br>
[conda] cudatoolkit               10.0.130                      0<br>
[conda] mkl                       2020.2                      256<br>
[conda] mkl-service               2.3.0            py36he904b0f_0<br>
[conda] numpy                     1.19.4           py36h8732dcd_1<br>
[conda] pytorch                   1.2.0           cuda100py36h938c94c_0<br>
[conda] torchvision               0.4.0           cuda100py36hecfc37a_0</p>

ptrblck: <p>Make sure <code>nvidia-smi</code> detects all devices and doesn’t print any error status for any device.<br>
If you’ve recently upgraded a driver, reboot the system.<br>
If that doesn’t help, create a new virtual environment and install the latest stable release (or nightly binary).</p>
Why pytorch consumes much more memory with small memory allocation?

be7f984b2f2e66ba7969: <p>Hi,</p>
<p>I am running pytorch with a simple cuda allocation.</p>
<pre><code class="lang-auto">import torch
a=torch.zeros((1,3,352,640), dtype=torch.float32).cuda()
</code></pre>
<p>The amount of memory should be (1<em>3</em>352*640 / 1000000) = 2.7MB</p>
<p>However, with nvidia-smi, it costs about 591MB.</p>
<p>And also the cpu memory is also increased by 1.2G.</p>
<p>this is unacceptable when running on edge device.</p>
<p>any idea?</p>

ptrblck: <p>The majority of the memory allocation on the GPU are used by the CUDA context, which is created during the first CUDA operation. You could lower is by building from source for your GPU architecture and remove unnecessary libs (such as NCCL in case you don’t need it).</p>

be7f984b2f2e66ba7969: <p>please see <a href="https://discuss.pytorch.org/t/building-pytorch-wheel-with-cmake/105102" class="inline-onebox">Building pytorch wheel with cmake</a></p>
Cuda return to false

awannano: <p>Hi I’m new with programming.</p>
<p>I’m facing some problem with cuda installation. I want to know, is cuda toolkit installation<br>
a. in window - using downloaded installer from (<a href="https://developer.nvidia.com/cuda-downloads" rel="noopener nofollow ugc">https://developer.nvidia.com/cuda-downloads</a>)<br>
b. in python - usingconda install pytorch torchvision torchaudio cudatoolkit=8.0 -c pytorch<br>
the same thing?<br>
Do I need both installed or only in python?</p>
<p>Hope someone can help me with this. Thanks.</p>

ptrblck: <p>If you install the binaries (via <code>conda</code> or <code>pip</code>) they will ship with their own CUDA runtime and you would only need to install the corresponding NVIDIA driver on your system.<br>
Have a look <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility">here</a> (Table 1) to find the appropriate driver versions for different CUDA releases.</p>

awannano: <p>I’m using window 10. I have GPU nvidia quadro 6000 version 377.83 (latest). If I refer to this link <a href="https://developer.nvidia.com/cuda-legacy-gpus" rel="noopener nofollow ugc">Check GPU-Cuda Computibility</a> and quadro 6000 computibilily is 2.0 which match with cuda 8.0</p>
<p>However, still torch.cuda.is_available() returning to False. Is this a common problem?</p>

ptrblck: <p>PyTorch supports CUDA&gt;=9.2 and the binaries ship with a min. compute capability of 3.7, so your device won’t be supported anymore.</p>

awannano: <p>I see. That solves the problem. No wonder the “false” problem keeps repeating.</p>
<p>Thanks a lot!</p>
Building pytorch wheel with cmake

be7f984b2f2e66ba7969: <p>Hi,</p>
<p>For some reasons I can only use cmake to build pytorch source</p>
<pre><code class="lang-auto">mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX="$HOME/pytorch_build
make -j4 &amp;&amp; make install
</code></pre>
<p>However, I didn’t see any wheel or any pytorch python interface in the build directory, only <code>libtorch_python.so</code> and <code>python3.7/caffe2</code> seems related to python.</p>
<p>any idea?</p>
Inplace elementwise multiplication of 3 dim tensors with CUDA

matthias.l: <p>Hi I have a tensor <code>x</code> of shape: [8, 8, 89] and a second tensor <code>s</code> of shape [8,8] containing only values <code>1</code> and <code>-1</code>.<br>
Now I want to expand <code>s</code> to the same shape of <code>x</code>:</p>
<pre><code class="lang-python">s = s.unsqueeze(2).expand(x.shape)
</code></pre>
<p>and multiple them element-wise:</p>
<pre><code class="lang-python">x = x*s
</code></pre>
<p>Two questions:</p>
<ol>
<li>Why do I get a <code>RuntimeError: CUDA error: invalid configuration argument</code> when running this code with CUDA.</li>
<li>Is there a better operator which does this inplace?</li>
</ol>
<p>Thank You!</p>

albanD: <p>Hi,</p>
<ol>
<li>I can’t reproduce this error when running on colab.<br>
Does the following code works for you?</li>
</ol>
<pre><code class="lang-python">import torch

s = torch.rand(8, 8, device="cuda")
x = torch.rand(8, 8, 89, device="cuda")

s = s.unsqueeze(2).expand(x.shape)

s * x
</code></pre>
<ol start="2">
<li>You can do this inplace by doing <code>x *= s</code>.</li>
</ol>

matthias.l: <p>Thank you!<br>
Ok. You are right. The issue must be related to something deeper. I just copied the part in the model which raised the issue, but now all operations on <code>x</code> seem to have same issue.<br>
Had to investigate this in more detail…</p>
<p>Is it possible to dump all relevant information of a tensor to better reproduce issues like that?</p>
<blockquote>
<p>x: torch.Size([8, 8, 89]) torch.float32 cuda:0<br>
s: torch.Size([8, 8, 89]) torch.float32 cuda:0</p>
</blockquote>

matthias.l: <p>I found the issue. The problem was from the operation before which calculates <code>x</code>. It was caused by a custom CUDA kernel which spawned 89 threads and 8x8 blocks.<br>
It seems that pytorch evaluates the graph lazy and the error popped up at the following operation.</p>

albanD: <p>It is the cuda API that is asynchronous. So errors will happen later on indeed.<br>
You can force it to be synchronous by setting <code>CUDA_LAUNCH_BLOCKING=1</code> env variable to get the error to point at the right place <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

matthias.l: <p>Ok the actual error was in my custom operator:<br>
The mistake was related to CUDA not directly to PyTorch.<br>
When launching the CUDA kernel with <code>(1,1,N)</code> with <code>N &gt; 64</code> threads there is a failure. But when you put the number of threads in a scalar instead of a 3d vector, everything works well.</p>
How to construct a U-Net with pretrained Resnet model in PyTorch?

HuakunShen: <p>I’ve seen some blogs talking about using a resnet as the encoder part of a U-Net.<br>
Can I use a pretrained resnet?<br>
In other words, how can we get the results intermediate layers from the pretrained resnet model since we need result from previous layers to do the cross connection.</p>
Why my deep learning model is taking even more time in colab?

krishna511: <p>I am using google colab to run my deep learning model. I dont know why is it taking more time in colab/<br>
I am using this snippet for device mounting in training the model.  Plz suggest something , its taking 30 min for 1 epoch.<br>
‘’’<br>
if args.use_gpu:</p>
<pre><code>use_cuda = torch.cuda.is_available()

device = torch.device("cuda" if use_cuda else "cpu")
</code></pre>
<p>else:</p>
<pre><code>device='cpu'
</code></pre>
<p>‘’’<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> plz help sir</p>

pchandrasekaran: <p>Is the notebook setup to use a GPU? Because <code>device</code> will still hold <code>"cpu"</code> if the settings aren’t changed.</p>

krishna511: <p>yes it is.<br>
But no change in training duration…</p>

ptrblck: <p>The difference between Colab and your local setup would come from all used processing units, i.e.:</p>
<ul>
<li>used CPU</li>
<li>data storage and bandwidth to access it</li>
<li>used GPU</li>
<li>different libraries</li>
</ul>
<p>etc.<br>
You could profile the code to check, which part of the training takes most of the time.</p>

krishna511: <p>Thanks for reply sir,<br>
But I am not getting last line, How to profile the code?<br>
Actually its taking more time then my local machine.</p>

ptrblck: <p>You could add timers to the code and check e.g. how long the data loading needs as done in the <a href="https://github.com/pytorch/examples/blob/a3f28a26851867b314f4471ec6ca1c2c048217f1/imagenet/main.py#L279-L282">ImageNet example</a>. Also, the <a href="https://pytorch.org/docs/stable/autograd.html?highlight=profile#torch.autograd.profiler.profile"><code>torch.autograd.profiler</code></a> might be useful.</p>
Custom dataset with multiple datasets

Rabeeh_Karimi: <p>Hi<br>
I have multiple datasets, I want to sample them with some weights like 2x dataset1 1x dataset2, … and then define a dataloader for this concatenated dataset. How I can define a custom dataset which can have sampling mechanism?<br>
Also the datasets are very large, and I cannot bring them to memory. thanks</p>
<p>After making concateneted sampled dataset, I want to define 1 dataloader for it, also I need to make sure each batch comes from one type of dataset. thanks</p>

ptrblck: <p>You could use a <code>WeightedRandomSampler</code> for the concatenated datasets.<br>
If you need to sample a complete batch from one of the internal datasets only, you could create a custom <code>sampler</code>, create the indices internally and return the batch indices to the custom dataset.</p>
How do you get data of next iteration when shuffle=True?

Juna: <p>Hi<br>
I’m trying to get data for the next iteration in the dataloader loop.<br>
The point is I need both data for the current iteration and the one for the next iteration or even for n times the next iteration.<br>
So I did following.</p>
<pre><code class="lang-auto">for num_iter, (img, label) in enumerate(train_loader):
        train_loader_list = list(train_loader)
        tuple = train_loader_list[num_iter]
</code></pre>
<p>I was going to get data for the next iteration by doing train_loader_list[num_iter+1]<br>
but tuple[0] and img were different!<br>
I think it’s because of shuffle=True.<br>
In this case, what’s the most efficient way to get both data for the current iteration and the one for the n times next iteration?</p>

Jingnan_Jia: <p>If you think the data for the current iteration and next iteration should be the same, why not just use the same data.</p>
<p>data_current_iteration = next(train_loader)<br>
data_next_iteration = data_current_iteration</p>
<p>Or do I misunderstand your question?</p>

Shima_Shahfar: <p>I am not sure if I understand your question correctly or not but I’d like to mention few things that I think might be helpful.</p>
<ol>
<li>Basically, when you set shuffle equal to <code>True</code> your dataset indices will change every epoch and you won’t get the same instances in two consecutive epochs since you reshuffled at every epoch. Also, your dataloader should give you the correct (image, label) pairs and if it doesn’t seem right, please check the output of your dataset one more time.</li>
<li>If you need more than one instance for your calculation at each iteration, I would suggest changing the <code>__getitem__</code> function of your dataset. (It is not a good idea if you are just trying to update the weights less frequently or training a GAN , …)</li>
<li>Here is another possible way but I cannot say if it’s the most efficient way for your case or not.</li>
</ol>
<pre><code class="lang-auto">for curr_iter in range(num_iter): 
        i = 0 
        while i &lt; number_of_iterations_you_need:
              data_iter = next(iter(train_loader))
              #do anything you want
              i +=1
</code></pre>
Sometimes Stucks Training Oddly

JisongXie: <p>My training sometimes stucks oddly. It stops printing.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93251efe793dbff08b233c651c6ca799191a856e.png" data-download-href="https://discuss.pytorch.org/uploads/default/93251efe793dbff08b233c651c6ca799191a856e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93251efe793dbff08b233c651c6ca799191a856e.png" alt="image" data-base62-sha1="kZHICswZVUp9tBuACfVm3AJVgRg" width="690" height="54" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">747×59 2.14 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And the GPU utils is 0%.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a17c190b278bb94f51c99c41fdf598fcacacffe.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a17c190b278bb94f51c99c41fdf598fcacacffe" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a17c190b278bb94f51c99c41fdf598fcacacffe.png" alt="image" data-base62-sha1="8hUABYQRE8shQWXAdZRWFb5E9c2" width="690" height="66" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">877×84 3.43 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
When I press Crtl + C to terminate it, it prints log like this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3d9a1430271be4cb6896ee27138a002c786d3f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/d3d9a1430271be4cb6896ee27138a002c786d3f6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3d9a1430271be4cb6896ee27138a002c786d3f6.png" alt="image" data-base62-sha1="ue710pEc4en8imWaNqeYPglnUbk" width="690" height="279" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1267×513 25.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It seems that the error occurs when loading data by multiprocessing.<br>
Anyone knows why?</p>
<p>The verison of some modules in my code:<br>
python version：3.7.9<br>
torch version：1.4.0<br>
cuda version：10.0<br>
I use the docker interactively, and use screen to run in background.</p>
Why does my normalization make images darker?

spadel: <p>I have written the following function to normalize images to the range [0, 1]:</p>
<pre><code class="lang-auto">def normalize(image):
    bn, kn, h, w = image.shape
    image = image.view(bn, kn, -1)
    image -= image.min(2, keepdim=True)[0]
    image /= image.max(2, keepdim=True)[0]
    image = image.view(bn, kn, h, w)

    return image
</code></pre>
<p>Unfortunately this makes images darker, as can be seen here:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b97be5dfb607d89f6831571887502f8358231bf.png" alt="Bildschirmfoto 2020-12-01 um 18.59.24" data-base62-sha1="hDlPXyK3LEVxvGt7e2d32TyHDC7" width="248" height="240"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec87dec93a53c744de00ba97903752873adfa09d.png" alt="Bildschirmfoto 2020-12-01 um 19.03.15" data-base62-sha1="xKrMAAjNioRtwHm9uPjh1N6xbqJ" width="138" height="132"></p>
<p>The first one is before and the second one after normalization. How can that be?</p>

ptrblck: <p>The posted images don’t seem to be only normalized but also rotated or cropped, so I’m unsure which transformations are applied on the image.<br>
Could you upload the original image so that we can check it?</p>
Why it is saying win_length is not defined in librosa.feature.melspectrogram?

krishna511: <p>Hi there<br>
I am using google colab , while executing code in local machine m getting no error , but running training.py file in colab , m getting this error.</p>
<pre><code>1534 mel_basis = filters.mel(sr, n_fft, **kwargs)
</code></pre>
<p>1535<br>
1536     return np.dot(mel_basis, S)</p>
<p>TypeError: mel() got an unexpected keyword argument ‘win_length’</p>
<p>but in doc of this librosa feature, I find this win_length variable. plz guide</p>

ptrblck: <p>I guess the used <code>librosa</code> versions might differ and you might need to update/downgrade the Colab version if your local code is working fine.</p>
Different train result when just defining extra layers in code!

Mirsadeghi: <p>Hi, I face a strange issue. I am training a simple fully connected network on mnist dataset.<br>
I have used ‘torch.manual_seed’ for code reproducabiliity.<br>
In general, I got an accuracy like acc = [0.3722, 0.3514, 0.4277] in the first three epochs.<br>
When I add a definition of an extra network (which I do not use in training and I do not feed it to optimizer), I get different results!!<br>
I check my code multiple times, but just commenting the line of the definition of an extra network returns the results to the general case even if I have used random shuffle in dataloader.</p>

ptrblck: <p>Initializing an additional layer will initialize all parameters by calling into the pseudorandom number generator and will thus change all following random calls. This is expected behavior and we had some topics about the same behavior here. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Mirsadeghi: <p>What should I do about this? I want to do some other operation while not affecting the base training?<br>
besides, Could you please link me to the other issues? <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>

ptrblck: <p>Interesting topics might be:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/whats-the-difference-between-those-two-simple-nn-models/37215/4" class="inline-onebox">What's the difference between those two simple nn models?</a></li>
<li><a href="https://discuss.pytorch.org/t/unused-model-parameters-affect-optimization-for-adam/27563/12" class="inline-onebox">Unused model parameters affect optimization for Adam</a></li>
<li><a href="https://discuss.pytorch.org/t/problem-using-torch-manual-seed/19445/11" class="inline-onebox">Problem using torch.manual_seed</a></li>
</ul>
<p>You would have to avoid calling into random methods (not possible when randomly initializing new layers) or (re-)seed the code after initializing the model so that the training loop is directly seeded.</p>
Matrix_exp gives unexpected result on complex matrix

ssrs: <p>Hi, I am happy to find PyTorch support for complex numbers, but when I used torch.matrix_exp, it returned unexpected result.<br>
I think torch.matrix_exp may have the same function as scipy.linalg.expm, and they return same result on real matrix:</p>
<pre><code class="lang-auto">import numpy as np
from scipy.linalg import expm
a=np.zeros((2,2),dtype=np.float64)
a[0,1]=1
a[1,0]=1
b=expm(a)
print(b)
</code></pre>
<p>[[1.54308063 1.17520119]<br>
[1.17520119 1.54308063]]</p>
<pre><code class="lang-auto">import torch
a=torch.zeros(2,2,dtype=torch.float64)
a[0,1]=1
a[1,0]=1
b=torch.matrix_exp(a)
print(b)
</code></pre>
<p>tensor([[1.5431, 1.1752],<br>
[1.1752, 1.5431]], dtype=torch.float64)</p>
<p>But the result differs on complex matrix:</p>
<pre><code class="lang-auto">import numpy as np
from scipy.linalg import expm
a=np.zeros((2,2),dtype=np.complex128)
a[0,1]=1
a[1,0]=1
b=expm(a)
print(b)
</code></pre>
<p>[[1.54308063+0.j 1.17520119+0.j]<br>
[1.17520119+0.j 1.54308063+0.j]]</p>
<pre><code class="lang-auto">import torch
a=torch.zeros(2,2,dtype=torch.complex128)
a[0,1]=1
a[1,0]=1
b=torch.matrix_exp(a)
print(b)
</code></pre>
<p>tensor([[-10.9104+0.j,   1.6732+0.j],<br>
[  1.6732+0.j, -10.9104+0.j]], dtype=torch.complex128)</p>
<p>I am using PyTorch 1.7.0, Python 3.8.5<br>
Thanks</p>

albanD: <p>Hi!</p>
<p>Support for complex numbers with <code>matrix_exp</code> was added yesterday: <a href="https://github.com/pytorch/pytorch/pull/48363">https://github.com/pytorch/pytorch/pull/48363</a> <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
You can install a nightly build and it should work fine now!</p>

ssrs: <p>Thanks, it helps a lot</p>
Static/Dynamic Quantization

mhamdan: <p>I tried quantizing a model using both static and dynamic quantization. Both schemes quantized the weights of the layers but did not quantize the biases. Is there a reason why? and how can I quantize the biases?</p>
<p>Implementation is similar to <a href="https://github.com/Sanjana7395/static_quantization/blob/master/quantization%20pytorch.ipynb" rel="noopener nofollow ugc">this</a></p>

dskhudia: <p>biases are not quantized and kept in fp32. For convs and Linears, bias are dynamically quantized before addition while doing convs/Linears.</p>

mhamdan: <p>Hi, Thanks for your reply.<br>
If you’re saying that biases are not quantized, what do you then mean by biases are dynamically quantized for linears?</p>

dskhudia: <p>When the linear is run, it converts biases to int32 before adding to matmul result.</p>

mhamdan: <p>I see. Do you have any idea why PyTorch does it like that?</p>

dskhudia: <p>if quantized, biases are usually quantized with a scale = activation_scale * weight_scale so that quantized bias can directly be added to matmul output in quantized domain. In pytorch eager mode (due to dynamic nature of pytorch graph), knowing activation scale statically is impossible.</p>
Libtorch, calling the jit.export method

Daewoong.Ahn: <p>In my case,</p>
<pre><code class="lang-auto">class NN(nn.Module):
    @torch.jit.unused
    def forward(self):
        pass

    @torch.jit.export
    def another(self):
        pass

nn = NN()
scripted = torch.jit.script(nn)
scripted.another()
</code></pre>
<p>It works fine on the <code>Python</code>(In real, much more complex structure)</p>
<p>But In cpp(libtorch),</p>
<pre><code class="lang-auto">error: ‘using Module = struct torch::jit::Module {aka struct torch::jit::Module}’ has no member named ‘another’
</code></pre>
<p>How can I call the <code>another</code> in cpp level?</p>

Daewoong.Ahn: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://g-airborne.com/favicon-32x32.png" class="site-icon" width="16" height="16">
      <a href="https://g-airborne.com/bringing-your-deep-learning-model-to-production-with-libtorch-part-3-advanced-libtorch/" target="_blank" rel="noopener nofollow ugc">g-airborne.com</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://g-airborne.com/bringing-your-deep-learning-model-to-production-with-libtorch-part-3-advanced-libtorch/" target="_blank" rel="noopener nofollow ugc">Garry's Blog - Advanced libtorch</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Can I instantiate layers in forward function?

Jingnan_Jia: <p>Normally we use pytorch like the following code.</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.layer1 = nn.Conv2d(3, 32, 3)
    self.layer2 = nn.Conv2d(32, 64, 3)
    self.layer3 = nn.Conv2d(64, 2, 3)
  def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.layer3(out)
    return out
</code></pre>
<p>My question is:<br>
Do I have to instantiate layers in init() function? Are the following 2 code segments also okay?<br>
Code_1</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.layer1 = nn.Conv2d(3, 32, 3)

  def forward(self, x):
    out = self.layer1(x)

    self.layer2 = nn.Conv2d(32, 64, 3)
    self.layer3 = nn.Conv2d(64, 2, 3)
    out = self.layer2(out)
    out = self.layer3(out)
    return out
</code></pre>
<p>Code_2</p>
<pre><code class="lang-auto">layer2 = nn.Conv2d(32, 64, 3)
layer3 = nn.Conv2d(64, 2, 3)

class MyNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.layer1 = nn.Conv2d(3, 32, 3)

  def forward(self, x):
    out = self.layer1(x)

    out = layer2(out)
    out = layer3(out)
    return out
</code></pre>

Majid_Al: <p>Pytorch uses that template to instantiate layers to keep track of functions and their derivatives in forward and backward pass. You can check out how those inner-workings come together in the source code. Also look into “tensorboard” to visualize the graph that you build and how it is updated in each forward and backward pass.<br>
For example in your “Code_1”, you have instantiated 2 of the convolution layers in the forward method. This results in generating 2 new convolution layers in each forward pass. Therefore losing track of parameters you intend to update. These update rules and derivations are defined in the nn.Module class that we inherit from each time we create a new class.</p>
<p>In the secod case, The layers are not defined within the class, thus they don’t inherit from the nn.Module class. So when you create your optimizer and pass the network parameters to be updated: <code>optimizer = Adam(MyNet.parameters(), learning_rate=0.001)</code>, those weights for convolutions outside the network don’t get passed to the optimizer.</p>
Multiple calls to autograd.grad with same graph increases memory usage

Brian_Davis: <p>I’m trying to generate multiple saliency maps for a model with multiple outputs and a single input image. I’m calling <code>torch.autograd.grad</code> once for each of the outputs as I want seperate gradients for each output.</p>
<p>My problem is that when I call <code>torch.autograd.grad</code> multiple times, each call increases the GPU memory usage. Here’s a code snippit:</p>
<pre><code>    input_gradients = []
    for output_scalar in multiple_output_scalars:
        self.model.zero_grad() #not needed right?
        input_gradients.append( torch.autograd.grad(
            outputs = output_scalar, 
            inputs = x, 
            retain_graph=True,
            create_graph=False)[0].cpu().detach() )
    input_gradients = torch.cat(input_gradients,dim=0)
</code></pre>
<p>I probably misunderstand what auto grad is doing, but to my understanding it should be using the computation graph to compute the gradiens and returning the gradients. Nothing new should be stored and since the graph is retained, nothing needs rebuilt. But clearly I’m wrong. Where is my thinking flawed and what is the right way to do this?</p>

albanD: <p>Hi,</p>
<p>Since you append the result to a list at every iteration, it is expected that it uses more and more memory no?<br>
Also no need to zero_grad when using autograd.grad indeed.</p>

Brian_Davis: <p>Sorry, I should clarify. It’s GPU memory that’s growing, while I store the gradients on the CPU.</p>

albanD: <p>Do you actually run out of memory? It might just be the allocator that takes more and more from the driver but does not actually use it.</p>

Brian_Davis: <p>I do run out of GPU memory</p>

albanD: <p>Ok,</p>
<p>Could you provide a small code snippet that we could run (on colab for example) that shows the issue please?</p>

Brian_Davis: <p>So, a simple model doesn’t reproduce the increase of memory at each call. The model that is causing the problem is quite complicated. It uses a CNN detector, hooks that store features which are ROIAligned later, and a several graph network calls. Any ideas on what in a model could cause incease in memory with the autograd call?</p>

Brian_Davis: <p>So removing the graph network stuff doesn’t change the memory error. This colab is my attempt to reproduce it, but it runs without any memory incrase: <a href="https://colab.research.google.com/drive/1IiVFdB2hUttfgrpgwCnOXYmxKN1-43S0?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1IiVFdB2hUttfgrpgwCnOXYmxKN1-43S0?usp=sharing</a></p>

albanD: <p>Hard to say without code.<br>
But of your hooks store stuff. That sounds dangerous <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

Brian_Davis: <p>There are only forward hooks, and they’d overwrite the last stored data.</p>

Brian_Davis: <p>Okay, nevermind. There was an extra backwards hook being added in the saliency code I copied. Clearing the data fixed the memory issue.<br>
Thanks for your help!</p>
What happens if I use a pretrained model along with a not pretraiend model?

Bled_Clement: <p>Hi folks,</p>
<p>I have an idea in my head and I’ve done a few experiments but it never works out. I was wondering if someone more experienced here would be able to chime in.</p>
<p>Let’s say I use a pre-trained ResNet to denoise an image. I’m using resnet as the encoder of an autoencoder and it works fine. Then let’s say I want to try to add a new network after this autoencoder by simply adding more convolutions and expanding the tensor again. We go from:</p>
<ul>
<li>Noisy_image -&gt; autoencoder -&gt; output_clean -&gt; loss</li>
</ul>
<p>to</p>
<ul>
<li>Noisy_image -&gt; autoencoder -&gt; small_network2 -&gt; output_clean -&gt; loss</li>
</ul>
<p>In every experiment I’ve ran, the performance is worse with this added network. What happens more often that not is that the autoencoder does all the work and the following networks just work to not get in the way.</p>
<p>I was wondering if the fact that Resnet is pretrained reduces the chance for other parts of the network to “learn”? If so I think I would benefit from learning from scratch but this would be time consuming.</p>

suraj.pt: <blockquote>
<p>Resnet is pretrained reduces the chance for other parts of the network to “learn”?</p>
</blockquote>
<p>I don’t think this is the case. When you train, do you freeze the Resnet and only update parameters in small_network2?</p>
Training and exporting model weights in float16

rharang: <p>I’d like to quantize my model weights to 16 bits for speed/memory savings in deployment.  The torch.cuda.AMP package – which appears to be the strong recommendation for training acceleration – returns model weights as 32 bit floats which appear to require a full 32 bits of precision to represent in model saving and loading (additionally, casting them to float16s for inference leads to performance loss).  The built-in torch.quantization tools appear to be limited to int8 outputs.  Simply calling ‘.half()’ on my model/inputs and attempting to train as normal has the stability issues you might expect. While the documentation is a bit ambiguous about being able to explicitly set the types of particular layers when using AMP, trying to set specific Linear layers to float16 results in the same GradScaler errors as reported elsewhere.</p>
<p>Is there a recommended best practice for training a model ultimately destined for export in 16-bit floating point format?</p>
In GAN training, getting RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

ruchit_1: <p>I’m training a GAN, but getting the following error. Can someone explain me why this is happening, and how to resolve it?</p>
<pre><code class="lang-auto">############################
        # (1) Update D network: maximize D(x)-1-D(G(z))
        ###########################
        real_img = torch.Tensor(target)
        if torch.cuda.is_available():
            real_img = real_img.cuda()
        z = torch.Tensor(data)
        if torch.cuda.is_available():
            z = z.cuda()
        fake_img = netG(z)

        netD.zero_grad()
        real_out_1 = netD(real_img)
        real_out = torch.mean(real_out_1)
        fake_out_1 = netD(fake_img)
        fake_out = torch.mean(fake_out_1)

        d_loss = -torch.log(real_out_1) - torch.log(1 - fake_out_1)
        d_loss = torch.mean(d_loss)
        d_loss.backward(retain_graph=True)
        optimizerD.step()
        

        ############################
        # (2) Update G network: minimize 1-D(G(z)) + Perception Loss + Image Loss
        ###########################
        netG.zero_grad()
        g_loss = generator_criterion(fake_out, fake_img, real_img)
        g_loss.backward()
        
        fake_img = netG(z)
        fake_out = netD(fake_img).mean()
        
        optimizerG.step()

</code></pre>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
119         netG.zero_grad()<br>
120         g_loss = generator_criterion(fake_out, fake_img, real_img)<br>
–&gt; 121         g_loss.backward()<br>
122<br>
123         fake_img = netG(z)</p>
<p>1 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
219                 retain_graph=retain_graph,<br>
220                 create_graph=create_graph)<br>
–&gt; 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
222<br>
223     def register_hook(self, hook):</p>
<p>/usr/local/lib/python3.6/dist-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
130     Variable.<em>execution_engine.run_backward(<br>
131         tensors, grad_tensors</em>, retain_graph, create_graph,<br>
–&gt; 132         allow_unreachable=True)  # allow_unreachable flag<br>
133<br>
134</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 1024, 1, 1]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<pre><code class="lang-auto"></code></pre>

ruchit_1: <p>Found solution :</p>
<p>All I needed to do was :</p>
<pre><code class="lang-auto">fake_out_1 = netD(fake_img.detach())
</code></pre>
<p>Correct me if I was wrong!</p>
Searching nested GPU tensors in a custom object

t_naumenko: <p>Hello Everyone,<br>
I am trying to build a system, which should optimize GPU memory utilization during training process. In order to deal with custom user models, I need an estimator of GPU memory for a custom object. A simple example of such an object would be:</p>
<pre><code class="lang-auto">import torch
from argparse import Namespace

x = torch.zeros(1024).cuda()
y = torch.ones([2, 40]).cuda()

tmp_namespace = Namespace()
tmp_namespace.find_me = x

object_to_check = {'a': [1, 2, {'y': y}], 'b': (6, tmp_namespace, (y, y))}
</code></pre>
<p>I want to write a function, which will tell me, what is the GPU memory usage for this object.</p>
<p>Some observations I already made:</p>
<ul>
<li>torch.cuda.memory_allocated() might help here. I tried this way:</li>
</ul>
<pre><code class="lang-auto">def object_gpu_memory(obj):

    start_memory = torch.cuda.memory_allocated()
    with torch.no_grad():
        tmp = copy.deepcopy(obj)

    end_memory = torch.cuda.memory_allocated()
    del tmp

    return end_memory - start_memory
</code></pre>
<p>but, sadly, it gives me</p>
<pre><code class="lang-auto">RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
</code></pre>
<p>for my use case. I want to support intermediate tensors as well.</p>
<ul>
<li>Python garbage collector may help to get all Tensor objects, but I did not figure it out how to apply it here.</li>
<li>Recursive traversion of a typical object with dir() method falls into infinite recursion.</li>
</ul>
<p>Is there a way to create such function? If so, could you give some hints how to create it?<br>
Thanks in advance.</p>
Do I have to inherit nn.Module?

Jingnan_Jia: <p>I have 2 networks with some common layers. So is the following code correct?</p>
<pre><code class="lang-auto">class CommonNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.conv1 = nn.Conv2d(3, 32, 3)
    self.conv2 = nn.Conv2d(32, 64, 3)

class Net1(CommonNet):
  def __init__(self):
    super().__init__()
    self.final_conv = nn.Conv2d(64, 4, 3)
  def forward(self, x):
    out = self.conv1(x)
    out = self.covn2(out)
    out = self.final_conv(out)
    return out

class Net2(CommonNet):
  def __init__(self):
    super().__init__()
    self.final_conv = nn.Conv2d(64, 2, 3)  
  def forward(self, x):
    out = self.conv1(x)
    out = self.covn2(out)
    out = self.final_conv(out)
    return out
</code></pre>

JuanFMontesinos: <p>You need to call super in commont net but the code is correct (there are typos).</p>
Can I change the order of layers in __init__()?

Jingnan_Jia: <p>Normally we use pytorch like the following code.</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.layer1 = nn.Conv2d(3, 32, 3)
    self.layer2 = nn.Conv2d(32, 64, 3)
    self.layer3 = nn.Conv2d(64, 2, 3)
  def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.layer3(out)
    return out
</code></pre>
<p>My question is: is the order of layers in <strong>init</strong>() function matter?<br>
Is the following code right?</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.layer3 = nn.Conv2d(64, 2, 3)
    self.layer2 = nn.Conv2d(32, 64, 3)
    self.layer1 = nn.Conv2d(3, 32, 3)
  def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.layer3(out)
    return out
</code></pre>

JuanFMontesinos: <p>It doesn’t matter. forward is which defines the real order. The only difference is they will be displayed differently when you print the model or save it.</p>
Efficient tensor construction from other tensors

Yann: <p>Hello,</p>
<p>I am trying to perform a complex operation for which my implementation is very inefficient right now.</p>
<p>I have a list of tuples of the same number of tensors of shape (batch, …) on the one hand, and a tensor of shape (batch, 1) representing indexes in the list.</p>
<p>for instance with a batch size of 4:</p>
<pre><code class="lang-python">l = [(t11, t12),(t21, t22),(t31, t32)]  # list of tuples of tensors
</code></pre>
<p>and</p>
<pre><code class="lang-python">idx = [0,2,1,1]   # index along batch dimension, pointing to positions in the list
</code></pre>
<p>I want to create a tuple of tensors like (ti1, ti2), but where the components of ti1 and ti2 along the batch dimension are picked from corresponding tensors according to the index.</p>
<p>The solution I came up with is this one:</p>
<pre><code class="lang-python">mod_tuple = tuple((torch.stack([l[i][itup][ibatch] for ibatch, i in enumerate(idx)]) for itup in range(tuple_size)))
</code></pre>
<p>However with a batch size of 128 it takes a veeeeery long time.<br>
Could you help me find an efficient implementation please?</p>
<p>Thanks in advance !</p>

ptrblck: <p>Would it be possible to create a tensor of <code>l</code> and directly index it e.g. via <code>gather</code> or are all <code>t**</code> tensors differently shaped?</p>

Yann: <p>Thanks for the answer. The t1* tensors are possibly differently shaped. Actually (t11, t12) is an observation, the list is a trajectory of observations, and what I try to do is to build a new observation by picking a different point in the trajectory for each element of the batch because only one observation is interesting in each trajectory. Then I can do a single forward pass in my model.</p>
<p>I guess it will be more efficient to do one forward pass per element of the trajectory, and then pick the relevant results instead of what I am trying to do here, I will try that.</p>