akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Sharing optimizer between processes

username: delgado
cooked: <p>I am wondering if it is possible to share optimizer between different threads. To be specific, when optimizer.step() is applied, modified state of the optimizer should be available for all processes.</p>

username: smth
cooked: <p>i cant think of a way to do this, because the optimizer also has python scalars.</p>
<p>What are you trying to do?</p>

username: delgado
cooked: <p>I am trying benchmark A3C algo with shared RMSProp optimizer vs seperate one for each thread, as described in <a href="https://arxiv.org/pdf/1602.01783.pdf" rel="nofollow noopener">this paper, p11</a></p>

username: apaszke
cooked: <p>No, I think most of the optimizer state is kept in Tensors, so it should be possible to share it. I’m not 100% now, I’ll need to take a look at RMSprop and can confirm it tomorrow.</p>

username: delgado
cooked: <p>Thanks. I will also look at the code and see if I can find some hack.</p>

username: ypxie
cooked: <p>Glad to know that you were looking into this.<br>
Is the current optimizer thread safe?<br>
Thanks!</p>

username: apaszke
cooked: <p>It’s not. You need to guard it with a mutex yourself</p>

username: mimoralea
cooked: <p>Just a quick follow-up for those wanting to implement A3C.</p>
<p>A3C uses a Hogwild! update style, which means updates are made lock-free. Workers possibly overwrite each other updates at times, but that’s OK.</p>
<p>The only thing needed for Adam or RMSprop to be shared with other processes is override the classes and call the <code>share_memory</code> method on the important variables. One can also cleverly work around the <code>step</code> counter which is not a tensor, so it can’t be <code>shared</code>.</p>
<p>But other than that, it is doable, and not too hard.</p>
Fine tuning FCN_ResNet101

username: samin_hamidi
cooked: <p>Hi everyone,</p>
<p>I want to finetune a FCN_ResNet101. I would like to change the last layer as my dataset has a different number of classes. Also, finetune only the FCN head.<br>
For the former, is it enough to only change the num_classes argument when defining the model or I need to use something like this:</p>
<p>model = torchvision.models.segmentation.fcn_resnet101(pretrained=True)<br>
model.classifier=nn.identity()<br>
model.Conv2d = nn.Conv2d(<br>
in_channels=256,<br>
out_channels=nb_classes,<br>
kernel_size=1,<br>
stride=1<br>
)</p>
<p>I took this piece of code from another thread. I am not sure if it is necessary to use nn.identity(). When I do, the last layer does not change but the last layer of the one to the last FCN!<br>
And, how many layers must be changed so my FCN head is re_trianed?<br>
I wrote it this way but I’m mostly confused about FCN_ResNet101 architecture.</p>
<p>model = torchvision.models.segmentation.fcn_resnet101(pretrained=True, progress=True, num_classes=?)<br>
<span class="hashtag">#model</span>.classifier[4] = nn.Identity()</p>
<p>“”"<br>
FCNHead(<br>
(0): Conv2d(2048, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): Dropout(p=0.1)<br>
(4): Conv2d(512, 21, kernel_size=(1, 1), stride=(1, 1))<br>
), FCNHead(<br>
(0): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU()<br>
(3): Dropout(p=0.1)<br>
(4): Conv2d(256, 21, kernel_size=(1, 1), stride=(1, 1))<br>
)]<br>
“”"</p>
<p><span class="hashtag">#setting</span> our own number of classes</p>
<p>layer_list = list(model.children())[-5:]<br>
model_small = nn.Sequential(*list(model.children()))[-5:]</p>
<p>for param in model_small.parameters():<br>
param.requires_grad = False</p>
<p>model_small.Conv2d = nn.Conv2d( in_channels=1024,kernel_size=(3,3),stride=(1,1))<br>
model_small.BatchNorm2d = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
model_small.ReLU = nn.ReLU()<br>
model_small.Dropout = nn.Dropout(p=0.1)<br>
model_small.Conv2d = nn.Conv2d(<br>
in_channels=256,<br>
out_channels=nb_classes,<br>
kernel_size=1,<br>
stride=1<br>
)<br>
model = model_small.to(device)<br>
Any guidance is very much appreciated!</p>

username: ptrblck
cooked: <p>No, the <code>nn.Identity</code> module is not necessary in your use case, but can be used to extract features.<br>
I would recommend to stick to the <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">Fine tuning tutorial</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Split Perceptual Loss Model and Inference Model over multiple GPUS

username: b4shy
cooked: <p>Hi everybody,</p>
<p>I am trying to implement some inpainting model, where I use the Perceptual Loss.<br>
The training routine works fine on a single GPU, but I run into a CUDA out of Memory Error, when using Multiple GPUS (with same batch size)<br>
This is the code I am using.</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if use_cuda else "cpu")

NET = DeFINe()
vgg16_partial = Vgg16()

if cuda_device_count &gt; 1:
    print("Use", cuda_device_count, "GPUs!")
    NET = torch.nn.DataParallel(NET)
    vgg16_partial = torch.nn.DataParallel(vgg16_partial)

NET.to(device)
vgg16_partial.to(device)
</code></pre>
<p>Is there any example on how to solve this kind of problem?<br>
My goal is to somehow split the models over all the devices.<br>
Best regards and thank you!</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> might use more memory on the default device as described <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">here</a>.<br>
The blog post also mentions some workarounds.<br>
Alternatively, have a look at <code>nn.DistributedDataParallel</code>.</p>
How to register_hook on Conv2d?

username: Umair_Javaid
cooked: <p>How to register_hook on Conv2d?</p>
user_suspended: yes

username: spanev
cooked: <p>Hi <a class="mention" href="/u/umair_javaid">@Umair_Javaid</a>,</p>
<p>Simply create a hook function and register it to the by calling <code>register_forward_hook</code>.</p>
<aside class="quote quote-modified" data-post="2" data-topic="55032">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spanev/48/14892_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/display-intermediate-features/55032/2">Display intermediate features</a> 
  </div>
  <blockquote>
    There are two things to do: get the intermediate features, display them. 
First, uou will need to use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_forward_hook" rel="noopener nofollow ugc">register_forward_hook</a> on the concerned convolution. 
To create it you have to write: 
def hook(module, input, output):
    # output is the [N, 64, 30, 30] Tensor containing your features

model.concerned_conv.register_forward_hook(hook)

Then depending on your display method, you will have. You can use torchvision and Tensorboard to do so. The hook body would be something like: 
def hook(module,…
  </blockquote>
</aside>
Can't shuffle dataset

username: Hika-Kondo
cooked: <p>when I try to shuffle dataset like this,</p>
<p><code>dataloader = torch.utils.data.DataLoader(dataset, batch_size=16, shuffle=True, num_workers=6)</code></p>
<p>I have an error berrow</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "datasets.py", line 41, in &lt;module&gt;
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=16, shuffle=True, num_workers=6)
  File "~/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 213, in __init__
    sampler = RandomSampler(dataset)
  File "~/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 92, in __init__
    if not isinstance(self.num_samples, int) or self.num_samples &lt;= 0:
  File "~/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 100, in num_samples
    return len(self.data_source)
TypeError: 'NoneType' object cannot be interpreted as an integer

</code></pre>

username: Hika-Kondo
cooked: <p>I have a bug in __len__method.</p>
Range of a tensor in C++

username: rincewind
cooked: <p>Hello,</p>
<p>I’m currently writing some C++ code with torch (to eventually make it a PyTorch extension).<br>
My question is rather simple, I have a torch::Tensor, let’s name it “my_tensor” and make it 3 dimensional.<br>
What is the C++ equivalent of Python “my_tensor[0:4, 2:, 3]”, for example?</p>
<p>Thanks!</p>

username: 21fa417e3fb06e56040c
cooked: <p>see <a href="https://discuss.pytorch.org/t/subtensor-add-and-division-in-libtorch/63224" class="inline-onebox">Subtensor add and division in libtorch</a></p>
Libtorch C++ windows 10 CMake

username: zaynabhabibi
cooked: <p>Hi,<br>
I’am trying to  build Libtorch C++ example-app in windows 10 with Cuda 9.0 but I have the following error in cMake :</p>
<p>CMake Error: The following variables are used in this project, but they are set to NOTFOUND.<br>
Please set them or make sure they are set and tested correctly in the CMake files:<br>
torch;caffe2_library;caffe2_gpu_library;C10_LIBRARY-NOTFOUND;C:/Program Files/NVIDIA Corporation/NvToolsExt/lib/x64/nvToolsExt64_1.lib;C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.0/lib/x64/cudart.lib;C10_CUDA_LIBRARY<br>
linked by target “example-app” in directory D:/testlibtorch/example-app</p>
<p>Did anyone solve this problem?<br>
Thank you,</p>

username: 21fa417e3fb06e56040c
cooked: <p>Please tell me more<br>
you see first here <a href="https://pytorch.org/cppdocs/installing.html#minimal-example" rel="nofollow noopener">https://pytorch.org/cppdocs/installing.html#minimal-example</a></p>

username: zaynabhabibi
cooked: <p>Hi,</p>
<p>Yes I see <a href="https://pytorch.org/cppdocs/installing.html#minimal-example" rel="nofollow noopener">https://pytorch.org/cppdocs/installing.html#minimal-example</a></p>
<p>I am on windows 10 and I have Visual Studio 2015 with cMake 3.16.1<br>
I download libtorch for cuda 9.0.<br>
First I had a problem with c10 but I link c10_LIBRARY to libtorch/lib<br>
Now, in cmake I have configure done and generate done.<br>
But when I open on VS esample-app.sln and I build the project I have a link error : impossible to open ‘…\libtorch\share\cmake\Torch.obj’<br>
Effectively, the file doesn’t exist.</p>
<p>Thank you.</p>

username: 21fa417e3fb06e56040c
cooked: <p>I think your compiler version is too low , Check your system<br>
On Windows<br>
At least Visual Studio 2017 Update 3 (version 15.3.3 with the toolset 14.11)  are needed.<br>
you see here <a href="https://github.com/pytorch/pytorch/blob/master/README.md" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/README.md</a></p>
Faster RCNN change backbone

username: Miguel_Campos
cooked: <p>Hello guys</p>
<p>I am trying to change the RESNET50 backbone of Faster RCNN by MobileNET.<br>
My code seems like:</p>
<blockquote>
<p>from torchvision.models.detection import FasterRCNN<br>
backbone = torchvision.models.mobilenet_v2(pretrained=True)<br>
backbone.out_channels = 1280<br>
model = FasterRCNN(backbone)</p>
</blockquote>
<p>But i get this error</p>
<blockquote>
<p>ValueError: num_classes should not be None when box_predictor is not specified</p>
</blockquote>
<p>I am starting with this kind of networks, so there are many things that i am still learning, any help would be great.<br>
The objective that i have is to compare the time inference of resnet 50+faster (default) vs mobilenet+faster, so i dont want either train or fine tuning the network</p>

username: Eta_C
cooked: <p><a href="https://pytorch.org/docs/stable/_modules/torchvision/models/detection/faster_rcnn.html#fasterrcnn_resnet50_fpn" rel="nofollow noopener">See how fasterrcnn_resnet50_fpn use FasterRCNN</a></p>
<pre><code class="lang-python">def fasterrcnn_resnet50_fpn(pretrained=False, progress=True,
                            num_classes=91, pretrained_backbone=True, **kwargs):
    if pretrained:
        pretrained_backbone = False
    backbone = resnet_fpn_backbone('resnet50', pretrained_backbone)
    model = FasterRCNN(backbone, num_classes, **kwargs)
    if pretrained:
        state_dict = load_state_dict_from_url(model_urls['fasterrcnn_resnet50_fpn_coco'],
                                              progress=progress)
        model.load_state_dict(state_dict)
    return model
</code></pre>
<pre><code class="lang-python"># FasterRCNN __init__ function
if num_classes is not None:
    if box_predictor is not None:
        raise ValueError("num_classes should be None when box_predictor is specified")
    else:
        if box_predictor is None:
            raise ValueError("num_classes should not be None when box_predictor "
                                 "is not specified")
</code></pre>
<p>Specify <code>num_classes</code> or <code>box_predictor</code>.</p>

username: Miguel_Campos
cooked: <p>I added this</p>
<blockquote>
<p>model = FasterRCNN(backbone, num_classes=2) # Aircraft + background<br>
…<br>
…<br>
pred = model(img)</p>
</blockquote>
<p>And this is what i get</p>
<blockquote>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight 1280 1280 3 3, but got 2-dimensional input of size [1, 1000] instead</p>
</blockquote>
<p>img has the next shape : (1, 3, 920, 1080)</p>
C++ frondend : at::Tensor const member functions

username: julien-mille
cooked: <p>I am confused by the declarations of member functions performing in-place processing in at::Tensor class (include/ATen/core/TensorBody.h)</p>
<p>These functions (such as resize_, add_ and all functions ending with <em>) do modify the current object, and return it by reference, e.g.<br>
Tensor &amp; add</em>(const Tensor &amp; other, Scalar alpha=1) <strong>const;</strong></p>
<p>They do modify the tensor on which they are called, so they do not convey physical nor logical const-ness, so why have them declared <strong>const</strong> ?</p>
Dice score for prediction shape [1,21,224,224] and target shape [1,224,224]

username: Shashank_Shekhar
cooked: <p>My predictions from the UNet is having shape[1,21,224,224]  but the target masks have shape [1,224,224], how should i merge prediction and use it for dice score calculation.</p>
<p>If i use torch.argmax() to get max prediction/labels for dice score calculation i am getting scores more then 1.</p>
<p>For calculating dice score i am using below function.</p>
<p>def dice (pred,targs):<br>
pred=(pred&gt;0).float()<br>
return 2. <em>(pred</em>targs).sum()/(pred+targs).sum()</p>
API differences of torchvision and torchtext

username: pietz
cooked: <p>Getting started with torchtext, I noticed a few API choices that seem to head into a different direction compared to the torchvision counterpart. I understand that these are very different types of problems based on very different types of data. Still, wouldn’t it be cleaner if the classes had a similar structure? On one hand we have transforms and data loaders on the other we use fields and iterators.</p>
<p>More specifically, chapter 5 of <a href="https://mlexplained.com/2018/02/08/a-comprehensive-tutorial-to-torchtext/" rel="nofollow noopener">this article</a> makes a solid argument of wrapping the iterator into another class so we don’t need to change the code of our torchtext training loop every single time. I believe, this change would essentially create the API of the DataLoader class making this part of the torchtext and torchvision libraries quite similar. I like this. It feels consistent.</p>
<ul>
<li>Are there any particular reasons why the 2 APIs seem to drift apart or did it just happen over time?</li>
<li>Would you agree that a unified API between torchvision and torchtext has its benefits?</li>
<li>Is this something someone is working on?</li>
<li>Am I overlooking something?</li>
</ul>
<p>Thanks,<br>
pietz</p>
Why can’t I use pytorch-GPU？

username: Q_lee
cooked: <p>I installed pytorch and cuda with</p>
<pre><code class="lang-auto">conda install pytorch torchvision cudatoolkit=10.1 
</code></pre>
<p>but when I call <code>torch.cuda.is_available()</code>,it returns False. Why? Is there anything else that should be installed?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9c100fbe398c2ae15f4b78e67825887c6acb7bb.png" alt="111" data-base62-sha1="odIctVqemMwbqfyUtv8FzSvEbXl" width="421" height="59"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/387df3ae9eba2d965a42de9a6d84b05ee30c9ffc.png" alt="2222" data-base62-sha1="83KAw9GkFFhoLswpg4PtArELySU" width="567" height="267"></p>

username: albanD
cooked: <p>Hi,</p>
<p>In nvidia-smi, it states that you have CUDA 10.2.<br>
You might have some conflicts between your libraries.<br>
You want to make sure that you have the cuda version that matches pytorch.</p>
<p>Also you can try to use cuda in pytorch by doing <code>torch.rand(1).cuda()</code> to see if the error message hints at what is not correct.</p>

username: Q_lee
cooked: <p>Thanks for the reply.<br>
I re-installed NVIDIA-driver version 10.1, and I checked all versions.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e1d717cd28fadbce05e28412e405df6c18474fc.png" alt="%E5%BE%AE%E4%BF%A1%E6%88%AA%E5%9B%BE_20191220115523" data-base62-sha1="8RuGtNjpNvsEhs4MoJ9wGgqOlNW" width="590" height="427"><br>
But It still doesn’t work.<img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Could you post the log of the <code>conda install</code> please?<br>
The local CUDA installation should not be a problem, as the binaries ship with their own CUDA and cudnn libs.</p>
The substraction of different size Tensor

username: Weng_zhiqiang
cooked: <p>Hi everyone,<br>
I’m a little confused with the substraction of the Tensor with different size, e.g. the Tensor(1, 3, 2) - Tensor(3, 1, 2).</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

tensor1 = torch.Tensor([[1, 2], [1, 2], [1, 2]])
tensor2 = torch.Tensor([[1, 1], [1, 1], [1, 1]])
a = tensor1.unsqueeze(0)
b = tensor2.unsqueeze(1)
result = a - b
</code></pre>
<p>After I excuted this code, the result says its size is (3, 3, 2), and the output of the substraction is just like each row of <strong>a</strong> minus the each dimension of <strong>b</strong>. However, I can’t undestand how this work. The <strong>a</strong>, in my mind, means the flat in the 3-d space. But the <strong>b</strong> is not the flat in this 3-d space. So, how could these two tensor work together?<br>
Sorry about this stupid question.</p>

username: Eta_C
cooked: <p>Hi, it is called broadcast.</p>
<p><a href="https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html" rel="noopener nofollow ugc">Here</a> is Numpy broadcast.<br>
<a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="noopener nofollow ugc">Here</a> is PyTorch broadcast.</p>
<blockquote>
<p>Two tensors are “broadcastable” if the following rules hold:</p>
<ul>
<li>Each tensor has at least one dimension.</li>
<li>When iterating over the dimension sizes, starting at the trailing dimension, the dimension sizes must either be equal, one of them is 1, or one of them does not exist.</li>
</ul>
</blockquote>

username: Weng_zhiqiang
cooked: <p>Hi Eta_C,<br>
Thanks for your reply. Maybe I should look through the tutorials first. Thanks for your helping. Having a good day~</p>
Windows Build Error C2338

username: cdhernandez
cooked: <p>I’ve been trying to set up pytorch on my pc with an old GPU for a while. Some previous details here: <a href="https://github.com/pytorch/pytorch/issues/31180" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/31180</a></p>
<p>I have updated visual studio to 2019 and now get the a C2338 error as in the log here: <a href="https://www.dropbox.com/s/6ew4sshbgb7hfuw/build2019.log?dl=0" rel="nofollow noopener">https://www.dropbox.com/s/6ew4sshbgb7hfuw/build2019.log?dl=0</a></p>
<p>The main links i found for this error are the pytorch discussion topic 31484 and the pytorch github topic 9122 (can only include 2 links apparently). The first is unhelpful while the second says to use vsx64 shell but i’m doing this through anaconda so I’ve been using an anaconda terminal i’m not sure if its relevant or how i would be able to install to anaconda using the vsx64 shell.</p>
<p>Not sure where to do from here, i’m kind of flailing around so any help would be appreciated. Note: env details included below.</p>
<p><em>Collecting environment information…</em><br>
<em>PyTorch version: N/A</em><br>
<em>Is debug build: N/A</em><br>
<em>CUDA used to build PyTorch: N/A</em></p>
<p><em>OS: Microsoft Windows 7 Professional</em><br>
<em>GCC version: Could not collect</em><br>
<em>CMake version: version 3.14.0</em></p>
<p><em>Python version: 3.7</em><br>
<em>Is CUDA available: N/A</em><br>
<em>CUDA runtime version: 10.1.243</em><br>
<em>GPU models and configuration: GPU 0: GeForce GTX 770M</em><br>
<em>Nvidia driver version: 425.31</em><br>
<em>cuDNN version: E:\python\Cuda\bin\cudnn64_7.dll</em></p>
<p><em>Versions of relevant libraries:</em><br>
<em>[pip] numpy==1.16.5</em><br>
<em>[pip] numpydoc==0.9.1</em><br>
*[conda] blas                      1.0                         mkl  *<br>
*[conda] mkl                       2019.4                      245  *<br>
*[conda] mkl-include               2019.4                      245  *<br>
*[conda] mkl-service               2.3.0            py37hb782905_0  *<br>
*[conda] mkl_fft                   1.0.14           py37h14836fe_0  *<br>
<em>[conda] mkl_random                1.1.0            py37h675688f_0</em></p>

username: peterjc123
cooked: <p>You’ll need this PR: <a href="https://github.com/pytorch/pytorch/pull/30959" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/30959</a>. For some reason, I cannot land it to master.</p>

username: cdhernandez
cooked: <p>That fixed it thank you for all your help.</p>
<p>Keywords: error C2338: CopyToCPU requires fundamental types</p>
<p>I went to the github commit linked above which is a branch of the peterjc123/pytorch repo. I cloned that repo and use the command checkout branch is_fundamental_fix (you can see all branches with git branch -a). You can verify that you are using the correct branch by checking that the changes in <a href="https://github.com/pytorch/pytorch/pull/30959/files" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/30959/files</a> are consistent with what you have on your machine. From there I did the normal pytorch install from source commands.</p>
Do operations like torch.bmm and torch.ger faster on gpu rather than cpu?

username: gebrahimi
cooked: <p>I need to do some operations on tensors like torch.bmm, torch.eig and torch.ger. Somehow if i convert the tensors to cpu and do the calculations, it works faster. Is it expected?</p>

username: richard
cooked: <p>There are generally a set of cases where running on CPU is faster than running on GPU: one common case where this happens if the input sizes are small. How large are your inputs?</p>

username: gebrahimi
cooked: <p>I am doing some operations on batch size 100 and tensors of size (784, 300), (300, 100), (100, 10). but mostly i work on the average overall the batches. so i use torch.mean to take the average of tensors on the batch and then use torch.bmm and torch.ger and torch.eig on tensors of size (784, 300), (300, 100), (100, 10).</p>
What's the difference between torch.nn.Module and torch.jit.ScriptModule?

username: dalalaa
cooked: <p>The model inherited from <code>torch.nn.Module</code> also can be converted to TorchScript, in which case should we use <code>torch.jit.ScriptModule</code> instead of <code>torch.nn.Module</code>?</p>

username: driazati
cooked: <p>You shouldn’t inherit from <code>torch.jit.ScriptModule</code>, that was an old API that we replaced in v.1.2.0</p>
<p>See <a href="https://pytorch.org/docs/master/jit.html#migrating-to-pytorch-1-2-recursive-scripting-api">here</a> for details</p>
Does pytorch supports int8 F.conv2d?

username: Marat
cooked: <p>Following:</p>
<pre><code class="lang-auto">import torch.nn.functional as F
import torch
filters = torch.randint(0, 2, (8,4,3,3)).type(torch.CharTensor).cuda()
inputs  = torch.randint(0, 2, (1,4,5,5)).type(torch.CharTensor).cuda()
r = F.conv2d(inputs, filters, padding=1)
print(r)
</code></pre>
<p>Gives:<br>
<code>RuntimeError: getCudnnDataType() not supported for Char</code></p>

username: Mostafa_Elhoushi
cooked: <p>I am getting the same problem for int32</p>

username: Mostafa_Elhoushi
cooked: <p>Here they’re saying PyTorch doesn’t yet support int8: <a href="https://github.com/pytorch/pytorch/issues/26274" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/26274</a></p>

username: Marat
cooked: <p>As I currently know if you need int8 for scoring you may use TensorRT it will “optimally” convert your fp32 version to int8 one</p>

username: Mostafa_Elhoushi
cooked: <p>I attended a talk by Facebook at NeurIPS and they mentioned that they support int64 rather than int8. (The goal of supporting int64 was to enable <a href="https://ai.facebook.com/blog/crypten-a-new-research-tool-for-secure-machine-learning-with-pytorch/" rel="nofollow noopener">cryptoTensors</a>, but you can still use it for regular tensors or models)</p>

username: Mostafa_Elhoushi
cooked: <p>and the speaker also mentioned that the PyTorch team developed FBGEMM (Facebook GEneral Matrix Mmultiply) library I recall him saying it supports integer convolution but not sure if it’s integrated into the PyTorch’s framework:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/FBGEMM" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/FBGEMM" target="_blank" rel="nofollow noopener">pytorch/FBGEMM</a></h3>

<p>FB (Facebook) + GEMM (General Matrix-Matrix Multiplication) - https://code.fb.com/ml-applications/fbgemm/ - pytorch/FBGEMM</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Can I cycle through multiple baseline models？

username: mp5088643
cooked: <p>I have multiple models, and I want to train one by one in the same program, but I find that this is not the same as the result of training one model at a time. I del the model variables, but still have an impact on the new model of the next training<br>
such as<br>
modellist = [LSTM,GRU,CNN]<br>
for model in  modellist：<br>
train model<br>
test model<br>
But，In this case, the model loss will be very different from the model trained separately under the same random seed</p>

username: jmandivarapu1
cooked: <aside class="quote no-group" data-username="mp5088643" data-post="1" data-topic="64541">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/c2a13f/48.png" class="avatar"> mp5088643:</div>
<blockquote>
<p>nt to train one by one in the same program, but I find that this is not the same as the result of training on</p>
</blockquote>
</aside>
<p>Create a dictionary instead of list. Then you can iterate over the keys and access the models and train them</p>
<pre><code class="lang-auto">{'lstm';:LSTM(),'cnn':CNN}```</code></pre>
Select unindexed rows

username: Minseok
cooked: <p>I would like to select the rows which are not indexed by a indexing list, such as:</p>
<p>x = torch.randn(10)<br>
indexing_list = torch.LongTensor([2,5])<br>
x[-indexing_list] # What I want to do: remove out rows which are not indexed, thus there remains 8 items</p>
<p>anyone has a good idea?</p>
<p>BTW, is it faster to use LongTensor as an indexing list or maybe just same as np.array and primitive list?</p>

username: LeviViana
cooked: <p>Not knowing precisely the use case, I’d suggest something like: <em>(i)</em> transform your undesired indexes in a set, <em>(ii)</em> construct a list of the complement of your indexes.</p>
<pre><code class="lang-auto"> python -m timeit --setup="import torch;x = torch.rand(10 ** 5);not_idx = set(torch.randint(10 ** 5, (10 ** 5,)).tolist())" "x[[e for e in range(10 ** 5) if e not in not_idx]]"
</code></pre>
<p>The step <code>e not in not_idx</code> is very fast in python.</p>

username: Eta_C
cooked: <p>Do you mean</p>
<pre><code class="lang-python">import torch

x = torch.randn(10)
indexing_list = [2,5]
index = torch.ones(x.shape[0], dtype=bool)
index[indexing_list] = False
selected_x = x[index]
print(selected_x)
</code></pre>

username: LeviViana
cooked: <p>Indeed, this is faster than what I’ve suggested:</p>
<pre><code class="lang-auto">python -m timeit --setup="import torch;x = torch.rand(10 ** 5);not_idx = torch.randint(10 ** 5, (10 ** 5,))" "idx = torch.ones(10 ** 5).bool(); idx[not_idx] = False; x[idx]"
</code></pre>
Validation phase output values too high

username: uertenli
cooked: <p>Hi!</p>
<p>I am currently fine tuning a pretrained DenseNet model from torchvision to action recognition task.</p>
<p>In training, model outputs values in the range [-10,10], however, when I pass to the validation phase, the outputs from the last layer of the model explode to a range [-10000, 10000]. This is just the input passed to network and the outputs are from the last fully connected layer.</p>
<p>I use the same loop for training and validation.</p>
<p>What may be the cause of this ?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Is the training data from the same distribution as the validation data and are you using the same preprocessing?<br>
Do you have batchnorm layers and how large are your training batches?</p>

username: uertenli
cooked: <p>I am using the Mini-Kinetics-200 dataset, so, the data are coming from the same distribution. I do the same rescaling on all frames of all of the videos, that is, rescaling them to 224 x 224. Batchnorm layers are present in the DenseNet-121 model from torchvision. I use batch size of 1, to simulate online training.</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>Are you using a batch size of 1 during training or just validation?<br>
In the former case, I would recommend to remove the batch norm layers, as single samples will most likely create invalid running estimates, which would explain the bad validation accuracy.</p>
<p>You could try to use e.g. <code>nn.InstanceNorm</code> layers instead.</p>

username: uertenli
cooked: <p>Thanks for the reply. I am using BN for both training and validation and will change them.</p>
ResNet 50, different input size

username: Giuseppe
cooked: <p>Hi guys, I would pass to a resNet50 pretrained the batch of dimension (16x9x224x224).<br>
16 is the number of batch, 9 is the channels of my image and 224 x 224 is the width and height respectively.<br>
How can I change the input dimension?<br>
So that only change of the net is the channels of images in input.<br>
Sorry for my bad English <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: kl_divergence
cooked: <p>With the adaptive average pool layer, resnet from torchvision can work with custom input dimensions now. Can you try posting the error if you’re encountering it ?</p>

username: Eta_C
cooked: <pre><code class="lang-python">from torchvision.models import resnet50

net = resnet50()
net.conv1.in_channels = 9
print(net)
</code></pre>
Ignore or punish illegal moves?

username: SUMMERSON
cooked: <p>Hello.</p>
<p>I’m training a bot to play a specific game. Some actions are considered illegal at particular states. I’m currently ignoring the illegal moves and selecting the second (or third if the second is illegal as well) highest q-value from outputted q-values for each action.</p>
<p>I was being paranoid about the efficiency of this method and decided to take a look at what other people do. I came across this article: <a href="https://medium.com/@carsten.friedrich/part-4-neural-network-q-learning-a-tic-tac-toe-player-that-learns-kind-of-2090ca4798d" rel="nofollow noopener">https://medium.com/@carsten.friedrich/part-4-neural-network-q-learning-a-tic-tac-toe-player-that-learns-kind-of-2090ca4798d</a></p>
<p>He is doing the same thing and justifies this method by saying "We will be ignoring the fact that for a particular board state some positions would already be taken and no longer be an option. The player will deal with this when choosing a move and ignore illegal moves no matter what their Q values are. That is, we do not try to teach the Neural Network what moves are legal or not. Again, general advice you find is that this is the better approach "</p>
<p>Relevant code of mine looks like following for now:</p>
<pre><code>#Exploit
	   else:
		   with torch.no_grad():
			   tensor_from_net = policy_net(state).to(self.device)
			   while (True):
				   max_index = tensor_from_net.argmax()
				#If illegal move is given as output by the model, punish that action and make it select an action again.
				if max_index.item() not in available_actions:
					tensor_from_net[max_index] = torch.tensor(-100)
				else:
					break
			return max_index.unsqueeze_(0)
</code></pre>
<p>That “punishment” in the while loop is temporary though, it is not affecting the net.</p>
<ol>
<li>Which approach is better? Ignore the illegal moves or punish the illegal moves?</li>
<li>If ignoring the illegal moves is better, is my approach okay? What can be done to enhance it? And if you think punishing the illegal moves is better, I’d be appreciated if you give me a hint on how to implement it.</li>
</ol>
OSError: image file is truncated (150 bytes not processed)

username: Mamsheikh
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e8a83ad924df3c910e9d2be67bf2c89f8d8f2a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e8a83ad924df3c910e9d2be67bf2c89f8d8f2a6" title="error"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e8a83ad924df3c910e9d2be67bf2c89f8d8f2a6.png" alt="error" data-base62-sha1="8VgmRMKA805pj4yPsxlCWjteorY" width="690" height="270" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">915×359 12.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p><code>PIL</code> throws this error, if the loaded image file is truncated.<br>
You could try to isolate this file and remove it or use</p>
<pre><code class="lang-python">from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True
</code></pre>
<p>to load this file nevertheless.</p>

username: Mamsheikh
cooked: <p>It is working, thanks for the tip</p>
L1_loss error RuntimeError: cudaEventSynchronize in future::wait: device-side assert triggered

username: mysql403
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4aa5a83abcb024a90990c79840e85d469d33347a.png" data-download-href="https://discuss.pytorch.org/uploads/default/4aa5a83abcb024a90990c79840e85d469d33347a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aa5a83abcb024a90990c79840e85d469d33347a_2_690x398.png" alt="image" data-base62-sha1="aEmfBceEoACE2HCB8etQ6oRQN6i" width="690" height="398" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aa5a83abcb024a90990c79840e85d469d33347a_2_690x398.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aa5a83abcb024a90990c79840e85d469d33347a_2_1035x597.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4aa5a83abcb024a90990c79840e85d469d33347a_2_1380x796.png 2x" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1400×809 145 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It suddenly show this error, and i have check the shape of l1loss parameters, anyone helps? <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
my code as follows:</p>
<pre><code class="lang-auto">
class RegL1Loss(nn.Module):
    def __init__(self):
        super(RegL1Loss, self).__init__()

    def forward(self, output, mask, ind, target):
        try:
            pred = _tranpose_and_gather_feat(output, ind)
            mask = mask.unsqueeze(2).expand_as(pred).float()

            loss = F.l1_loss(pred * mask, target * mask, size_average=False)
        except:
            print('pred', pred)
            print('mask:', mask)
            print('target:', target)
            print(pred.shape, mask.shape, target.shape)
            temp1 = pred*mask
            temp2 = target*mask
            print(temp1, temp2)
            print(temp1.shape, temp2.shape)
        # except:
        #     print('output shape:', output.shape, 'ind shape:', ind.shape, 'mask shape:', mask.shape)
        #     print('ind：', ind)
        #     print('output:', output)
        #     assert 1==2
        loss = loss / (mask.sum() + 1e-4)
        return loss

</code></pre>

username: ptrblck
cooked: <p>The error points to an invalid index in <code>gather</code>, most likely thrown by <code>_transpose_and_gather_feat</code>.</p>
<pre><code class="lang-python">Assertion `indexValue &gt;= 0 &amp;&amp; indexValue &lt; stc.sizes[dim]` failed
</code></pre>

username: mysql403
cooked: <p>Does you mean the gather may out of the boundary? I will check it, thanks very much</p>

username: mysql403
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you very much, and i have solved this problem. As you said, i found an out of boundary issue in _transpose_and_gather_feat function, and fix it, thanks again:grinning:</p>
Help with implementing semantic segmentation in three ways

username: Shashank_Shekhar
cooked: <p>If Semantic segmentation is implemented in below three ways using <strong>BCEWithLogitsLoss</strong> and <strong>Dice score metric</strong>, will it give same outputs?<br>
Assume Input Image Dimension <strong>[Batch,Channels,height,width]</strong> &amp; <strong>number of classes is 21(VOC)</strong>.</p>
<p>a) Converting the labels into one hot encoding where target label Dimension is [Batch,Num_classes,height,width].<br>
b) Converting the labels into categorical values where target label Dimension is [Batch,1,height,width]. <strong>implemented repo</strong> <a href="https://github.com/mapillary/inplace_abn" rel="nofollow noopener">https://github.com/mapillary/inplace_abn</a><br>
c) Converting the labels into RGB values where target label Dimension is [Batch,3,height,width] <strong>implemented repo</strong> <a href="https://github.com/jfzhang95/pytorch-deeplab-xception" rel="nofollow noopener">https://github.com/jfzhang95/pytorch-deeplab-xception</a></p>
<p><strong>For all this methods whether prediction output shape should be [Batch,Channels,height,width] ?</strong></p>
To(device('cuda:1')) still leake memory to cuda:0

username: assyl_coop
cooked: <p>I have model trained on cuda:0 pytorch 0.4.<br>
But when I try to reload model on cuda:1 pytorch 1.2 its still shared betweed 0.1 gpus.<br>
I have tried map_location to map from cuda to cpu, from cuda:0 to cuda:1.<br>
Nothing works yet.</p>

username: ptrblck
cooked: <p>Are you sure the model is still on <code>cuda:0</code>, i.e. are you seeing the same amount of used memory on both GPUs?<br>
The CUDA context might have been created on GPU0, which will take some memory.</p>
<p>If you don’t want to use GPU0 at all, you could just mask it via:</p>
<pre><code class="lang-python">CUDA_VISIBLE_DEVICES=1 python scripy.py
</code></pre>
<p>Note that GPU1 will be mapped to <code>cuda:0</code> in that case.</p>
How adversarial autoencoders achieve predictions

username: Foolish
cooked: <p>acc=total_correct/total_num</p>
<p>How does this return the predicted value？</p>

username: ptrblck
cooked: <p>This line of code does not return the predicted value, but calculates the accuracy.<br>
Could you explain your use case and issue a bit?</p>

username: Foolish
cooked: <p>x=x.view(x.size(0),3<em>64</em>64)<br>
out=autoencoder(x)<br>
pred=out.argmax(dim=1)<br>
correct=pred.eq(y).sum().float().item()<br>
total_correct+=correct<br>
total_num=len(test_loader.dataset)<br>
acc=total_correct/total_num</p>
<p>I still use the onehot encoding to return the predicted value, but this is wrong, because I am using an autoencoder, so I don’t know how to return the predicted value.</p>

username: Foolish
cooked: <p>x=x.view(x.size(0),3 <em>64</em> 64)<br>
out=autoencoder(x)<br>
pred=out.argmax(dim=1)<br>
correct=pred.eq(y).sum().float().item()<br>
total_correct+=correct<br>
total_num=len(test_loader.dataset)<br>
acc=total_correct/total_num</p>
<p>I still use the onehot encoding to return the predicted value, but this is wrong, because I am using an autoencoder, so I don’t know how to return the predicted value.</p>

username: ptrblck
cooked: <p>If you would like to reproduce the input, <code>out</code> should already be the prediction.<br>
I’m not familiar with your use case, so could you explain, what kind of input you are trying to reconstruct?<br>
Currently, if seems as if you have some labels in your input, as you are using <code>argmax</code> on your output.</p>
How to calculate mAP after every epoch?

username: Beinan_Wang
cooked: <p>Most python scripts that calculate mAP like the ones in py faster-cnn and detectron2 require writing intermediate results,  Is there any correctly implemented code to calculate mAP that doesn’t require writing detections to files first?</p>

username: ppwwyyxx
cooked: <p>detectron2 does not write intermediate results to files to compute mAP.</p>
Update input without updating network

username: gmkim90
cooked: <p>Hello. Can i update the input tensor without updating network weight?<br>
For example,</p>
<pre><code class="lang-auto">input = torch.rand(10,5)  # minibatch size = 10, data dimension = 5
net = nn.Linear(5,3)
optimizer = optim.Adam(net.parameters(), lr=0.1)
target = torch.rand(10, 3) #minibatch size = 10, data dimension = 3

output = net(input)

loss = torch.pow(target-output, 2).sum()

loss.backward()
print(net.weight.sum()) # tensor(-1.0748, grad_fn=&lt;SumBackward0&gt;)
optimizer.step()
print(net.weight.sum()) # tensor(0.4252, grad_fn=&lt;SumBackward0&gt;)

</code></pre>
<p>this code, which is a typical case, updates network weight to minimize the loss function.<br>
Can I update the input without updating network weight for minimizing the loss function?</p>

username: ptrblck
cooked: <p>That should be possible, if you create your input using <code>requires_grad=True</code> and pass it to the optimizer.<br>
Have a look at <a href="https://discuss.pytorch.org/t/how-to-add-cosine-similarity-score-in-cross-entropy-loss/64401/2">this cosine similarity example</a> I’ve written yesterday.</p>
No module named 'torchvision.models.quantization'

username: dalalaa
cooked: <pre><code class="lang-auto">&gt;&gt;&gt; import torchvision.models.quantization as models
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
ModuleNotFoundError: No module named 'torchvision.models.quantization'
&gt;&gt;&gt; torchvision.__version__
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
NameError: name 'torchvision' is not defined
&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; torchvision.__version__
'0.4.2+cpu'
</code></pre>
<p>What is the version of torchvision needed in this <a href="https://pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html#part-2-finetuning-the-quantizable-model" rel="nofollow noopener">toturial</a></p>

username: lly-zero-one
cooked: <p>Can you try version 0.5? It could work on my local.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; torchvision.__version__
'0.5.0a0+333af7a'
&gt;&gt;&gt; import torchvision.models.quantization as models
</code></pre>

username: dalalaa
cooked: <p>Thank you for you reply, but when I run <code>pip install torchvision==0.5.0</code>, I got the following error:</p>
<pre><code class="lang-auto">Looking in indexes: http://mirrors.aliyun.com/pypi/simple/
ERROR: Could not find a version that satisfies the requirement torchvision==0.5.0 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.3.0, 0.4.0, 0.4.1, 0.4.2)
ERROR: No matching distribution found for torchvision==0.5.0`
</code></pre>
<p>Should I build torchvision from source?</p>

username: jerryzh168
cooked: <p>cc <a class="mention" href="/u/lly-zero-one">@lly-zero-one</a> <a class="mention" href="/u/raghuramank100">@raghuramank100</a></p>

username: raghuramank100
cooked: <p><a class="mention" href="/u/fmassa">@fmassa</a>: Can you help answer this question?</p>