akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Different time cost with different weights on the same model

username: nullptr-leo
cooked: <p>Hello,</p>
<p>I found different time cost with different weights on the same mobilenetv3 model using PyTorch.</p>
<p>The following code is used to measure time:</p>
<pre><code class="lang-auto">transform_fn = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225],
    ),
])
image = cv2.imread('test-image.jpg')
image = transform_fn(cv2.resize(image, (96,96), interpolation=cv2.INTER_LINEAR)).unsqueeze(0).cpu()

model.load_state_dict(torch.load('model_path', map_location='cpu'))
model.eval()

count = 100
with torch.no_grad():
    start_ts = time.time()
    for i in range(count):
        outputs = model(image)
    elapsed = (time.time() - start_ts) * 1000
    print('elapsed time:', elapsed / count)
</code></pre>
<p>I doubted if it is caused by the number of zero parameters in the model. Then I have counted the zero params in the different weights. But it turned out that the pretrained weights have less zero parameters.</p>
<p><strong>Pretrained:</strong><br>
Time: 8ms<br>
Zero-params number (eps=1e-6): 541<br>
Zero-params number (eps=1e-2): 114071</p>
<p><strong>After 100 epoch based on Pretrained:</strong><br>
Time: 20ms<br>
Zero-params number (eps=1e-6): 1538906<br>
Zero-params number (eps=1e-2): 1619014</p>
<p>Zero parameters counting code:</p>
<pre><code class="lang-auto">eps = 1e-2
zero_cnt = 0
params = list(model.parameters())
for param in params:
    zeros_count = torch.sum(torch.where(torch.abs(param) &lt; eps, torch.ones_like(param), torch.zeros_like(param))).int().item()
    zero_cnt += zeros_count
print('pretrained zero params:', zero_cnt)
</code></pre>
<p>Anybody know why this happens?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>You might see a performance hit, if you are handling a lot of denormal values.<br>
Set <a href="https://pytorch.org/docs/stable/torch.html#torch.set_flush_denormal" rel="nofollow noopener">torch.set_flush_denormal(True)</a> and run the code again.</p>

username: nullptr-leo
cooked: <p>Thanks. It’s caused by denormalized floats.</p>
<p>Both set_flush_denormal and setting all parameters close to zero to exactly zero can solve this problem.</p>
<ol>
<li>torch.set_flush_denormal(True)</li>
<li>Set all parameters close to zero to exactly zero</li>
</ol>
<pre><code class="lang-auto">for param in model.parameters():
    param.data = torch.where(torch.abs(param) &lt; eps, torch.zeros_like(param), param)
</code></pre>

username: bsting
cooked: <p>Is there anyway to prevent trained weights to have denormal values during the training time? we trained a model with ResNet50 as the backbone with PyTorch v1.0.0a0 and v1.4.0. Model trained with v1.4.0 has denormal value in trained weights and v1.0.0a0 does not, do you know the reason?</p>

username: addisonklinke
cooked: <p><a class="mention" href="/u/bsting">@bsting</a> Did you find any useful resources for avoiding denormal values during training? I am also curious</p>
Docker file for pytorch and computer vision task

username: henry_Kang
cooked: <p>Hello. This is my first time to use the docker file and docker stuff.<br>
My problem is that I cannot use the nvidia docker by my computer. I usually use the miniconda from school computer but school computer does not allow to use the docker and nvidia docer so I can not test it. I follow the<br>
detectron and add some of my stuff for it.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/HenryKang1/pytorch_dockerfile_contest" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/70066387?s=400&amp;v=4" class="thumbnail onebox-avatar" width="420" height="420">

<h3><a href="https://github.com/HenryKang1/pytorch_dockerfile_contest" target="_blank" rel="noopener nofollow ugc">HenryKang1/pytorch_dockerfile_contest</a></h3>

<p>Pytorch dockerfile for temporary cloud (still working on) - HenryKang1/pytorch_dockerfile_contest</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
Does anyone can figure it out it is ok?<br>
I have to provide docker file to cloud gpu company and they will provide install the specific library which I want to use. So I hope it is ok.<br>
I also recommend to my friend this docker file<br>
“”<br>
FROM <a href="http://nvcr.io/nvidia/pytorch:19.12-py3" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:19.12-py3</a><br>
USER root
<p>RUN apt-get update &amp;&amp; apt-get install -y \<br>
curl <br>
ca-certificates <br>
sudo <br>
unzip <br>
htop <br>
wget <br>
git <br>
bzip2 <br>
libx11-6 <br>
&amp;&amp; rm -rf /var/lib/apt/lists/*<br>
RUN apt-get update -y<br>
“”<br>
This is official nvidia docker he said.<br>
I can not test my computer so I am not sure they include the opencv and albumentations.<br>
Thank you.</p>
Cnn for multiple face attribute classification (binary)

username: flipflop
cooked: <p>I am doing a tutorial on classifying the multiple face attributes . My idea is to crate objects of model multiple times to train each attribute . Can we do one model to train all attributes at a time ?  because as what I can imagine ,the cnn model will output  binary classes -1 or 1 for one attributes . Are there any ways to output 10 attributes with (-1 or 1) instead of assigning a model to one attribute</p>

username: ptrblck
cooked: <p>If you would like to use a multi-label classification (multiple outputs can have an active or inactive class), then you could use <code>nn.BCEWithLogitsLoss</code> as the criterion and make sure your model outputs have the shape <code>[batch_size, nb_classes]</code>.</p>
<p>I’m not sure if <code>-1</code> and <code>1</code> would correspond to a positive and negative case, but the target for <code>nn.BCEWithLogitsLoss</code> should contain values in <code>[0, 1]</code>.</p>

username: flipflop
cooked: <p>Hi thank you for your answers , I am building a resNet 50 ,which output 2 feature at then end , but i need to produce 40 face attributes for classification, in class (0 and 1) , i can do for one attribute,which is very standard , I have actually come cross an question you have answered in this forum which create multiple fc layer and return multiple output at then end   ,so i use module list to create 40 model output where each one has class probability for 1 and 0 .   it seems work , but my training is ridiculously long(probability 10 hours for just one epoch) ,(dateset is celebrity dateset, around 170000 training image,7k each in size) . I still trying to perfect the code . Is my approach correct? appreciate your reply thx</p>

username: ptrblck
cooked: <p>As previously said, if these face attributes can be independently active (1) or inactive (0), then you are dealing with a multi-label approach and could use a single output layer with <code>out_features=40</code> and <code>nn.BCEWithLogitsLoss</code> as the loss function.</p>
How to load complete UCF101 dataset?

username: div_sh
cooked: <p>Hi!<br>
I am using the UCF101 dataset ( <code>torchvision.datasets.</code> <code>UCF101</code>). I want to load the complete dataset to make custom splits. However, I only see the boolean ‘train’ as an argument? Is there a way to create a DataLoader for the entire dataset?</p>
<p>Thanks!</p>

username: thyeros
cooked: <p>perhaps, you can load both train/test and merge (like concat) into one? since we can do split, I guess we should be able to merge too.</p>

username: div_sh
cooked: <p>I guess I’ll try that. I was hoping there would be a better way to load the entire dataset instead of combining the train/test splits and the folds.</p>
<p>Thanks! I’ll get back after I try it</p>
Backprop from loss from a different network

username: mhong94
cooked: <p>What happens when you call step() but the backprop is coming from a different network?  The architecture from the 2 networks could be different.  Here is some sample pseudocode</p>
<blockquote>
<p>net_1 = Net_1()<br>
net_2 = Net_2()<br>
optimizer_1 = Optimizer(net_1.params)<br>
optimizer_2 = Optimizer(net_2.params)<br>
outputs = Net_1(inputs)<br>
loss = CrossEntropyLoss(outputs,targets)<br>
optimizer_2.zero_grad()<br>
loss.backward()<br>
optimizer_2.step()</p>
</blockquote>
<p>I backprop through net_1, but I am stepping with network 2.</p>

username: albanD
cooked: <p>Hi,</p>
<p>This will use the gradients in network 2 to perform the step.<br>
So if you didn’t use it and didn’t run any backward, these grads will be 0 and this step op will do nothing.</p>
Continue training from checkpoint returns high loss values, while reasonable with .eval()

username: yairkit
cooked: <p>I load my trained model from checkpoint for a fine-tune training.</p>
<p>then when I do:<br>
model.eval()<br>
model(x)<br>
output seems OK, loss scale is same as at the end of pre-train.</p>
<p>but for:<br>
model.train()<br>
model(x)<br>
output is totally different, very bad - just like it’s a “training from scratch”.</p>
<ul>
<li>the model pretrained with DDP</li>
<li>the model has BN layers</li>
</ul>
<p>Am I doing something wrong?<br>
Thanks</p>

username: Kushaj
cooked: <p>For <code>model.train</code> are you also updating the parameters? If you want to do inference on your model you should use</p>
<pre><code class="lang-python">with torch.no_grad():
    model.eval()
    # inference
</code></pre>

username: rvarm1
cooked: <p>In addition, you can try setting <code>torch.backends.cudnn.enabled = False</code> when training using SyncBatchNorm and DDP, as discussed in <a href="https://discuss.pytorch.org/t/training-performance-degrades-with-distributeddataparallel/47152/27" class="inline-onebox">Training performance degrades with DistributedDataParallel</a>.</p>
Splitting single large fully connected layer over multi GPUs

username: idaham
cooked: <p>I am trying to build a network in Pytorch with a very large fully connected top layer, on the order of input 80000, output 15000 elements (there are more layers after).</p>
<p>The top layer alone requires too much CUDA memory to fit on one GPU during training (even with batch size 1).</p>
<p>Pytorch’s DataParallel for GPU splitting still needs to put the model on all GPUs (as far as I know), and gather the data on one at the end, so doesn’t help me with the memory issue.</p>
<p>Other posts discuss splitting a large model onto several GPUs (e.g. <a href="https://discuss.pytorch.org/t/split-single-model-in-multiple-gpus/13239">Split single model in multiple gpus</a>). From what I understand, using something like (from linked post):</p>
<pre><code class="lang-auto">self.large_submodule1.cuda(0)
self.large_submodule1.cuda(1)
...
</code></pre>
<p>only seems to be relevant when you can split whole modules, e.g. the first whole FC layer, the second FC layer etc. I don’t know if and how it can be used for splitting one module, here one FC layer? In the MWE below, that would mean splitting fc onto multiple GPUs.</p>
<pre><code class="lang-auto">class network(nn.Module):
    def __init__(self,sizeIn,sizeOut):
        super(network, self).__init__()
        ### Linear FC layers
        self.fc = nn.Sequential(
            nn.Linear(sizeIn, sizeOut),
            nn.Tanh(),
        )  
    def forward(self, x):
        x = self.fc(x)
        return x
</code></pre>
<p>Any tips are much appreciated! Thanks.</p>

username: Najeeb_Nabwani
cooked: <p>Did you manage to find a solution?</p>
Setup environment beginner for pytorch

username: Evelyne_Cal
cooked: <p>Hi All,</p>
<p>I am beginner to this field. Currently i would like to setup environment for my GPU.<br>
I am lost here and there.</p>
<ol>
<li>I try to install the driver 418 as I want cuda 10.1 which has support for pytorch. But I got Driver 450.66 instead. and when i type nvidia-smi, it show Cuda version 11, I search in google it said the cuda here is driver API compatibility. If I install another cuda version 10.1, will my environment is messed up?</li>
</ol>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/4/241609873c7758ae7a48ab0d8efa7230f0cdf11c.png" alt="image" data-base62-sha1="59ep0Ufmgv5PNeY9Cx9PeQFHZyQ" width="638" height="43"></p>
<ol start="2">
<li>
<p>When I install cuda, shall I also install the cudnn altogether?<br>
However I found a folder for cudnn. “/usr/include/cudnn.h”</p>
</li>
<li>
<p>Since I got not a clean device, I am not sure what they have installed in the GPU. So I a little bit confuse if my environment is messed up. How can i know if all the configuration setup is correct for deep learning environment using pytorch?</p>
</li>
</ol>

username: albanD
cooked: <p>Hi,</p>
<p>If you have a new device, I would actually advise with using conda to install these things. You only need to install the nvidia driver (which you already have) and everything else will come in the form of conda package in a given environment. This means that if you mess things up, just remove that environment and create a new one and all is good again!</p>
<p>But once you have the nvidia drivers and conda, you can just follow the instructions from the website <a href="https://pytorch.org/get-started/locally/">https://pytorch.org/get-started/locally/</a> to install with conda. And it will automatically install the right version of cuda for you (as well as all other required dependencies)!</p>

username: Evelyne_Cal
cooked: <p>Hi AlbanD,</p>
<p>Thank you for your advise.<br>
So the step would be: driver --&gt; Conda --&gt; create environment --&gt; install Pytorch?<br>
Therefore every create a new environment we shall install pytorch with cuda?<br>
Or shall I install pytorch with cuda first then create environment?</p>
<p>My GPU is in server, and has several users. Would that be wise if I install driver and conda in the root, then enable conda to all users?<br>
Thank you so much</p>

username: albanD
cooked: <p>Especially for a shared server, I would recommend to have as few things in the global install as possible. That gives user freedom to install whichever variant they want. And prevent them from breaking the whole system if they do bad things.<br>
In particular because installing cuda within conda is very simple and safe.</p>
<p>So I would suggest:<br>
(for everyone): driver -&gt; Conda<br>
(for each user): create env -&gt; install pytorch (that will pull cuda automatically)</p>
<p>Note that you can also make users install their own conda as well if you want to ensure more separation between users.<br>
The benefit of the share conda is that packages will be re-used, reducing disk footprint.</p>
Forward and Backward calls in parallel in synthetic gradient method?

username: iacolippo
cooked: <p>Hi folks</p>
<p>I am reimplementing this paper <a href="https://arxiv.org/abs/1909.01311" rel="noopener nofollow ugc">https://arxiv.org/abs/1909.01311</a> a method that allows backward and forward unlocking (i.e. you can compute the “gradients” of a layer as soon as you have executed the forward of such layer). This is a dummy implementation</p>
<pre><code class="lang-auto">class Linear(nn.Linear):
    def __init__(self, in_features, out_features, bias=True):
        super().__init__(in_features, out_features, bias)

    def forward(self, x, y):
        return super().forward(x), y


class ErrorFeedback(nn.Module):
    def __init__(self):
        super().__init__()
        self.rm = None

    def forward(self, x, y):
        if self.rm is None:
            device = x.device
            self.rm = torch.randn(y.shape[1], x.shape[1]).to(device) / sqrt(x.shape[1])
        x.backward(torch.mm(y, self.rm))
        return x.detach(), y


class Sequentialy(nn.Sequential):
    def __init__(self, *args):
        super().__init__(*args)

    def forward(self, x, y):
        for module in self:
            x, y = module(x, y)
        return x

model = Sequentialy(OrderedDict([("lin1", Linear(input_features, 2000)), ("ef1", ErrorFeedback()),
                                 ("lin2", Linear(2000, 2000)), ("ef2", ErrorFeedback()),
                                 ("lin3", Linear(2000, 1000)), ("ef3", ErrorFeedback()),
                                 ("lin4", Linear(1000, n_classes))])).to(device)
</code></pre>
<p>My question is: how does it work with the GPU calls being async? Is the backward call inside each <code>ErrorFeedback</code> layer executed at the same time as the forward of the following layer? If not, is there a way to do that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If there are no synchronization points (I don’t see one in your code), then yes it will be.</p>
How to extract features from sparse Tensors

username: a_d
cooked: <p>Hello everyone,<br>
I have around 5 fields of ordinal data that have to be one-hot-encoded. These fields essentially describe the context in which a sentence was made. The one-hot-encoded would be sparse tensors. Although I am not sure, my understanding is that due to sparsity traditional layers applied such data would yield meaningless results. I was wondering if there is any way to extract features from these sparse tensors such that I would be able to “include the Context” in the sentence.</p>
Replicating the Pretrained Semantic Segmentation

username: ArashAkbarinia
cooked: <p>I would like to reproduce the reported accuracies of <a href="https://pytorch.org/docs/stable/torchvision/models.html#semantic-segmentation" rel="nofollow noopener">pre-trained segmentation models</a>.</p>
<p>I executed the training code from <a href="https://github.com/pytorch/vision/blob/master/references/segmentation/train.py" rel="nofollow noopener">references</a> under default parameters with dataset <strong>coco</strong>. The trained DeepLabV3 ResNet101 reaches in the validation-set about 60% IoU (with and without the auxiliary loss). This is much lower than the reported 67.4%.</p>
<p>I do appreciate any help on this issue. Thanks a lot in advance :).</p>

username: Kanishk_Jain
cooked: <p>Hi, were you able to resolve your problem?<br>
I am also facing a similar issue, I have removed an aditional 3000 images from training set, as they were overlapping with my validation and test set, total training images after filtering are 92,034. Best Mean IOU accuray of 53.5 occured at 8th epoch, and it has been fluctuating between 50-53 since 2nd epoch.<br>
I am using the reference scripts for segmentation and following command line for training</p>
<pre><code class="lang-auto">OMP_NUM_THREADS=4 CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 --use_env train.py --lr 0.02 --dataset coco -b 4 -j 4 --model deeplabv3_resnet101 --aux-loss
</code></pre>

username: ArashAkbarinia
cooked: <p>Hi Kanishk,<br>
No, I haven’t been able to resolve this. In PyTorch <a href="https://github.com/pytorch/vision/tree/master/references/segmentation" rel="noopener nofollow ugc">GitHub</a> it’s mentioned all models have been trained on 8x V100 GPUs with the following command</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet50 --aux-loss
</code></pre>
<p>I only have access to two GPUs. As a work around, I tried with nproc_per_node=2 and -b 16, but still couldn#t reproduce the reported accuracies.</p>
Perturbing distributions

username: arijitthegame
cooked: <p>Hi all,</p>
<p>What is the quickest way to create some perturbations of the distributions already available in PyTorch distributions? For example, if I want to create a new distribution by doing $(1-\epsilon)N(\mu,\sigma^2) + \epsilon \delta_x$, what would be the quickest way to do it? My other goal is to perturb some pdf’s with Gaussian noise. If f(x) is a pdf of a probability density function, then I would like to construct a new density by perturbing it with some Gaussian noise by essentially the same equation as above $(1-\epsilon)f(x) + \epsilon N(\mu,\sigma^2)$.</p>
<p>Thanks very much.</p>

username: googlebot
cooked: <p>Affine transformed gaussian is another gaussian distribution. Look at <a href="https://pytorch.org/docs/master/distributions.html#torch.distributions.transforms.AffineTransform" rel="noopener nofollow ugc">https://pytorch.org/docs/master/distributions.html#torch.distributions.transforms.AffineTransform</a> and other bijective transforms there.</p>
<p>Generally, you can’t mess with pdfs, as they have to integrate to 1. Valid approaches to construct pdfs include mixture distributions, compound distributions, normalizing flows (advanced version of bijective transforms).</p>

username: arijitthegame
cooked: <p>Thank you for your answer. What I was writing is basically creating various mixture distributions. I still can not figure out how to create mixture distributions except for the code of the mixture in the same family in pytorch.</p>

username: googlebot
cooked: <p>Well, mixture’s log_prob() is not hard to write (logsumexp() can be handy). But there is not much else to put into distribution objects, they’d just contain Categorical and component distribution objects. And differentiable sampling won’t work because of Categorical.</p>

username: arijitthegame
cooked: <p>Ah. Got it. Thank you.</p>
How to use .pth model in iOS App

username: Rahish_Kansal
cooked: <p>Hello All, Hope you doing well with your loving ones!!</p>
<p>Want to use the PoseNet 3D model in iOS app. For which decided to go for <a href="https://github.com/Daniil-Osokin/lightweight-human-pose-estimation-3d-demo.pytorch" rel="noopener nofollow ugc">https://github.com/Daniil-Osokin/lightweight-human-pose-estimation-3d-demo.pytorch</a> . For which we can download the Pre trained Model from <a href="https://drive.google.com/file/d/1niBUbUecPhKt3GyeDNukobL4OQ3jqssH/view" rel="noopener nofollow ugc">https://drive.google.com/file/d/1niBUbUecPhKt3GyeDNukobL4OQ3jqssH/view</a>.</p>
<p>Here I have downloaded the model which having .pth as extension. As per the Pytorch ‘Hello World’<br>
.pt is required. when using ‘LibTorch’ pod.</p>
<p>as per apple it should be converted to .mlmodel to use the native APIs.</p>
<p>But I am not able to proceed anyway. as neither able to convert the .pth model to .pt nor .mlmodel</p>
<p>Any help will be appreciated</p>
<p>Thanks &amp; Regards<br>
Rahish</p>
Script error: Module has no attribute _modules

username: dimens66
cooked: <pre><code class="lang-auto">RuntimeError: 
Module 'SparseSequential' has no attribute '_modules' :
  File "/home/gcy/code/spconv-master/spconv/modules.py", line 132
    def forward(self, input):
        print(888888888, self._modules)
                         ~~~~~~~~~~~~~ &lt;--- HERE
        for module in self._modules:
          ......
</code></pre>
<p>NOW I have a class <strong>SparseSquential</strong> derived from nn.modulle. When I script this class, I got the error above. As we all know , the <strong>_modules</strong> is a member variable ‘OrderedDict’ in the nn.module. So how to solve it. THanks .</p>
<p>My environment is pytorch:1.5.0 .</p>

username: suraj.pt
cooked: <p>Hi,<br>
Check out <code>.modules()</code> or <code>.named_modules()</code> - both return an iterator of modules in the network.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.modules" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.modules" target="_blank" rel="noopener nofollow ugc">Module — PyTorch 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Loading pretrained model

username: mk_sherwani
cooked: <p>I have trained a model using CYCLEGAN: I have the model.pth file, now I am trying to load the model and look for the saliency map features and also I want to generate the output on each encoding layer.</p>
<p>I have tried several things but got no luck. Can anyone help?</p>

username: Caruso
cooked: <p>Maybe this Tutorial about <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference" rel="noopener nofollow ugc">saving and loading models in PyTorch</a>  helps you.</p>
How do I load png images together with alpha channel?

username: TeraTux
cooked: <p>using this code</p>
<pre><code class="lang-auto">data_path = '.'
train_dataset = torchvision.datasets.ImageFolder(
    root=data_path,
    transform=torchvision.transforms.ToTensor()
)
train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=1,
    num_workers=1,
    shuffle=True
)
</code></pre>
<p>I can load multiple png images but all transparent parts turn into black. I want to keep all four channels. I searched how to do it but people have the inverse problem - they have all four channels and they want to drop the alpha channel. So what shall I do so the alpha channel is loaded also?</p>

username: TeraTux
cooked: <p>I figured it out myself.</p>
<pre><code class="lang-auto">from scipy import misc
data_path = '.'
def my_loader(path):
    return misc.imread(path);
train_dataset = torchvision.datasets.ImageFolder(
    root=data_path,
    transform=torchvision.transforms.ToTensor(),
    loader = my_loader
)
train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=1,
    num_workers=1,
    shuffle=True
)
</code></pre>
Weight decay vs L2 regularisation

username: ash95
cooked: <p>I just want to confirm whether the weight decay parameter in optimisers is equivalent to applying L2 regularisation. According to fastai’s <a href="https://www.fast.ai/2018/07/02/adam-weight-decay/" rel="noopener nofollow ugc">article</a> on this, weight decay and L2 regularisation are only equivalent when used in vanilla SGD. There was also the <a href="https://arxiv.org/abs/1711.05101" rel="noopener nofollow ugc">Decoupling Weight decay</a> paper that states weight decay is a better alternative to L2 loss. Just wanted to how this is done in Pytorch.</p>
SGD vs Batch size 1

username: edshkim98
cooked: <p>Hello everyone,<br>
I am currently training my network in mini batch size using Adam optimizer.<br>
However, because I’ve read some posts saying if i choose the right lr and lr_schedule parameters,<br>
sgd could perform better tahn adams, I am planning to try SGD.<br>
From my understanding, SGD updates the parameters in each sample which has batch size of 1 since batch size greater than 1 is just a normal GD.<br>
But, I am seeing lots of pytorch code using SGD with batch size greater than 1 (e.g. 16).<br>
How does pytorch SGD then work and update the parameters?<br>
Also, can I still use batch normalization when i use SGD? SGD is when batch size is 1, so surely batch normalization will either not work or perform really badly.</p>

username: Pengzhangzhi
cooked: <p>Hi!<br>
First of all, batch size greater than 1 is min batch instead of  a normal GD.<br>
The number of batch size determines the type of GD(SGD with batch size=1 or mini-batch with batch size greater than 1).<br>
In addition,I personally recommend you use Adam and mini-batch.Since the two of them outperform much steady and robust training.<br>
Finally, you can’t use batch normalization with SGD-batch size=1, because batch normalization compute the average over a batch, since your batch size=1, there is no need to compute average value over a single value.<br>
I hope I would make this clear.<br>
If you have any more questions, please let me know.<br>
Have a nice day : )</p>

username: edshkim98
cooked: <p>Thank you for the reply!<br>
However, my question is that I see some codes in pytorch which they use batch size greater than 1 with SGD and this means it is not SGD really anymore. So, how does pytorch calculate SGD when batch size is greater than 1?</p>

username: Pengzhangzhi
cooked: <p>Sorry,after searching the wiki,turns out I was wrong at(SGD is batch size=1).<br>
Really sorry!<br>
SGD is randomly chosen an example in the batch as the gradient.<br>
Use randomly picked gradient to replace the gradient computed over all examples or batch.<br>
So It’s reasonable to use batch size that greater than 1.<br>
However, from my experience, It’s very unstable using SGD,which would cause your loss changes dramatically.<br>
If you want to know more, please <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="noopener nofollow ugc">click here</a> and check out.</p>
<p>Hope I didn’t make you confused: )</p>
Multi-system model- Batch norm VS layer norm

username: almog
cooked: <p>Hello,<br>
I’m new to PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I have a regression task and I use a model that receives two different sequential inputs, produces LSTM to each input separately, concatenates the last hidden of each LSTM, and predicts a value using a linear layer of out_size 1.  (my forward() function is written below)</p>
<p>I’m using an accumulated gradient as explained here: [<a href="https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822/8" class="inline-onebox">How to implement accumulated gradient？</a>]  (the second option), so my model receives a single sample in each forward() call.</p>
<p>I want to add normalization to my model.</p>
<ol>
<li>Is there a problem to add batch normalization because I’m using an accumulated gradient?</li>
<li>Should I add batch normalization or layer normalization?</li>
<li>Where in my model should I add the normalization? before or after LSTM?</li>
<li>To which part in the model should I add it? to input1 and input2 separately? after concatenation? add in both places?</li>
</ol>
<p>My forward function in the model:</p>
<pre><code>def forward(self, input1, input2):
    # input1 part
        embeds = self.word_embedding(input1) # glove word embedding 
        encoder1_out = self.encoder1(embeds) #BiLSTM
        attention_out = self.HAN(encoder1_out) # hirerchical attention network 
        
    # input2 part
        encoder2_out = self.encoder2(input2) #BiLSTM

    # combined part
       info_vector = torch.cat((attention_out, torch.flatten(encoder1_out).unsqueeze(0)), dim=1)          
       return self.linear(info_vector) # [1, hidden_dim_1 + flatten_hiddden_dim_2] -&gt; [1]
</code></pre>
<p>Thank you!<br>
Almog</p>
Parallelize loading model to GPU

username: ap579
cooked: <p>Hi!</p>
<p>I’m trying to run many models on a single GPU by switching them in and out as needed (they don’t all fit in GPU memory together), but I’m finding that loading each model:</p>
<pre><code>model = model.to('cuda')
</code></pre>
<p>takes 20-80ms (ex/ VGG16: ~80ms). If I want to load two different VGG16 models at a time, is there a way to parallelize loading such that the total load time is &lt; 160ms?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>You could call the <code>to()</code> methods multiple times on different models, but note that the memory bandwidth is limited so you cannot push the parameters faster to the device than your system allows.</p>
Conversion error at validation step

username: Varun_Tirupathi
cooked: <p>Hi Everyone I’m facing this conversion error after completing the validation step of my UNet regression model it is throwing this error: I don’t know am i providing the sufficient info for your understanding</p>
<p><strong>File “C:\Python\Python 3.8.5\lib\site-packages\torch\utils\tensorboard_convert_np.py”, line 29, in make_np</strong><br>
raise NotImplementedError(<br>
<strong>NotImplementedError: Got &lt;class ‘torch.nn.modules.loss.MSELoss’&gt;, but numpy array, torch tensor, or caffe2 blob name are expected.</strong> The code where im facing error is attached below:</p>
<pre><code class="lang-auto">    def validation_end(self, outputs):   # getting the mean of stack of validation losses
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        tensorboard_logs = {'val_loss': avg_loss}
        return {'avg_val_loss': avg_loss, 'log': tensorboard_logs}</code></pre>

username: ptrblck
cooked: <p>Based on the error message it seems you are passing the class <code>nn.MSELoss</code> to the tensorboard visualization instead of a tensor or array containing the loss value.<br>
Could you check the <code>type</code> of all objects and make sure they are valid values and not the loss function itself? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
"All or nothing" loss function? multilabel classification

username: wboy
cooked: <p><strong>Set up</strong><br>
You have an input dataset <code>X</code>, and each row has multiple labels. Eg, 3 possible labels, <code>[1,0,1]</code> etc</p>
<p><strong>Problem</strong><br>
The typical approach is to use BCEwithlogits loss or multi label soft margin loss. But what if the problem is now switched to <em>All</em> the labels must be correct, or dont predict anything at all?</p>
<p>What loss function do we pick for this? I thought of coding a custom loss function that returns 0 if all the labels match, else 1 but it seems “hacky”.</p>
<p>Thanks!</p>
Ignoring a specific label during training in segmentation

username: mila
cooked: <p>Hello, I started working with deep learning with Pytorch not very long ago and I have an issue with segmentation. My labels have three values : ‘0: no data’, ‘1: no vegetation’ and ‘2: vegetation’. I don’t want my algorithm to learn and classify ‘0: no data’ labels. After digging on internet I saw that one possible solution consists in putting a weight of 0 in the loss function, but I’m thinking that it might be problematic with respect to the computation of the gradient, I don’t know if I’m right though. Is there an elegant way to ignore the ‘0: no data’ label from training apart from setting a weight to “0” in the loss function? Thank you in advance.</p>

username: ptrblck
cooked: <p>You could alternatively use <code>ignore_index</code> in <code>nn.CrossEntropyLoss</code> to ignore this class during the loss calculation.</p>
Pytorch transform issue ,

username: Coder1221
cooked: <p>how i can get the orignal image back?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aa8097086c82903217a456a036886e1a6ff38df.png" data-download-href="https://discuss.pytorch.org/uploads/default/6aa8097086c82903217a456a036886e1a6ff38df" title="Screenshot from 2020-10-13 11-15-07"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aa8097086c82903217a456a036886e1a6ff38df_2_690x387.png" alt="Screenshot from 2020-10-13 11-15-07" data-base62-sha1="fdwD0FVGOtyg9igYQ0FKZeYSeJN" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aa8097086c82903217a456a036886e1a6ff38df_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aa8097086c82903217a456a036886e1a6ff38df_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aa8097086c82903217a456a036886e1a6ff38df.png 2x" data-dominant-color="372F3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-10-13 11-15-07</span><span class="informations">1366×768 207 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The usage of <code>reshape</code> is wrong as it will interleave the image.<br>
Use <code>permute</code> to permute the dimensions of the tensor instead.</p>
<p>If you want to get the original pixel values back, you would have to “denormalize” the image such that the values are again in the original range (most likely <code>[0, 255]</code>) instead of <code>[0, 1]</code>, which is done by <code>ToTensor()</code>.</p>
Type mismatch occurs in C++ code with torchscript

username: Nanco-L
cooked: <p>Hello, I tried to generate torch model and use it in C++ code.<br>
Since I want to set dtype of all variables in model to double, I use <code>torch.set_default_dtype(torch.float64)</code> in my python code for generating torchscript file.<br>
And I create C++ variable with dtype double:</p>
<pre><code class="lang-auto">torch::Tensor i_tensor = torch::zeros({3,10}, torch::dtype(torch::kFloat64));
auto i_tensor_a = i_tensor.accessor&lt;double,2&gt;();
</code></pre>
<p>But I face the error like below:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::runtime_error'
  what():  The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "&lt;string&gt;", line 3, in &lt;foward op&gt;

      def addmm(self: Tensor, mat1: Tensor, mat2: Tensor, beta: number = 1.0, alpha: number = 1.0):
          return self + mat1.mm(mat2)
                        ~~~~~~~ &lt;--- HERE

      def batch_norm(input : Tensor, running_mean : Optional[Tensor], running_var : Optional[Tensor], training : bool, momentum : float, eps : float) -&gt; Tensor:
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #3 'mat2' in call to _th_addmm_out
</code></pre>
<p>How can I fix it?</p>
<p>Note: It works well when I change double to float in C++ code and remove the line contains <code>set_default_dtype</code></p>

username: Nanco-L
cooked: <p>I solve this issue and leave this thread to someone who face same problem.</p>
<p>The reason of error is tensors generated in the model.<br>
Dtype setting in python code with <code>torch.set_default_dtype()</code> is not saved in torchscript.<br>
Thus, one need to set dtype explicitly to avoid type mismatch issue.<br>
But <code>torch.set_default_dtype()</code> is still useful because it automatically set the dtype of network weights, biases, etc.</p>
GPU out of memory after a few steps, accumulation error?

username: Majid_Al
cooked: <p>I am training a Roberta masked language model for which I read my input as batches of sentences from a huge file. For batch sizes of 4 to 16 I run out of GPU memory after a few batches. Batch sizes over 16 run out of memory right away. Only batch_size of 2 works. I made sure not to accumulate anything. Is the error result of the size of the model or I am managing my memory badly?</p>
<pre><code class="lang-auto">def train():
    model.to(device)
    model.train()
    total_loss = 0
    total_preds = 0
    for step,batch in enumerate(train_dataloader):
        if step % 512 == 0 and step != 0:
            print(f'Batch {step}---loss: {maskedlmloss.item()}')
            writer.add_scalar('loss/training_loss', maskedlmloss.item(), step)
            writer.add_scalar('optimizer/lr', get_lr(optimizer), step)
            writer.flush()

        encoded_inputs = tokenizer(batch, padding=True, truncation=True, return_tensors='pt')
        dic = collator(encoded_inputs['input_ids'].unbind())
        input_ids, labels = dic['input_ids'], dic['labels']
        
        model.zero_grad()
        maskedlmloss, prediction_scores = model(input_ids.to(device), labels = labels.to(device))

                
        total_loss += float(maskedlmloss.item())
        total_preds += 1

        maskedlmloss.backward()
        optimizer.step()
        scheduler.step()
    return total_loss/total_preds
</code></pre>
<p>Thanks for your help in advance</p>

username: ptrblck
cooked: <p>Are you using variable input shapes? If so, could you check if the largest batch would fit into the model?</p>

username: Majid_Al
cooked: <p>All my batches are of the same size and I play with this size to find the largest that fits in the model. I am not sure if that’s what you are asking but the largest batch_size on which the model can carry out its computations without running out of memory is 2.</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>Are you seeing any memory increase in <code>nvidia-smi</code> for a batch size of 2?</p>
Chained Dataloaders

username: Vladimir_Blagojevic
cooked: <p>Hello everyone,</p>
<p>The framework I need to plug in expects an instance of data loader, yet I need to train on two different data sets each having a different collate fn. So I can’t simply use ConcatDataset; I need something like ConcatDataloader. Is there such a best somewhere? Do you see other ways to solve this issue?</p>
<p>Many thanks.</p>

username: ptrblck
cooked: <p>Would the output of the different <code>Dataset</code>s and collate functions be the same or would they be different?<br>
In the latter case I guess you don’t want to shuffle the indices/samples, since this would create “mixed” batches?<br>
Maybe you could create a custom <code>collate_fn</code>, which uses different work flows based on the passed indices? I.e. would it work if you combine the two collate functions and change the behavior based on the currently loaded data?</p>

username: Vladimir_Blagojevic
cooked: <p>Thanks for your response <a class="mention" href="/u/ptrblck">@ptrblck</a>, I overlooked “mixed batches”. This unified Dataloader would be used in BERT pre-training. The research paper suggests using one sequence length encoding first and then the second. Therefore, “To speed up pretraing in our experiments, we pre-train the model with a sequence length of 128 for 90% of the steps. Then, we train the rest 10% of the steps of sequence of 512 to learn the positional embeddings”.</p>
<p>It would be easier to define the custom collate_fn if the indices are passed to it but it only receives the samples/data directly, right? Perhaps, another approach now that the context is more clear?</p>

username: ptrblck
cooked: <p>If you want to select specific samples, I think you should implement a custom <code>sampler</code>.<br>
Samplers are used to create the indices, which are passed to the <code>Dataset.__getitem__</code> and can thus implement e.g. shuffling or weighted sampling.<br>
Once the samples are loaded they will be passed to the <code>collate_fn</code>, which will then create a batch (by default it will just <code>torch.stack</code> the samples).</p>
<p>Would that make sense or am I still misunderstanding the use case?</p>
How to linearly interpolate between two models?

username: Ravi_Raju
cooked: <p>Hi I’m trying to linearly interpolate between two models but it doesn’t seem to be working. Here is my function to do the interpolation.  I have the output attached below.</p>
<pre><code class="lang-auto">`def interpolate_models(model, model1):
    alphas = np.linspace(0,1,10)
    acc_array = np.zeros(alphas.shape)
    
    iter = 0
    tic = time.perf_counter()
    for alpha in alphas:   
        new_model = copy.deepcopy(model)
        new_model_1 = copy.deepcopy(model1)
        
        for n, p in new_model.named_parameters():
            p.data = (alpha) * p.data + (1 - alpha) * {n:p for n, p in new_model_1.named_parameters()}[n].data

        _, inter_acc = evaluate_pgd(new_model, val_loader, 20,1,None)
        print(inter_acc)
        acc_array[iter] = 100 - inter_acc
        iter = iter + 1
        del new_model
        del new_model_1
    toc = time.perf_counter()
    print(f"Eval took {(toc - tic)/60:0.4f} mins")
    return alphas, acc_array`
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e993ee3481c96b74cd5cf16cb85975fc7a835726.png" data-download-href="https://discuss.pytorch.org/uploads/default/e993ee3481c96b74cd5cf16cb85975fc7a835726" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e993ee3481c96b74cd5cf16cb85975fc7a835726_2_690x325.png" alt="image" data-base62-sha1="xkkbNdVGLMvMnUUDWGc4dP7aTye" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e993ee3481c96b74cd5cf16cb85975fc7a835726_2_690x325.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e993ee3481c96b74cd5cf16cb85975fc7a835726_2_1035x487.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e993ee3481c96b74cd5cf16cb85975fc7a835726_2_1380x650.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1510×712 60 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Clearly, the accuracy for model is about what I would expect but not for model1. Is my interpolation function totally wrong or is pytorch doing something weird to the weights under the hood?</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>The manipulation of the <code>.data</code> attribute is not recommended as it might yield unwanted side effects.<br>
Could you try to “interpolate” the models by using their <code>state_dict</code>s and load the new one into a model instead?</p>

username: Ravi_Raju
cooked: <p>Yes, thank you. As soon as I tried your suggestion, it worked. In my case, I forgot to include the batch norm parameters in my interpolation so of course my results will be wrong. <code>state_dicts</code> helped me catch that and resolve my bug. Thanks again!</p>
Parsing prim::Constant ListType objects in nodes using the python_ir

username: kilurb
cooked: <p>When I load a frozen model, I want to be able to parse the nodes into their respective values for analysis.  I can use node.i(“value”) to get IntType integer values into a Python object.  The same works for FloatType floats using node.f("value).  But I cannot figure out a way to parse ListType prim::Constant values into Python.  Is there a way to do this?</p>
Conda cuda version and local cuda path synchronizing issues

username: makeastir
cooked: <p>Hi.</p>
<p>I suffering from cuda version issues.<br>
In my case likes below :</p>
<ol>
<li>A case : pytorch 1.5, cuda 11.0</li>
<li>B case : pytorch 1.0, cuda 9.0</li>
</ol>
<p>To run A case project, then i correct my cuda local path to 11.0.<br>
Likewise B case project also correct my cuda local path to 9.0.</p>
<p>It’s really inconvenient job.</p>
<p>But someone said that if i installed conda virtual environment using specific version of cudatoolkit then do not change local cuda path.</p>
<p>For example, A project based on CUDA 9.0.<br>
And my local machine linked with CUDA 11.0 but if i installed conda virtual environment with cudatoolkit 9.0 then not necessary change local cuda path. A project works clear through CUDA 9.0.</p>
<p>Is it right or have any other opinion?<br>
Thank you for reading my question.</p>

username: ptrblck
cooked: <p>The conda binaries and pip wheels ship with their own CUDA (cudnn, NCCL, etc.) runtimes and your local installations won’t be used in built-in PyTorch methods.<br>
If you need to build a custom CUDA extension, your local CUDA installation will be used and you should make sure the binaries use the same CUDA version as your local one.</p>
<p>Besides that your local libs will be used if you build PyTorch from source.</p>

username: makeastir
cooked: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your timely reply.</p>
<p>I already install on local machine that CUDA 10.1, 11.0, 9.0.<br>
And my local machine’s /usr/local/cuda was linked at CUDA 9.0.<br>
At that time, i want to running B project which based on CUDA 11.0.<br>
Also already constructed conda virtual environment using pytorch 1.5 cuda 11.0.</p>
<p>At this point, should i re-linked my local machine’s cuda path from 9.0 to 11.0 ? Or no require to change local machine’s cuda path because my conda virtual environment already constructed that included CUDA 11.0 and cudatoolkit ?<br>
On the other words, In anaconda virtual environment pytorch equaly calculate like CUDA 11.0 ?</p>
<p>Sorry for poor english and thank you for reply.</p>

username: ptrblck
cooked: <p>If you’ve built PyTorch using different CUDA versions (local installations) in different conda environments, you wouldn’t need to change the CUDA path as long as you are not rebuilding PyTorch or CUDA extensions.<br>
I.e. running PyTorch operators alone would work in your conda environments using the libraries built with the different CUDA compilers.</p>