akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CelebA dataset download errors

Harry_Coppock: <p>Dear Sir/Madam,</p>
<p>I am having issues downloading the CelebA dataset. It appears that some of the data is not in .zip format which is throwing an error in celeba.py:</p>
<pre><code class="lang-auto">3098it [00:00, 4697741.79it/s]
3098it [00:00, 930932.35it/s]
3103it [00:00, 943931.34it/s]
3098it [00:00, 913138.00it/s]
3098it [00:00, 3096009.96it/s]
3098it [00:00, 803484.65it/s]
Traceback (most recent call last):
  File "train_vqvae.py", line 101, in &lt;module&gt;
    dataset = CelebA(root_path, split='all', transform=transform, download=True)
  File "/vol/bitbucket/hgc19/env/lib/python3.6/site-packages/torchvision/datasets/celeba.py", line 66, in __init__
    self.download()
  File "/vol/bitbucket/hgc19/env/lib/python3.6/site-packages/torchvision/datasets/celeba.py", line 120, in download
    with zipfile.ZipFile(os.path.join(self.root, self.base_folder, "img_align_celeba.zip"), "r") as f:
  File "/usr/lib/python3.6/zipfile.py", line 1131, in __init__
    self._RealGetContents()
  File "/usr/lib/python3.6/zipfile.py", line 1198, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

</code></pre>
<p>Any help would be greatly appreciated.</p>
<p>Thanks in advance.</p>

Caruso: <p>Im not totally sure, but it tries to download the files from Google Drive. Some of the files it can download, if they are small enough (like list_bbox_celeba.txt with the link <a href="https://docs.google.com/u/0/uc?id=0B7EVK8r0v71pbThiMVRxWXZ4dU0" rel="nofollow noopener">https://docs.google.com/u/0/uc?id=0B7EVK8r0v71pbThiMVRxWXZ4dU0</a>), but some files are to big and you get a extra warning page (e.g. for the img_align_celeba.zip <a href="https://docs.google.com/u/0/uc?id=0B7EVK8r0v71pZjFTYXZWM3FlRnM" rel="nofollow noopener">https://docs.google.com/u/0/uc?id=0B7EVK8r0v71pZjFTYXZWM3FlRnM</a>).</p>
<p>I recommend to download the dataset manully from google drive <a href="https://drive.google.com/drive/folders/0B7EVK8r0v71pWEZsZE9oNnFzTm8" rel="nofollow noopener">https://drive.google.com/drive/folders/0B7EVK8r0v71pWEZsZE9oNnFzTm8</a> and use this download folder as the root for the <code>torchvision.datasets.CelebA</code> class call!</p>

Harry_Coppock: <p>Thank you for your speedy reply. That worked great. In the end I used datasets.ImageFolder and utils.data.Dataloader.</p>
<p>Cheers</p>
How to install pytorch from source with GPU

Haider_Ali_Shuvo: <p>I have been just started learning pytorch and it has been difficult journey so far .<br>
First installed pytorch but DLL error.<br>
Next installed pytorch with GPU with pip but my Graphics card wasnt supported .<br>
So next installed pytorch from source but this time it cant detect Cuda .<br>
Can anyone explain How to install Pytorch from source with GPU in windows ?<br>
The official documents seems very unclear for newbiw like me.</p>

peterjc123: <p>What GPU card are you using?</p>
<aside class="quote no-group" data-username="Haider_Ali_Shuvo" data-post="1" data-topic="82699">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/haider_ali_shuvo/48/24642_2.png" class="avatar"> Haider_Ali_Shuvo:</div>
<blockquote>
<p>but this time it cant detect Cuda .</p>
</blockquote>
</aside>
<p>Have you installed CUDA correctly? Which version of CUDA are you using?</p>
<aside class="quote no-group" data-username="Haider_Ali_Shuvo" data-post="1" data-topic="82699">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/haider_ali_shuvo/48/24642_2.png" class="avatar"> Haider_Ali_Shuvo:</div>
<blockquote>
<p>The official documents seems very unclear for newbiw like me.</p>
</blockquote>
</aside>
<p>The build script is hosted at <a href="https://github.com/pytorch/builder/blob/master/windows/build_pytorch.bat" rel="noopener nofollow ugc">https://github.com/pytorch/builder/blob/master/windows/build_pytorch.bat</a>. We should work on a better tutorial, but it is not our top priority.</p>
Splitting an input tensor

Dr_Dumbenstein: <p>Didn’t find “nn.SplitTable” that is available in Torch Lua. Doing this in order to make it goes in a split to split convolution process. What is the equivalent Plz?<br>
line :  seq:add(nn.SplitTable(d+1, 3)).</p>

Caruso: <p>Have you tried the torch.split (<a href="https://pytorch.org/docs/stable/torch.html#torch.split" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.split</a>) or torch.chunk (<a href="https://pytorch.org/docs/stable/torch.html#torch.chunk" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.chunk</a>) function?</p>

Dr_Dumbenstein: <p>I found this sheet: <a href="https://discuss.pytorch.org/t/directory-for-table-layers-in-pytorch/274" class="inline-onebox">Directory for Table Layers in PyTorch</a><br>
Thnx Caruso</p>
Directory for Table Layers in PyTorch

amdegroot: <p>Since it seems like several people (myself included) have been asking about how to get around using Torch Containers in PyTorch, I compiled a list of easy example implementations in PyTorch of most of the Torch Table Containers and show them side by side with their Torch counterparts.  Hopefully this repo clears up table layer uncertainties. (also feel free to let me know if I missed anything)  <a href="https://github.com/amdegroot/pytorch-containers/blob/master/README.md" rel="nofollow noopener">https://github.com/amdegroot/pytorch-containers/blob/master/README.md</a></p>

colesbury: <p>This is great! Thanks for putting this together</p>

amdegroot: <p>Yeah no problem, love doing this stuff, hopefully it helps!</p>

Mata_Fu: <p>Thanks a lot!<br>
I’m confused on this part and can’t find any tutorials.</p>

Dr_Dumbenstein: <p>This is exactly what i was looking for. I was stuck on how to implement a nn.SplitTable (Lua) with pytorch</p>
Meaning of wasserstein distance

god_sp33d: <p>So, I am basically training a GAN with WGAN-gp setup. After I train the critic (lets say 5 times) If I estimate the Wasserstein distance between real and fake samples like (critic(real) - critic(fake)) it gives me a positive real number. After few epochs the Wasserstein distance between becomes negative and goes on decreasing. So, my question is basically what does this positive distance imply ? And also what does this negative value imply (which I got after few epochs) ? So, does that mean one of G or D got too strong? How do we fix this issue ??</p>

vmirly1: <p>In Wassertein GAN, the discriminator does not really play the role of a critic between the real and fake image samples. Instead, it is providing a way to estimate the difference in distribution of real and fake samples.</p>
<p>And it is expected that the discriminator loss decreases over time. But if keeps on decreasing, I suspect that the weights are not clamped. So, you may want to clip the weights of the discriminator within a small range like <code>[-0.01, 0.01]</code> after each update applied to the weights of the discriminator.</p>
<p>Also, this blog-post very nicely describes the Wasserstein distance: <a href="https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html" rel="nofollow noopener">https://lilianweng.github.io/lil-log/2017/08/20/from-GAN-to-WGAN.html</a></p>

god_sp33d: <p>Hi Vahid,</p>
<p>Thanks for you input. I got what you are saying about the estimating the difference between distribution of real and fake. But, what does negative value mean ? Yeah, It is expected than discriminator loss decreases over time as we are reducing the distance between real and fake but here I was referring to critic(real) - critic(fake) (which is computed after critic training in a given epoch and then looking at the distance between real and fake i.e., after something critic has learned)</p>
<p>And also, are you sure about gradient clipping? I am using Improved training of WGAN where it uses gradient penalty unlike WGAN which uses gradient clipping. The blog kinda seems a detailed explanation, thank you I will go through it.</p>

crissallan: <p>Hi, have you figure this out? I think I met the same problem as yours. The loss of the discriminator keeps decrease to very large negative numbers, and the things generated by the generator is a mess.</p>
How can I read the .pt file(which save by python script not by jit script) in C++ by torch::load from libtorch?

bill4278: <p>How can I read the .pt file(which save by python script) in C++ by torch::load from libtorch?</p>
<p>At first, I have trained a model in python, and have already saved the weight file in .pt format.<br>
So is it possible I read this .pt file in C++ using libtorch with torch::load?</p>
<p>Thanks</p>

yf225: <p>It’s currently not possible to read .pt file saved by Python script, because Python script’s <code>torch.save</code> method uses pickling and it’s not understood by C++.</p>
<p>In order to load a pre-trained model into libtorch, the model should be written in TorchScript, saved with <code>model.save(filename)</code> and loaded into libtorch with <code>torch::jit::load()</code>.</p>

bill4278: <p>Thanks a lot, I have loaded the .pt file from torch.jit. Script into my C++ model, but still need to do few changes. Anyway that works, thanks for you attention.</p>

xq114: <p>Is there any plan to support loading a model without translate it into TorchScript?</p>
Add_image() doesn't work well

Landu: <p>i’m using both of save_image() and Summarywriter.add_image()<br>
i made an image from several images. ( N,C,H,W-&gt;C,H<em>N,W</em>N)</p>
<p>but i found that MNIST (1,H,W) works well in both, but color image doesn’t work well<br>
here’s the images<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a56cebf9123e9dc2b70329a517043b45dbac90eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/a56cebf9123e9dc2b70329a517043b45dbac90eb" title="스크린샷, 2020-05-25 07-36-27"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a56cebf9123e9dc2b70329a517043b45dbac90eb_2_690x338.png" alt="스크린샷, 2020-05-25 07-36-27" data-base62-sha1="nBq94UoylXrpkWxxHhmONBOVATh" width="690" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/5/a56cebf9123e9dc2b70329a517043b45dbac90eb_2_690x338.png, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a56cebf9123e9dc2b70329a517043b45dbac90eb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/5/a56cebf9123e9dc2b70329a517043b45dbac90eb.png 2x" data-dominant-color="82756E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷, 2020-05-25 07-36-27</span><span class="informations">808×396 466 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
the left one is using add_image()   (on tensorboard)<br>
and the right one is using save_image()  (on my local folder)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a88ccc3a5f3fbbc16c46a1e28af766ff1670d12b.png" alt="스크린샷, 2020-05-25 07-37-51" data-base62-sha1="o33S7gjbsYLsvgtdwQe0g4jnrQL" width="494" height="341"><br>
and also here is the code</p>
<p>i also try changing the order of RGB channel, but it doesn’t work</p>
<p>Thanks</p>

ptrblck: <p><code>save_image</code> “unnormalizes” the image via:</p>
<pre><code class="lang-python">ndarr = grid.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 0).to('cpu', torch.uint8).numpy()
</code></pre>
<p>I don’t know, if <code>add_image</code> of TensorBoard does the same, but it looks like the colors might be clipped, so you could try to apply the same method as in <code>save_image</code>.</p>

Landu: <p>thx i tried some ways, and found why that didn’t work.<br>
i guess in-place operation was problem.</p>
<p>i gave denorm(image) to each function(save_image,add_image) and it works (i.e. save_image(denorm(image))<br>
but when i gave image which is unnormalized to each function it didn’t (i.e. image=denorm(image), save_image(image))</p>
How to load Tensorboard static event file

saluei: <p>I use tensorboard version 1.15.0 and pytorch 1.5.0,</p>
<p>from torch.utils.tensorboard import SummaryWriter<br>
tns_wr = SummaryWriter(os.path.join(’./log’))<br>
x_val=(epochIndex * SampleCountInEpoch ) + ((batchIndex+1) *  BatchSize)<br>
tns_wr.add_scalar(‘Train/Loss’, AvgLoss,x_val)</p>
<p>Everything is OK and I see diagram in tensorboard when code (training) is in process.<br>
BUT after training finished and I close tensorboard service if I want to refer the saved event file awhile, tensorboard do not show diagram immediately , and it pretend the train code is running again and I must WAIT tensorboard load event file that exist and saved from previous run!!!<br>
why tensorboard do not load previously saved event file immediately ?</p>
GPU vs CPU speed in torch.histc

saluei: <p>I have two functions for calculating entropy in first one  (named: ImageEntropy_gpu) use torch.histc function and run it in GPU (nvidia 2070) and in second one (named: ImageEntropy_cpu) use numpy library and run at CPU (i7-7800x)</p>
<p>def ImageEntropy_gpu(img):<br>
sz=img.view(-1).size()[0]<br>
hist_probability=torch.histc(img.view(-1), bins=256)/sz<br>
nonzero_probability=hist_probability[hist_probability&gt;0]<br>
entropy =-torch.sum( torch.mul(nonzero_probability, torch.log2(nonzero_probability))).item()<br>
return round(entropy,4)</p>
<p>def ImageEntropy_cpu(img):<br>
marg = np.histogramdd(np.ravel(img), bins = 256)[0]/img.size<br>
marg = list(filter(lambda p: p &gt; 0, np.ravel(marg)))<br>
entropy = -np.sum(np.multiply(marg, np.log2(marg)))<br>
return round(entropy,4)<br>
CPU version function (ImageEntropy_cpu) run 20 time FASTER than GPU version (ImageEntropy_gpu)!!!<br>
can someone explain WHY this happen, may be something is wrong in these two function ??</p>

ptrblck: <p>The performance will depend on the workload you are deploying to the device.<br>
While small workloads will be faster on the CPU due to the kernel launch latency on the GPU, you should see a speedup for bigger sizes.<br>
Using this code:</p>
<pre><code class="lang-python">nb_iters = 100

for device in ['cpu', 'cuda']:
    for s in torch.logspace(1, 8, steps=8).int():
        img = torch.randint(0, 256, (s,), device=device).float()
        torch.cuda.synchronize()
        t0 = time.time()
        for _ in range(nb_iters):
            out = torch.histc(img, bins=256)
        torch.cuda.synchronize()
        t1 = time.time()
        print('device {}, size {}, time {}'.format(
            device, s, (t1 - t0)/nb_iters))
</code></pre>
<p>I see these results on an <code>Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz</code> and a <code>V100</code>:</p>
<pre><code class="lang-python">device cpu, size 10, time 8.225440979003906e-06
device cpu, size 100, time 7.455348968505859e-06
device cpu, size 1000, time 1.646280288696289e-05
device cpu, size 10000, time 0.00010733842849731445
device cpu, size 100000, time 0.0010244464874267578
device cpu, size 1000000, time 0.010178050994873046
device cpu, size 10000000, time 0.10184217929840088
device cpu, size 100000000, time 1.036203260421753
device cuda, size 10, time 0.00018221616744995117
device cuda, size 100, time 0.00018065452575683594
device cuda, size 1000, time 0.00018761157989501953
device cuda, size 10000, time 0.00019278764724731446
device cuda, size 100000, time 0.00019084692001342774
device cuda, size 1000000, time 0.00026309967041015625
device cuda, size 10000000, time 0.0009484291076660156
device cuda, size 100000000, time 0.007037463188171386
</code></pre>

saluei: <p>Thank’s , your code and reply is perfect,<br>
I don’t know about “kernel launch latency on the GPU” that is total latency to run kernels on GPU and<br>
data transmission latency between CPU and GPU.<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:"></p>
Imbalance of pos/neg examples within each class with MultiLabelSoftMarginLoss

kedar: <p>Hello. I read a lot of posts about class imbalance, but I am not sure that they are applicable in my case because the majority is talking about the imbalance between classes in a multi-class classification problem, while in my case I am rather concerned about the imbalance of positive and negative examples within each class. The following is a tally of the number of pos/neg examples with each class:</p>
<p>Total number of examples = 4000<br>
class1: n_positive = 948, n_negative = 4000 - 948 = 3052<br>
class2: n_positive = 575, n_negative = 4000 - 575 = 3425<br>
class3: n_positive = 917, n_negative = 4000 - 917 = 3083<br>
class4: n_positive = 672, n_negative = 4000 - 672 = 3328</p>
<p>So we have a lot of rows in the dataset where there is [0,0,0,0]</p>
<p>As you can see there is an imbalance between classes as well. I am using MultiLabelSoftMarginLoss loss function which has a <em>weight</em> parameter to tackle that problem. I can pass to it a tensor of size C with the following values [1/948, 1/575, 1/917, 1/672] (if I thinking correctly?) to get rid of classes imbalance.</p>
<p>But most importantly how to deal with the issue of imbalance within each class. For example, there are 948 only positive examples for class 1 and 3052 negative examples. I guess that I should weight them as well. How to tell the loss function to not only penalize the most frequently occuring class but also most frequently occuring value (0/1) within each class, otherwise model will always predict [0,0,0,0].</p>
<p>Maybe in that case  <code>BCEWithLogitsLoss</code> with <em>pos_weight</em> = [3052/948, 3425/575, 3083/917, 3328/671] and <em>weight</em> = [1/948, 1/575, 1/917, 1/672] (as in MultiLabelSoftMarginLoss)? What do you think about it?</p>
<p>Any advice would be very appreciated.</p>

ptrblck: <aside class="quote no-group" data-username="kedar" data-post="1" data-topic="82686">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/7bcc69/48.png" class="avatar"> kedar:</div>
<blockquote>
<p>Maybe in that case <code>BCEWithLogitsLoss</code> with <em>pos_weight</em> = [3052/948, 3425/575, 3083/917, 3328/671] and <em>weight</em> = [1/948, 1/575, 1/917, 1/672] (as in MultiLabelSoftMarginLoss)? What do you think about it?</p>
</blockquote>
</aside>
<p>I was also thinking about this approach while reading your post, so I would at least try it and see, if your training benefits from it.</p>
Run pretrained model in parallel

tonyrewin: <p>If there any good practices to parallel model prediction calculations, not training. So if we have a pretrained pth-model how to run it on a few GPUs? Is there a doc with this specific topic covered?</p>

tymokvo: <p>You should have a look at the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel" rel="nofollow noopener"><code>nn.DataParallel</code></a> and <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel" rel="nofollow noopener"><code>nn.distributed.DistributedDataParallel</code></a> docs. Both are fairly easy objects to work with. It will basically split your input in the batch dimension across the ids of the GPUs that you pass at initialization.</p>
<p>Note that some model architectures are not able to be parallelized across devices.</p>
<p>Also, if you are just running inference, you may not see any benefit to multi-GPU parallelization. Or even using a GPU at all. You might try running the model after a call to <code>model.eval()</code> if you are experiencing performance issues.</p>

tonyrewin: <blockquote>
<p>It will basically split your input in the batch dimension across the ids of the GPUs that you pass at initialization.</p>
</blockquote>
<p>So this code is enough:</p>
<pre><code class="lang-auto">    model = torch.nn.Module(options)

    ...

    if torch.cuda.is_available():
        ids = [i for i in range(torch.cuda.device_count())]
        model = torch.nn.DataParallel(model, device_ids=ids).cuda()
        os.environ['CUDA_VISIBLE_DEVICES'] = ids.join(',')[:-1]
        print("Using ", len(ids), " GPUs!")

    ...

    model_results = model(input)
</code></pre>
<blockquote>
<p>Note that some model architectures are not able to be parallelized across devices.</p>
</blockquote>
<p>What does it depend on?</p>

tonyrewin: <pre><code class="lang-auto">    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DataParallel:
        Missing key(s) in state_dict: "module.conv_first.weight", 
really long list here...
</code></pre>
<p>Should we prepare the code anyhow for wrapping in DataParallel?</p>

ptrblck: <p><code>nn.DataParallel</code> adds a <code>.module</code> attribute to the model, so that you might see these key errors while trying to load a <code>state_dict</code> from a plain PyTorch model.<br>
You could either add/remove the <code>.module</code> keys manually or store and load the <code>state_dict</code> using the plain model without the <code>nn.DataParallel</code> wrapper.<br>
To store the <code>state_dict</code> you would use <code>torch.save(model.module.state_dict(), path)</code>, while you could just load the <code>state_dict</code> before wrapping the model into <code>nn.DataParallel</code>.</p>
Questions about the Transformers NER usage example

lazy_like_a_fox: <p>I’m just learn Pytorch and was following the NER example here <a href="https://huggingface.co/transformers/usage.html#named-entity-recognition" rel="nofollow noopener">https://huggingface.co/transformers/usage.html#named-entity-recognition</a> that produces this output:</p>
<pre><code class="lang-auto">[
    {'word': 'Hu', 'score': 0.9995632767677307, 'entity': 'I-ORG'},
    {'word': '##gging', 'score': 0.9915938973426819, 'entity': 'I-ORG'},
    {'word': 'Face', 'score': 0.9982671737670898, 'entity': 'I-ORG'},
    {'word': 'Inc', 'score': 0.9994403719902039, 'entity': 'I-ORG'},
    {'word': 'New', 'score': 0.9994346499443054, 'entity': 'I-LOC'},
    {'word': 'York', 'score': 0.9993270635604858, 'entity': 'I-LOC'},
    {'word': 'City', 'score': 0.9993864893913269, 'entity': 'I-LOC'},
    {'word': 'D', 'score': 0.9825621843338013, 'entity': 'I-LOC'},
    {'word': '##UM', 'score': 0.936983048915863, 'entity': 'I-LOC'},
    {'word': '##BO', 'score': 0.8987102508544922, 'entity': 'I-LOC'},
    {'word': 'Manhattan', 'score': 0.9758241176605225, 'entity': 'I-LOC'},
    {'word': 'Bridge', 'score': 0.990249514579773, 'entity': 'I-LOC'}
]
</code></pre>
<ol>
<li>Shouldn’t the first entity be labeled ‘B-ORG’?</li>
<li>Do I need to write my own function to stitch split words like ‘Hugging’ back together or is there something prebuilt for that?</li>
</ol>

ptrblck: <p>This question seems to be specific to the documentation of Huggingface’s Transformers.<br>
CC <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> to answer this specific question or to refer to some documentation.</p>
Dataloader worker thread using too much memory with mp.set_start_method("spawn")

kmaeng: <p>I am setting multiprocessing start method as “spawn”, using:<br>
<code>mp.set_start_method("spawn")</code><br>
and then using dataloader.</p>
<p>However, when I try to increase the num_workers for the dataloader, the memory bloats.<br>
This makes sense because I think what is happening is that each dataloader worker thread<br>
now became a spawned process (maybe even with its own python interpreter) and copies the<br>
entire memory instead of sharing, etc.</p>
<p>However, as you can imagine, this is not what I wanted. I am doing Hogwild! by referencing<br>
the example from here: <a href="https://github.com/pytorch/examples/blob/master/mnist_hogwild/train.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/mnist_hogwild/train.py</a><br>
I want the start method to be “spawn” for each hogwild threads, but not for the dataloader worker threads. Is there a way I can use the good old “fork” for the dataloader threads and only use “spawn” for the hogwild threads, so that using many dataloader worker threads does not bloat the memory?<br>
Or if I am understanding the problem incorrectly, let me know!</p>
<p>Any suggestions are highly appreciated.<br>
Thank you in advance!</p>
Why is bias=False in DCGAN tutorial for first part of discriminator?

collinarnett: <p>Why is bias turned off in the first Conv2d layer of the discriminator in the DCGAN tutorial when there is no batchnorm2d layer following it?</p>
<pre><code class="lang-auto">class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is (nc) x 64 x 64
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf) x 32 x 32
            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*2) x 16 x 16
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*4) x 8 x 8
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*8) x 4 x 4
            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, input):
        return self.main(input)
</code></pre>
<p>AFAIK you should only turn the bias off if conv2d is followed by batchnorm.</p>
Why downscale 64x64 images by a factor of 100?

collinarnett: <p>I’m reading this paper on a DCGAN and the authors write:</p>
<blockquote>
<p>For upsampling by the generator, we use strided convolution transpose operations instead of pixel<br>
shuffling [43] or interpolation, as we found this to work better in practice. We set the slope of the<br>
LeakyReLU units to 0.2 and the dropout rate to 0.1 during training. Importantly, we did not normalize<br>
input maps but scaled them down by a constant factor.</p>
</blockquote>
<p>They also mention it in the model architecture:</p>
<pre><code class="lang-auto">------------------64 GAN------------------
down-scale factor = 100

--Generator--
nz = 100
ConvTranspose2d( 512, 4, 1, 0)
BatchNorm2d(512)
LeakyReLU(0.2),
ConvTranspose2d(256, 4, 2, 1) 
BatchNorm2d(256)
LeakyReLU(0.2)
ConvTranspose2d(128, 4, 2, 1) 
BatchNorm2d(128)
LeakyReLU(0.2)
ConvTranspose2d(64, 4, 2, 1) 
BatchNorm2d(64)
LeakyReLU(0.2)
ConvTranspose2d(1, 4, 2, 1) 
Clamp(&gt;0)
Enforce Symmetry
</code></pre>
<p>Wouldn’t this mean that the image goes from 64x64 to 6x6? How can the generator/discriminator ever produce images that resemble this if they’re pulling from such a small image?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f7cc21ef3436517dc0165b80303a535bb95e472.png" alt="image" data-base62-sha1="2d0seKgTikTe7TwWiO2gzjflA0W" width="432" height="288"></p>

ptrblck: <p>I don’t think “downscaling” is referring to the spatial size, but to a multiplicative factor.</p>
<p>Here normalization is mentioned:</p>
<aside class="quote no-group" data-username="collinarnett" data-post="1" data-topic="82627">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/collinarnett/48/23242_2.png" class="avatar"> collinarnett:</div>
<blockquote>
<p>Importantly, we did not normalize<br>
input maps but scaled them down by a constant factor.</p>
</blockquote>
</aside>
<p>which seems to point towards using a constant factor for the downscaling via:</p>
<pre data-code-wrap="python"><code class="lang-python">act = act * 1/factor
</code></pre>

collinarnett: <p>Ok that makes more sense thank you!</p>
Scheduler for triggering user defined functions ( no necessary optimizer )

Ali_Mirzaeyan: <p>Hi, is it possible to use PyTorch scheduler for triggering functions other than the optimizer ?<br>
for example, here is a typical practice of schedulers</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min')
for epoch in range(10):
    train(...)
     val_loss = validate(...)
     # Note that step should be called after validate()
    scheduler.step(val_loss)
</code></pre>
<p>but I want the best practice for having something like this:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(**my_func**, 'min')
for epoch in range(10):
    train(...)
     val_loss = validate(...)
     # Note that step should be called after validate()
    scheduler.step(val_loss)
</code></pre>

ptrblck: <p>The scheduler expects the optimizer as an input to be able to manipulate the learning rate of it.<br>
What’s your use case and how would you like to change the learning rate of the optimizer without passing it?</p>

googlebot: <p>there is <a href="https://pytorch.org/ignite" rel="nofollow noopener">ignite</a> if you need flexible callbacks</p>

Ali_Mirzaeyan: <p>I don’t want to change the learning rate, I just want to use the scheduler signal for invoking another function. for example, whenever <code>ReduceLROnPlateau</code> decides to change the learning rate ( because val_loss not going down anymore ) I want use this as an indicator for invoking a function or changing a part of model, not necessarily the learning rate.</p>
<pre><code class="lang-auto">def my_func():
    print('hi')
</code></pre>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min')
for epoch in range(10):
    train(...)
     val_loss = validate(...)
     #scheduler.step(val_loss)
     if scheduler decide to change the learning rate :
          my_func()
</code></pre>

ptrblck: <p>In that case you might want to use <a class="mention" href="/u/googlebot">@googlebot</a>’s suggestion of using callbacks from higher-level wrappers or alternatively derive your custom scheduler from the desired scheduler class and manipulate the <code>step()</code> method as well as its <code>__init__</code>.</p>
How to use a tensor of indices to copy values between two other tensors

darylnak: <p>Suppose I have the following tensors:</p>
<pre><code class="lang-auto">x = 
[[1,2,3],
 [4,5,6],
 [7,8,9]]

y = 
[[11,21,31],
 [41,51,61],
 [71,81,91]]

z=
[[1],
 [2],
 [0]]
</code></pre>
<p>Where <code>x,y</code> are both <code>(h,w)</code> and <code>z</code> is <code>(h,1)</code>. I want to create a new tensor <code>k</code> that it is a copy of <code>x</code> except I want to copy the <code>z[i][0]</code> column at row <code>i</code> from <code>y</code> into row <code>i</code> at column <code>z[i][0]</code> in <code>x</code>. Thus, using the tensors from above, <code>k</code> should be:</p>
<pre><code class="lang-auto">k = 
[[1,21,3],
 [4,5,61],
 [71,8,9]]
</code></pre>
<p>I though using <code>k = x.scatter_(1, z, y)</code> would accomplish this, but it seems to only use values from the first column of <code>y</code>, so I obviously don’t know how <code>scatter_</code> works. How can I leverage PyTorch to accomplish this so I can avoid using a for-loop?</p>

ptrblck: <p>A combination of <code>gather</code> and <code>scatter_</code> should work:</p>
<pre><code class="lang-python">x = torch.tensor([[1,2,3],
                  [4,5,6],
                  [7,8,9]])

y = torch.tensor([[11,21,31],
                  [41,51,61],
                  [71,81,91]])

z = torch.tensor([[1],
                  [2],
                  [0]])


k = x.scatter_(1, z, y.gather(1, z))
print(k)
&gt; tensor([[ 1, 21,  3],
          [ 4,  5, 61],
          [71,  8,  9]])
</code></pre>

darylnak: <p>This is awesome! Didn’t even know about gather.</p>
Get probabilities from a model with log softmax

Ali_Amiri: <p>Hey,<br>
I train a model with log_softmax activation in the last layer,<br>
then while evaluating the model, I should only print the model(x) to get the real probs, right?<br>
so if it’s that why Keras only takes softmax instead of log softmax and it totally works?</p>

ptrblck: <p>PyTorch uses <code>log_softmax</code> instead of first applying <code>softmax</code> and later <code>log</code> for numerical stability as described in the <a href="https://en.wikipedia.org/wiki/LogSumExp">LogSumExp trick</a>.</p>
<p>If you want to print the probabilities, you could just use <code>torch.exp</code> on the output.</p>

Ali_Amiri: <p>thank you for the reply</p>
<p>so it means that if i use log_softmax, then i should call exp on the output - exp(out)<br>
otherwise the argmax will return incorrect output for classifying the data ?</p>

Caruso: <p>If you just want the argmax you can keep the log_softmax and use argmax after that, but if you want the correct softmax probabilities you should use exp(out)</p>

Ali_Amiri: <p>thank you for your answer</p>
Setattr and getattr methods

Shubhankar: <h2>
<img src="https://discuss.pytorch.org/images/emoji/apple/rocket.png?v=9" title=":rocket:" class="emoji" alt=":rocket:"> Feature</h2>
<p>It would be really helpful if we can implement <strong>setattr</strong> methods in respective classes (say transforms.transform.Normalize) so that attributes of objects can be set recursively say for example by using a loop.</p>
<h2>Motivation</h2>
<p>It would be helpful to loop through multiple attributes and set them on the fly in order to compare performance among various parameters and attributes. Say for example if we take transformations  and want to loop through multiple sets of them say set1: Grayscale, ToTensor, Normalize with a, b, c parameters and set2: with d, e, f parameters or some other transformations altogether. Currently I am not aware of any such implementations.</p>

ptrblck: <p>You should be able to directly access the attributes of the transformations, as they are implemented as Python classes:</p>
<pre><code class="lang-python">norm = transforms.Normalize(mean=torch.tensor([0.5, 0.5, 0.5]),
                            std=torch.tensor([0.5, 0.5, 0.5]))
print(norm)
norm.mean = torch.tensor([1.5, 1.5, 1.5])
print(norm)
</code></pre>
<p>The <code>setattr</code> method should do the same, no?</p>

Shubhankar: <p>It worked. I was looking for _<em>setattr</em>_ and _<em>getattr</em>_ methods in the code but since I couldn’t find them I thought they weren’t implemented.</p>
Change input size of pretrained model mobilenetv2

Wei_Wong: <p>I want to change the input size of mobilenet v2 as the backbone of object detector, like 320x320, 448 x448, and fine tune it,  I wonder if it is possible? Will it hugely affect the performance?</p>

Caruso: <p>It it totally possible to use those new input shapes, because the forward method calls a adaptive_average_pool function before the Linear layer is called and therefor there should be no shape issues!<br>
The performance will be a bit slower, because there are more dot products being calculated, but it should not be magnificent!</p>

Wei_Wong: <p>Thank you for your answer!</p>
Dimension questions about ToTensor()

JamryZzz: <p>hello,i have a problem.i load pics in grayscale use:<br>
<strong>Image.open(filepath).convert(‘L’)</strong><br>
then transform it to Tensor use:<br>
<strong>ToTensor()</strong><br>
in train mode,i found the result have 3 channels like <strong>[b,3,h,w]</strong><br>
but in test mode, i found the result have 1 channel like <strong>[b,1,h,w]</strong><br>
the transform code is totally the same, just <strong>ToTensor()</strong></p>
<p>how can i handle this?</p>

Caruso: <p>Hi,</p>
<p>can you show us a bit of you code, because this seems strange?</p>
Pytorch Basics Unable to understand padding

mehra-deepak: <p>Hello all ,i’m new to PyTorch can anyone please how do we choose the padding is there any formula or like what number should i decide for padding while making the architecture of CNN.</p>
<p>Below are the two examples of padding , in which 2 numbers have been taken into account but i don’t know how.</p>
<p>Thank u in advance.</p>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0)
</code></pre>
<pre><code class="lang-auto">self.conv1 = nn.Conv2d(1, 16, 2, stride=2)
</code></pre>

Caruso: <p>Hi,</p>
<p>the value for padding is up to you. If you use a normal convolution without padding and stride=1, the output will be of the size <code>output = (input - kernel) + 1</code> and your output gets smaller after the convolution (except <code>kernel = 1</code>). This is exactly what would happen in your first example:</p>
<aside class="quote no-group" data-username="mehra-deepak" data-post="1" data-topic="82690">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mehra-deepak/48/30471_2.png" class="avatar"> mehra-deepak:</div>
<blockquote>
<p><code>self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0)</code></p>
</blockquote>
</aside>
<p>To counteract this, padding can be applied. This way you add a border of some values (e.g. 0) around your input and increases its size by two times the padding value ( <code>input  = input  + 2*padding</code>).<br>
If you want the output to be the same shape as the input, the formula would be<br>
<code>padding = (kernel - 1) // 2</code>.</p>
<p>Maybe this animation will help you understand it better: <a href="https://github.com/vdumoulin/conv_arithmetic" class="inline-onebox" rel="noopener nofollow ugc">GitHub - vdumoulin/conv_arithmetic: A technical report on convolution arithmetic in the context of deep learning</a><br>
Or this pdf containing some visualizations and formulas : <a href="https://arxiv.org/pdf/1603.07285.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1603.07285.pdf</a><br>
<br>
<br>
Your second example( <code>self.conv1 = nn.Conv2d(1, 16, 2, stride=2)</code>), has a kernel_size = 2,<br>
stride = 2  and padding = 0. In this case your kernel ‘moves’ 2 positions per convolution, which is exactly the kernel_size, resulting in halving the original input size, so that your <code>output = input / 2</code></p>

mehra-deepak: <p>Thanks Caruso that clarifies a lot, you are the best.</p>
GAN artifacts in image corners

s1ddok: <p>Hello, pytorch community!</p>
<p>I’m currently training GAN models and totally satisfied with it’s result, but it has a very annoying bug. In corners of the image (mostly in bottom right, but others appear sometimes as well) I see strange pattern that model generates. Those artifacts take a small portion of 1024x1024 (maybe 1/16th) image but obviously ruin the perception.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/296cb7bd50eb833e15f2b2079464b5ca98be7bf9.png" alt="Снимок экрана 2020-05-24 в 16.49.20" data-base62-sha1="5Usvbmv2c85gyy4tU1NX2mHdiSl" width="222" height="142"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c149be4baf0acc4182b7017d1c6c2a4a9eceb679.png" alt="image" data-base62-sha1="rzU93UxDcu6nXS79kOZMUAxpcLv" width="202" height="170"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50a55a7d56c1c2781d4e0b8457ea1f14dc947d11.jpeg" alt="image" data-base62-sha1="bvqsSCuCIowhoAGfw2BjyXQohrz" width="436" height="284"></p>
<p>I’m using Pix2PixHD-like network with MultiscaleDiscriminator.</p>
<p>How one should debug it? I think zero-padding may be involved here, but I am afraid that reflect padding will lose spatial info. Are there any recommendations on how to fight this? I think that pointing discriminator’s attention to corners somehow may work, but I have no idea how to approach that</p>
CUDA_tensor_apply3 missing in CUDAApplyUtils.cuh

sporadicCactus: <p>After switching from pytorch 1.4.0 to 1.5.0 I noticed the CUDA_tensor_apply3 function is gone from CUDAApplyUtils.cuh, while the description of the latter says</p>
<blockquote>
<p>The following CUDA_tensor_applyN (where N currently can be 1, 2, 3, or 4) functions apply a pointwise operator to N tensor(s).</p>
</blockquote>
<p>Is this change intended? If so, what should I use instead of CUDA_tensor_apply3?</p>

ptrblck: <p><code>CUDA_tensor_apply3</code> was removed in favor of <code>TensorIterator</code>.<br>
An example migration is shown in <a href="https://github.com/pytorch/pytorch/pull/34025/files">this PR</a>.</p>

sporadicCactus: <p>Thank you. I stumbled into another problem: my fresh installation of pytorch lacks <code>ATen/native/cuda</code> folder, which contains headers with <code>gpu_kernel</code> function used together with <code>TensorIterator</code> in the example you provided. I installed pytorch via <code>conda install -c pytorch pytorch torchvision --cudatoolkit=10.2</code>.</p>

ptrblck: <p>Were you using <code>CUDA_tensor_apply3</code> in a custom CUDA extension or what is your use case?</p>

sporadicCactus: <p>That’s right, I was using it when implementing custom activation functions. Often there is a substantial gain in performance compared to python version.</p>
Backpropogation for many minibatches

kiru883: <p>Hello, i try to implement fastRCNN during getting outputs(forward method) several minipackages are formed from one package (picture) and for each you need to find gradients and update weights, there are tensors at the output where each row is a output of the one mini-batch</p>
<p>def forward:</p>
<pre><code>    cnn_out = self.cnn_block(Ximg)
    ###########################
    ###########################  ROI POOL
    scaled_rois = []
    shape_x, shape_y = cnn_out.size(3), cnn_out.size(2)
    for roi_pack in Xroi:
        roi_pack[:, 0], roi_pack[:, 1], roi_pack[:, 2], roi_pack[:, 3] = torch.floor(roi_pack[:, 0]*shape_x),  \
                                                                         torch.floor(roi_pack[:, 1]*shape_y),  \
                                                                         torch.ceil(roi_pack[:, 2]*shape_x)-1, \
                                                                         torch.ceil(roi_pack[:, 3]*shape_y)-1
        scaled_rois.append(roi_pack)

    roi_out = roi_pool(cnn_out, scaled_rois, output_size=self.rois_output_size)
    ###########################
    ###########################
    fc_inp = roi_out.view(1, Xroi[0].size(0), cnn_out.size(1)*self.rois_output_size[0]*self.rois_output_size[1])
    fc_out = self.fc_block(fc_inp)
    ###########################
    classif_out = self.classifier_output(fc_out)
    ###########################
    regress_out = self.regressor_output(fc_out)
    return classif_out, regress_out
</code></pre>
<p>update weights (???):</p>
<pre><code>        loss_clf = -torch.log(out_clf+clip_coef)
        loss_reg = labels[0].flatten()*torch.sum(torch.abs(labels[1] - out_reg), 2)
        losses = (loss_clf.flatten() + loss_reg).flatten()

        # backprop
        for loss in losses:
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
</code></pre>
<p>I founded 1 loss for each row in classif_out and regress_out, how update weights for each loss?<br>
Thanks for you attention!</p>

TinfoilHat0: <p>Hi,</p>
<p>I’m not sure if that’s what you are asking but you can do a single weight update for bunch of different batches by simply delaying zeroing the gradients.</p>

kiru883: <p>Not, is not working, question is opened.</p>
Tile/repeat different number of times

justanhduc: <p>Hello. How can I tile/repeat a tensor in such a way that each value is tiled/repeated a different number of times other than looping then concatenating?<br>
For e.g., from <code>torch.Tensor([0, 1])</code>, how can I tile the first value two times and the second three times to get <code>torch.Tensor([0, 0, 1, 1, 1])</code>. Thanks in advance!</p>

ptrblck: <p><code>torch.repeat_interleave</code> should work:</p>
<pre><code class="lang-python">x = torch.tensor([0, 1])
print(torch.repeat_interleave(x, torch.tensor([2, 3])))
&gt; tensor([0, 0, 1, 1, 1])
</code></pre>

justanhduc: <p>Awesome thanks. I thought <code>torch.repeat_interleave</code> worked for only integers.</p>
Split an input tensor

Dr_Dumbenstein: <p>New to Pytorch, Didn’t find “nn.SplitTable” that is available in Torch Lua. What is the equivalent Plz?<br>
line :  seq:add(nn.SplitTable(d+1, 3))</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/splitting-an-input-tensor/82655/2">here</a> with a potential answer.</p>
Use torch.jit.trace to wrap all the training code

Chengji_Yao: <p>Hello, guys. I seems that torchscript cannot wrap all the training code. Is there any plan to let jit.trace / jit.script to support <em>loss.backward()</em> and <em>optmizer.step()</em> in the future?</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import torch.optim as optim</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(1, 6, 3)
    self.conv2 = nn.Conv2d(6, 16, 3)
    self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension 
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
    x = F.max_pool2d(F.relu(self.conv2(x)), 2)
    x = x.view(-1, self.num_flat_features(x))
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

def num_flat_features(self, x):
    size = x.size()[1:]  # all dimensions except the batch dimension
    num_features = 1
    for s in size:
        num_features *= s
    return num_features
</code></pre>
<p>net = Net()<br>
target = torch.randn(1, 10)<br>
data = torch.randn(1, 1, 32, 32)<br>
optimizer = optim.SGD(net.parameters(), lr=0.01)</p>
<p><span class="mention">@torch.jit.script</span><br>
def train_net(data, target):<br>
out = net(data)<br>
criterion = nn.MSELoss()<br>
loss = criterion(out, target)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
print(loss)</p>
<p>for _ in range(10):<br>
train_net(data, target)</p>
</blockquote>