akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
The order of param_group in optim.param_groups

username: sudonto
cooked: <p>Hi experts, I am new to Pytorch. I am fine tuning ResNet50 model using my own dataset. At first, I optimize only paramaters in “layer4” block + fc layer by passing only these parameters to SGD(). After several epochs, I want to optimize parameters in “layer3” + “layer4” + fc. Notice that I add “layer3” block. I accomplish this using <code>optim.add_param_group()</code>. So, now parameters  of “layer3” come AFTER “layer4” block and fc in <code>optim.param_groups()</code>.</p>
<p>My question is whether the order of param_group in optim.param_groups matter?Thank you.</p>

username: ptrblck
cooked: <p>No, the order doesn’t matter, since all <code>param_groups</code> will be sequentially updated as seen <a href="https://github.com/pytorch/pytorch/blob/26928b164f3e43a48bc57fc41ca277583501054d/torch/optim/sgd.py#L88">here</a>.</p>
mAP function for TORCHVISION OBJECT DETECTION FINETUNING TUTORIAL

username: bubu_ka
cooked: <p>Hi I follow the TORCHVISION OBJECT DETECTION FINETUNING TUTORIAL:<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a> to finish prediction. I want to compute the box mAP of my test data(about 100). How to compute the mAP for test data? Is there a function to compute it?</p>

username: ptrblck
cooked: <p>The tutorial shows the IoU metric at the end using the <a href="https://github.com/pytorch/vision/blob/f71316fa02d9313c3053818ba3b2e66c1da0215c/references/detection/coco_eval.py#L19">CocoEvaluator</a>. Could you just reuse the <code>evaluate</code> method or the linked class?</p>

username: bubu_ka
cooked: <p>thanks for your answer. I want to predict my a new test data, and get prediction box data, and mAP result of prediction.</p>
Output linear layer

username: JA_code
cooked: <p>I have my input image which is 128 x 96 size, and it is fed to the net, after a set of layers, I want to output the image of the same size.</p>
<p>My last layer now is ConvAct, and if to check the size of tensor, it is (10, 1, 96, 128), so to have a linear layer which would return me 128*96 image - what input size I should use?</p>
<p>(12288,…)?</p>

username: ptrblck
cooked: <p>Would you like to return the output image shape directly from the linear layer or would you like to reshape the output to <code>[batch_size, 1, 128, 96]</code>?</p>
<p>You could use conv layers only, which could return the desired shape.<br>
If you really want to use a linear layer, you could use:</p>
<pre><code class="lang-python">lin = nn.Linear(1*96*128, 1*128*96)

act = torch.randn(10, 1, 96, 128) # your activation in the forward pass
out = lin(x)
out = out.view(out.size(0), 1, 128, 96)
return out
</code></pre>
<p>Could you explain, why the spatial dimensions are transposed? It this on purpose or a typo/error?</p>
Size of a tensor (trivial question)

username: nasant
cooked: <p>Hello,</p>
<p>Does the tuple of ints size (3,) correspond to a tensor of size 3?<br>
If so, why for example in the<br>
torch.randint(3, 5, (3,))<br>
I cannot call it like<br>
torch.randint(3, 5, 3)?</p>
<p>What exactly does the (3,) represent?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>It does stand for a tensor of <code>torch.Size([3])</code>.<br>
I think it makes it a bit easier in the backend to always pass tuples in unpacking the dimensions.</p>
How to plot training and test accuracy of CNN

username: Asad_Khan
cooked: <p>I am facing problem regarding plotting of training loss and test loss of Cnn.Can anyone help me in this regard</p>

username: ebarsoum
cooked: <p>Can you provide more detail about the issue?</p>

username: Asad_Khan
cooked: <p>Thanks for responding.I am done with it .now stuck in confusion matrix</p>
How do I convert code written in TensorFlow to PyTorch?

username: rostom007
cooked: <p>Hi</p>
<p>I want to convert a Restricted boltzman machine model (code in python) from TensorFlow to PyTorch. Is there any tool or software or method that can be helpful?</p>

username: smth
cooked: <p>there aren’t any automated tools that will help to do this (that I know of)</p>
How to use the transformers Q&A pipeline for a large corpus

username: navicstein
cooked: <p>Hello all happy week ahead, I have a little question about expanding <code>SQUaD</code> model used by huggingface in the transformers DistilBERT model, according to <img src="https://discuss.pytorch.org/images/emoji/apple/point_down.png?v=9" title=":point_down:" class="emoji" alt=":point_down:"><br>
<a href="https://huggingface.co/transformers/usage.html#extractive-question-answering" class="onebox" target="_blank" rel="nofollow noopener">https://huggingface.co/transformers/usage.html#extractive-question-answering</a></p>
<p>Am tokenizing the question and the <code>context</code> differently so that the context tensors can be loaded once (this comes handy when deploying to production and servered via a REST API, I don’t have to tokenize the context passage everytime an endpoint is called) However, those models are designed to find answers within rather small text passage, and my context doesn’t directly fit into the models “vocab_size”, I need help with how to go about feeding the net with a very large “context” (about 200 pages in one “.txt” file ( like a bible.txt file))</p>
<h2>what I’ve searched👇</h2>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/deepset-ai/haystack/" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:600/396;"><img src="https://repository-images.githubusercontent.com/221654678/b209d200-11f1-11ea-8837-90c43839b572" class="thumbnail" width="600" height="396"></div>

<h3><a href="https://github.com/deepset-ai/haystack/" target="_blank" rel="nofollow noopener">deepset-ai/haystack</a></h3>

<p>:mag: Transformers at scale for question answering &amp; search - deepset-ai/haystack</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>But I would appreciate if anyone have a “light weight” example of how this <img src="https://discuss.pytorch.org/images/emoji/apple/point_up.png?v=9" title=":point_up:" class="emoji" alt=":point_up:"> can be achieved</p>
Issue regarding channel size in DCGAN for grayscale image

username: nalin_thoummala
cooked: <p>I tried to modify this example code <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html</a> of DCGAN for my own dataset. The exmple code is for RGB image, but my data is grayscale image. Therefore, I set the value of number of channels ‘nc’ as 1. However, when I tried running the program, i got thid error:<br>
RuntimeError: Given groups=1, weight of size 64 1 4 4, expected input[128, 3, 64, 64] to have 1 channels, but got 3 channels instead. I do know that its a problem about number of channels but I have no idea what else should I change to solve this error.</p>
<p>Any help would be appreciated!</p>

username: mathematics
cooked: <p>May be when making first conv layer you accidentally written 3 instead of one.</p>

username: nalin_thoummala
cooked: <p>Hi, i did specify it as 3. this is my code:<br>
<span class="hashtag">#i</span> set the channel as 1<br>
nc = 1<br>
<span class="hashtag">#and</span> this is the model for discriminator<br>
class Discriminator(nn.Module):<br>
def <strong>init</strong>(self, ngpu):<br>
super(Discriminator, self).<strong>init</strong>()<br>
self.ngpu = ngpu<br>
self.main = nn.Sequential(<br>
# input is (nc) x 64 x 64<br>
nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),<br>
nn.LeakyReLU(0.2, inplace=True),<br>
# state size. (ndf) x 32 x 32<br>
nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),<br>
nn.BatchNorm2d(ndf * 2),<br>
nn.LeakyReLU(0.2, inplace=True),<br>
# state size. (ndf<em>2) x 16 x 16<br>
nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),<br>
nn.BatchNorm2d(ndf * 4),<br>
nn.LeakyReLU(0.2, inplace=True),<br>
# state size. (ndf</em>4) x 8 x 8<br>
nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),<br>
nn.BatchNorm2d(ndf * 8),<br>
nn.LeakyReLU(0.2, inplace=True),<br>
# state size. (ndf*8) x 4 x 4<br>
nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),<br>
nn.Sigmoid()<br>
)</p>
<pre><code>def forward(self, input):
    return self.main(input)</code></pre>

username: mathematics
cooked: <p>I think , as err says,</p>
<p>model wants input of size</p>
<pre><code class="lang-auto">128, 3, 64, 64 # ( batch, channel, h, w )
</code></pre>
<p>but you are passing</p>
<pre><code class="lang-auto">64 1 4 4 # ( batch, channel, h, w ) 

</code></pre>
<p>please print those discriminator values and debug it , where is error coming from .<br>
Also please use proper code formatting while posting.</p>
<p>Welcome Here <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:">.<br>
If not solved, moderators will it , please calm.</p>

username: cskarthik7
cooked: <p>Try by adding transforms.Grayscale( num_output_channels=1) in your tranform section and see if any error is coming</p>
Can backward function decrease memory of gpu?

username: sumching51
cooked: <pre><code class="lang-auto">def backward_theta(self, x, y):
        self.vgg16.train()
        loss, prob_trees = self.get_loss(x, y)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
</code></pre>
<p>This code above can run normally, but the code below raise the error “CUDA out of memory”. Is something wrong in the else place?</p>
<pre><code class="lang-auto">def backward_theta(self, x, y):
        self.vgg16.train()
        loss, prob_trees = self.get_loss(x, y)
        self.optimizer.zero_grad()
        #loss.backward()
        self.optimizer.step()
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>The backward pass will clear all the buffers saved in the computational graph. So yes it can do that.<br>
If you want to do the same, you need to make sure all references to the computational graph are gone. so <code>del loss</code> and <code>del prob_trees</code> here most likely.</p>

username: sumching51
cooked: <p>Thx a lot!!!</p>
Pytorch RuntimeError: expected device cpu but got device cuda:0

username: xPetricko
cooked: <p>Hello, i am doing someting wrong but i dont know what and i get this runtime error.</p>
<blockquote>
<p>d_r = [T.sum(T.FloatTensor([self.gamma**i for i in range(r[j:].size(0))])<br>
RuntimeError: expected device cpu but got device cuda:0</p>
</blockquote>
<p>Codes is:</p>
<pre><code class="lang-auto">
s = T.tensor(self.buffer['s'], dtype=T.double).to(self.device)
a = T.tensor(self.buffer['a'], dtype=T.double).to(self.device).view(-1, 1)
r = T.tensor(self.buffer['r'], dtype=T.double).to(self.device)
s_n = T.tensor(self.buffer['s_n'], dtype=T.double).to(self.device)
d = T.tensor(self.buffer['d'], dtype=T.double).to(self.device).view(-1, 1)

d_r = [T.sum(T.FloatTensor([self.gamma**i for i in range(r[j:].size(0))])\
             * r[j:]) for j in range(r.size(0))]
</code></pre>
<p>Can someone help me what i am doing wrong?</p>

username: ibro45
cooked: <p>could you share a minimal code that the others can just copy and paste to reproduce the error?</p>

username: xPetricko
cooked: <p>There is link to GitHub repo<br>
<a href="https://github.com/xPetricko/A2C-pytorch" rel="nofollow noopener">https://github.com/xPetricko/A2C-pytorch</a></p>
<p>But it needs packages for Gym to run in it.</p>
<p>The runtime error is from file a2c.py</p>
Strange contrastive loss function

username: lycaenidae
cooked: <p>I know that loss function does not seem true but can we infer something related to what the wrong point is? Or it is simply because of something wrong in my code. It is a positive part of the contrastive loss. Thanks a lot!<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00c56849115abf95f391b1d4a075331a08d77b46.png" alt="Screenshot 2020-05-09 at 12.54.12" data-base62-sha1="6OWu9EIEROM2l2J4ltFB6e1PaS" width="638" height="102"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87909e81820d18cd7cfac3af9faa08230f072543.png" data-download-href="https://discuss.pytorch.org/uploads/default/87909e81820d18cd7cfac3af9faa08230f072543" title="Screenshot 2020-05-09 at 12.51.51"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87909e81820d18cd7cfac3af9faa08230f072543_2_690x398.png" alt="Screenshot 2020-05-09 at 12.51.51" data-base62-sha1="jlgiPbQxcMZQReKEiUFcJ85Lm7x" width="690" height="398" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/87909e81820d18cd7cfac3af9faa08230f072543_2_690x398.png, https://discuss.pytorch.org/uploads/default/original/3X/8/7/87909e81820d18cd7cfac3af9faa08230f072543.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/7/87909e81820d18cd7cfac3af9faa08230f072543.png 2x" data-dominant-color="F9F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-05-09 at 12.51.51</span><span class="informations">886×512 32.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Torch.cuda.is_available() False even though correct CUDA version and driver are installed

username: oezguensi
cooked: <p>Hi everybody,</p>
<p>I installed Pytorch 1.5.0 using <code>conda install pytorch torchvision cudatoolkit=10.2 -c pytorch</code><br>
nvidia-smi outputs:<br>
<code>NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2</code></p>
<p>Regarding the information on this site: <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver" rel="nofollow noopener">https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver</a> everything should be fine.</p>
<p>However CUDA is still not working, as <code>torch.cuda.is_available()</code> returns False.</p>
<p>When I run:<br>
<code>import torch</code><br>
<code>torch.zeros(1).cuda()</code><br>
I get the error message: <code>cuda runtime error (802) : system not yet initialized at /opt/conda/conda-bld/pytorch_1587428270644/work/aten/src/THC/THCGeneral.cpp:47</code></p>
<p>Does someone know how I can solve this?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Did you install the driver recently and if so, did you reboot your machine?<br>
Is the GPU working besides PyTorch, i.e. are you able to run some CUDA samples?</p>

username: oezguensi
cooked: <p>Thank you for the response. It was indeed a driver issue. Seemed to be corrupted. Downgrading it and rebooting helped.</p>
Torchvision.transform

username: Ishal_Garg
cooked: <p>What does torchvision.transforms take as an input? Is it an image array or the tuple of the image and its label</p>

username: ptrblck
cooked: <p><code>torchvision.transforms</code> usualy take a single input, not a list.<br>
What transformation are you looking for?</p>

username: Ishal_Garg
cooked: <p>I have an image as numpy array of the shape (1024,1248,3). I want to apply transformations of resize and ToTensor.</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Resize"><code>transforms.Reize</code></a> as well as <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ToTensor"><code>ToTensor</code></a> take a <code>PIL.Image</code> as their input, so you would need to transform the numpy array to a <code>PIL.Image</code> first.</p>

username: Ishal_Garg
cooked: <p>Thanks a lot. I will just look to that</p>
Flickr dataset input for Image Captioning

username: manal_elaidouni
cooked: <p>Flickr dataset gives 5 captions for every image.<br>
When creating the dataset <strong>getitem</strong> (). Should I return image X with a list of lists of caption ids Y?</p>

username: ptrblck
cooked: <p>I’m not familiar with the Flickr dataset. Are these captions encoded with integer values or do you want to return the strings in a list?</p>

username: ashutosh_singh
cooked: <p>Sir I was trying to deploy my image captioning project on web . So i have to use model weight. During creating image captioning project I set the epoch to 20! So total I got 20 model weights file.So if i want to use model weights in some other file so should i need to use all the model weights file or the only last model weight file i got during training?<br>
Plss reply sir</p>

username: ptrblck
cooked: <p>Usually you would use the checkpoint with the lowest validation loss, which you should save during training.</p>

username: ashutosh_singh
cooked: <p>how to save it? can you please provide the code? and how to check lowest validation loss?</p>

username: ptrblck
cooked: <p>Have a look at the <a href="https://github.com/pytorch/examples/blob/b9f3b2ebb9464959bdbf0c3ac77124a704954828/imagenet/main.py#L248-L260">ImageNet example</a> to see how to save the checkpoint for the best epoch.</p>
What is wrong with my training procedure

username: arian
cooked: <p>Hi, I’m training to copy a model structure from Keras to Pytorch, but the problem is that there is no progress in learning. It’s like there is no actual training.</p>
<pre><code class="lang-auto">###################
# train the model #
###################
model.train()
steps_train = math.ceil(loaders['train_size']/batch_size)
print(f"****training stpes: {steps_train}*****")
for batch_idx in tqdm(range(steps_train)):
    data, true_map, true_binary = next(loaders['train'])
    # move to GPU
    if use_cuda:
        data, true_binary, true_map = data.cuda(), true_binary.cuda(), true_map.cuda()
    ## find the loss and update the model parameters accordingly
    # clear the gradients of all optimized variables
    optimizer.zero_grad()
    with torch.set_grad_enabled(True):
        # forward pass: compute predicted outputs by passing inputs to the model
        output_map, output_binary = model(data)
        # calculate the batch loss
        loss = criterion(output_binary, output_map, true_binary, true_map)
        # backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # perform a single optimization step (parameter update)
        optimizer.step()
</code></pre>
<p>This is my custom loss function:</p>
<pre><code class="lang-auto">def my_loss(output_binary,output_map, true_binary, true_map):
    loss_binary = torch.mean((output_binary - true_binary)**2)
    loss_map = torch.mean((output_map - true_map)**2)
    return 0.5*loss_binary + 0.5*loss_map

optimizer = optim.Adam(denseNet.parameters())
</code></pre>
<p>Are this part of the code legit? or have I made a rookie mistake here?</p>

username: iamgroot42
cooked: <p>Is <code>denseNet</code> the same variable as <code>model</code> ? Looking at this snippet, it looks live you’re defining the optimizer with the parameters of <code>denseNet</code>, but calculating all losses and metrics on <code>model</code>, which will remain unchanged because of this error.</p>

username: arian
cooked: <p>Thanks for your response, you are right, but the reason for mismatch in names is that train process is in a function, and model is the name of local variable and in fact is same as denseNet.</p>
Understand adapative averge pooling 2d

username: weidezhang
cooked: <p>Hi, I’m trying to understand adapative average pooling 2d does.</p>
<p>For example, for the following code:<br>
input = torch.randn(1,1,4,4)<br>
m = torch.nn.AdaptiveAvgPool2d((5,7))<br>
output = m(input)</p>
<p>it executes fine but what confuses me is why it works even the input HxW is less than kernel size. Does it automatically padding 0 to the size of the kernel ?</p>
<p>Thanks a lot,</p>

username: ptrblck
cooked: <p>I don’t think it’s padding anything, but instead iterates the output pixels (as seen <a href="https://github.com/pytorch/pytorch/blob/877c96cddfebee00385307f9e1b1f3b4ec72bfdc/aten/src/ATen/native/AdaptiveAveragePooling.cpp#L20-L72">here</a>) and calculates the average of the corresponding input positions (as seen <a href="https://github.com/pytorch/pytorch/blob/877c96cddfebee00385307f9e1b1f3b4ec72bfdc/aten/src/ATen/native/AdaptiveAveragePooling.cpp#L12-L18">here</a>).</p>
Why does my output from a pretrained VGG19 model keep changing after model.eval()?

username: sail
cooked: <p>I am playing around with pre-trained models. In this case, VGG19. After training the full model and running model.eval(), I input a single image into the model to see what the output returns. If I run it for a second time, the output has completely changed for the same image.</p>
<p>Now I am wondering, is the dropout layer not freezing? When I run model.classifier[5].training, it returns false. Which tells me it is frozen.</p>
<p>I’ve posted my code <a href="https://github.com/tsail/PreTrainedCNNPractice/blob/master/PreTrainedCNNDemo.ipynb" rel="nofollow noopener">here</a>.</p>

username: ptrblck
cooked: <p>You are using <code>classify_img</code>, which uses <code>TRANSFORM_IMG</code>, which uses random transformations:</p>
<pre><code class="lang-python">TRANSFORM_IMG = transforms.Compose([
                                    transforms.RandomResizedCrop(size=256, scale=(0.8, 1.0)),
                                    transforms.RandomRotation(degrees=15),
                                    transforms.RandomHorizontalFlip(),
                                    transforms.CenterCrop(size=224),
                                    transforms.ToTensor(),
                                    transforms.Normalize([0.485, 0.456, 0.406],
                                                         [0.229, 0.224, 0.225])
                                   ])
</code></pre>
<p>so the difference is expected.<br>
Create a transformation for the validation/test case without the random transformations and it should work. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Confusion Matrix On Tensorboard

username: dachosen1
cooked: <p>Has anyone seen a confusion matrix implemented on Tensorboard?</p>

username: ptrblck
cooked: <p>I think you could create the confusion matrix with <code>sklearn</code> (or calculate it manually), create a plot with <code>matplotlib</code> and send the image to tensorboard.<br>
I’m not sure, if tensorboard can directly display this kind of data.</p>
Back propagation of gradients when using multiple loss functions

username: Sushruth_N
cooked: <p>I have a network as follows.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19474dc527674de440c49a1cfacb42560c2a1f92.png" data-download-href="https://discuss.pytorch.org/uploads/default/19474dc527674de440c49a1cfacb42560c2a1f92" title="IMG_0013"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19474dc527674de440c49a1cfacb42560c2a1f92_2_444x500.png" alt="IMG_0013" data-base62-sha1="3BCHFOoY4fXb05aApOQNLvOlTk6" width="444" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19474dc527674de440c49a1cfacb42560c2a1f92_2_444x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19474dc527674de440c49a1cfacb42560c2a1f92_2_666x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19474dc527674de440c49a1cfacb42560c2a1f92_2_888x1000.png 2x" data-dominant-color="FFFCFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_0013</span><span class="informations">1536×1726 356 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please don’t mind the picture representation of the network.</p>
<p>As in the pic, I have three loss functions. To do gradient descent, can I add all the three losses and form a loss L and do L.backward(). Or should I call backward individually on each loss?</p>

username: russellizadi
cooked: <p>You need to specify parameters of each of loss function (say l1(w1), l2(w2)) in the corresponding optimizer (say optim1(w1), optim2(w2)) and then find the gradients of both loss functions with respect to each parameter (l1.backward(), and l2.backward()) and then update each parameter (optim1.step() and optim2.step()).</p>

username: Sushruth_N
cooked: <p>So adding up the losses and back propagating just once doesn’t work ?</p>

username: russellizadi
cooked: <p>I think I didn’t understand your question at the first place. If you have only one set of parameters and one graph, you should basically add up loss functions (with weights probably based on their importance) and operate back-propagation normally. If you have two different graphs with different loss functions, the previous solution may work. I hope it helps.</p>

username: Sushruth_N
cooked: <p>Sorry if my question wasn’t clear. It is like I have a single graph. But I am calculating different losses at different points in the network as shown above. So I can now add all the losses and back propagate it through one backward() call. This is my understanding.</p>

username: russellizadi
cooked: <p>That is true. Good luck.</p>
How to get the correct shape of the tensor in custom dataset

username: olisp
cooked: <p>Hello,</p>
<p>I am using custom <code>Dataset</code> class, but the problem is that when I get the data from the <code>Dataloader</code> I am left with <code>array</code> that has different tensor shape than I want.</p>
<p>shape that I get:       <code>torch.Size([1, 56, 128, 128])</code><br>
shape that I want:    <code>torch.Size([1, 56, 1, 128, 128])</code></p>
<p>my approach was to:</p>
<ol>
<li>
<p>to apply <code>numpy.expand_dims</code> on the <code>array</code> and get   <code>torch.Size([1, 1, 56, 128, 128])</code></p>
</li>
<li>
<p>then to <code>np.transpose</code> on the <code>array</code> to get the shape I want <code>torch.Size([1, 56, 1, 128, 128])</code></p>
</li>
</ol>
<p>after first step I am getting the error:</p>
<pre><code class="lang-auto">raise ValueError('pic should be 2/3 dimensional. Got {} dimensions.'.format(pic.ndim))
ValueError: pic should be 2/3 dimensional. Got 4 dimensions.
</code></pre>
<p>if I do the transpose first, none of the combination of <code>np.transpose(array, axes=(1,2,0))</code> yields the shape <code>torch.Size([56, 1, 128, 128])</code></p>
<p>if I convert <code>array</code> to <code>Tensor</code> first and then do <code>torch.unsqueeze</code> I get the error:</p>
<pre><code class="lang-auto">
raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got &lt;class 'torch.Tensor'&gt;
</code></pre>
<p>here is my code:</p>
<pre><code class="lang-auto">class patientdataset(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
        # np_load_old = np.load
        # np.load = lambda *a, **k: np_load_old(*a, allow_pickle=True, **k)

        image= np.asarray(np.load(img_path))


        image= np.transpose(image, axes=(1,2,0))
        image = torch.Tensor (image)

        image = torch.unsqueeze(image, dim=1)



        y_label = torch.tensor(np.asarray(self.annotations.iloc[index,1]))

        if self.transform:
            image = self.transform(image)

        return (image, y_label)
</code></pre>

username: klory
cooked: <p>your code should be fine, can you provide the call stack for the error?</p>
How to remove the layer of features of pretrained MobilenetV2?

username: Wei_Wong
cooked: <p>I know how to remove the last layer of mobinet_v2, but how to remove certain layer of features?<br>
Mobilenet has the following structure,<br>
(features):<br>
layer 1<br>
layer 2<br>
layer 3<br>
…<br>
…<br>
(classifier):<br>
output layer</p>

username: emcp
cooked: <p>Are you attempting to do fine-tuning or something else like transfer learning ?</p>
<p>Meanwhile I saw this tutorial but the source code link gives me a 404</p>
<p><a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a></p>
<p>Here is how I freeze a pre-trained model to do what is now called transfer learning… and I erase the last fc layer</p>
<pre><code class="lang-auto">if model_conf["hyperParameters"]["freeze_pretrained_gradients"]:
            print("Using backbone as fixed feature extractor")
            modules = list(backbone_nn.children())[:-1]  # delete the last fc layer.
            backbone_nn = nn.Sequential(*modules)

            # FasterRCNN needs to know the number of
            # output channels in a backbone. For resnet101, it's 2048
            for param in backbone_nn.parameters():
                param.requires_grad = False
            backbone_nn.out_channels = 2048
</code></pre>
<p>the key is that .children() call</p>
Tensorboard on GCP

username: dachosen1
cooked: <p>I’m trying to run Tensorflow on a  GCP jupyter notebook. The program is running and is generating log output. I’ve tried two approaches.</p>
<p>Approach 1:</p>
<pre><code class="lang-auto">LOG_DIR = '/tmp/log'
get_ipython().system_raw(
    'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &amp;'
    .format(LOG_DIR)
)

! wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip
! unzip ngrok-stable-linux-amd64.zip

get_ipython().system_raw('./ngrok http 6006 &amp;')

! curl -s http://localhost:4040/api/tunnels | python3 -c \
    "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"
</code></pre>
<p>The approach generated a url, and i got an error that says No dashboards are active for the current data set.  I’m not sure why the directory didn’t update to the LOG_DIR path.</p>
<p>it still says: <em>Data location: ./runs</em></p>
<p>Approach two:</p>
<p>run the below on the terminal</p>
<pre><code class="lang-auto">jupyter@pytorch-20200422-192725:~$ tensorboard --logdir runs  --bind_all
2020-04-23 08:04:18.694994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-04-23 08:04:18.696468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
TensorBoard 2.1.1 at http://pytorch-20200422-192725:6007/ (Press CTRL+C to quit)
</code></pre>
<p>I tried <a href="http://pytorch-20200422-192725:6007/" rel="nofollow noopener">http://pytorch-20200422-192725:6007/</a> , http://[GCP internal IP ]:6007/  and http://[GCP external IP ]:6007/</p>
<p>all three gave me the same error: Hmm. We’re having trouble finding that site.</p>
<p>I have allow HTTP requests enabled behind the firewall.</p>
<p>how do I see the ltensorboard log outputs from a virtual machine?</p>

username: dachosen1
cooked: <p>I have allow HTTP requests enabled behind the firewall.<br>
Answer: Specify the http port</p>
Seemigly right dimensions for convolution but still getting "expected stride to be a single integer value or a list of 3 values to match the convolution dimensions, but got stride=[1, 1]"

username: csailnadi
cooked: <p>I am getting<br>
<code>RuntimeError: expected stride to be a single integer value or a list of 3 values to match the convolution dimensions, but got stride=[1, 1]</code>. Other posts say that this error occurs when there is batch dimension is missing, but it seems I get proper dimensions, the output of print is <code>x.shape in  torch.Size([5, 512, 14, 14])</code>. So what might be a problem here?</p>
<pre><code class="lang-auto">class SimpleConvo(nn.Module):
    def __init__(self, channels, kernel_size, dim=2):
        super(SimpleConvo, self).__init__()
        kernel = torch.ones(size=(channels, kernel_size, kernel_size))
        kernel = kernel.view(1, 1, *kernel.size())
        kernel = kernel.repeat(channels, *[1] * (kernel.dim() - 1))
        self.register_buffer('weight', kernel)
        self.groups = channels
        self.conv = F.conv2d if dim == 2 else F.conv3d

    def forward(self, x):
        print("x.shape in ", x.shape)
        x = self.conv(x, weight=self.weight.cuda(), groups=self.groups)
        return x
</code></pre>
<p>EDIT:<br>
Usage:</p>
<pre><code class="lang-auto">convos = SimpleConvo(channels=512, kernel_size=3)
mat = convos(mat)
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>You get this in the dim=3 case? Or dim=2 ?<br>
Would you have a sample input that shows the error please?</p>

username: csailnadi
cooked: <p>I have updated my post. it is just default dim=2<br>
EDIT: actually i should not define for kernel batch dimension. It should be just<br>
kernel = torch.ones(size=(kernel_size, kernel_size))</p>
<p>problem solved</p>
Precision problem while converting to FloatTensor

username: Shanto_Islam
cooked: <p>I am facing precision problem while converting from list to FloatTensor. Looked through the <a href="https://pytorch.org/docs/stable/tensors.html" rel="nofollow noopener">documentation</a> but unable to find any parameter or variable that lets me set/fix the precision.</p>
<p>Example given below:</p>
<pre><code class="lang-auto">print(weight)
&gt; [ 0.02551583  0.01355686 -0.04717987 ...  0.01422889 -0.01558862
   0.01234896]

print(torch.FloatTensor(weight))
&gt; tensor([ 0.0255,  0.0136, -0.0472,  ...,  0.0142, -0.0156,  0.0123])
</code></pre>
<p>Hoping for a quick solve.</p>

username: russellizadi
cooked: <p>I don’t think you can have float tensor to capture all of the precision points. You can use the <code>set_printoptions</code> to see why the <code>FloatTensor</code> is not accurate after some points. If you want more precision, use <code>DoubleTensor</code>:</p>
<pre><code class="lang-auto">import torch
weight = [1.0000600001, 4.0000900001]
torch.set_printoptions(precision=10)

print(torch.FloatTensor(weight))
print(torch.DoubleTensor(weight))
</code></pre>
<pre><code class="lang-auto">tensor([1.0000599623, 4.0000901222])
tensor([1.0000600001, 4.0000900001], dtype=torch.float64)
</code></pre>

username: Shanto_Islam
cooked: <p>DoubleTensor promoted CudNN error. However, FloatTensor with <code>torch.set_printoptions(precision=10)</code> worked. But the question is, will it be true for all up and coming tensors? Then a lot of precision will be lost if I want all other tensors to be default and have exactly those precision which it obtains.</p>

username: russellizadi
cooked: <p>The <code>set_printoptions</code> only changes the printing interface and it has nothing to do with the precision. If you want to use float data type, you will have up to 7 precision accuracy. As far as I know, when it comes to parameter estimation, the float data type is more than enough. That’s probably why Geforce GPU is preferred over other types like Quadro with support more floating points.</p>
Tensor index question

username: KevinFan
cooked: <p>Hello!<br>
I have a tensor of size ([128,3,64,64])<br>
lets say this tensor is called testBatch</p>
<p>testBatch[127:].size()<br>
–&gt; torch.Size([1, 3, 64, 64])</p>
<p>testBatch[:1].size()<br>
–&gt; torch.Size([1, 3, 64, 64])</p>
<p>Could someone explain the differences of these two index methods</p>

username: ptrblck
cooked: <p>Indexing uses the pattern <code>[start:stop:step]</code>.<br>
In the first approach you will get all samples from the start index 127 to the end (so just the last sample), while the second approach will give you all samples from the beginning to index 1, excluding index 1 (so just the first sample):</p>
<pre><code class="lang-python">x = torch.randn(128, 3, 64, 64)

a = x[127:]
b = x[:1]

print((a == x[-1]).all())
&gt; tensor(True)
print((b == x[0]).all())
&gt; tensor(True)
</code></pre>

username: KevinFan
cooked: <p>Very clear and concise explanation. Thanks so much for taking the time to answer my question!</p>
Multiprocessing / DDP - Barrier Blocks loss.backward()

username: soulslicer
cooked: <p>I have a toy example for DDP. I have two processes, and one process has more “iterations” than the other</p>
<pre><code>    if self.id == 0:
        ep = 10
        sl = 0.1
    elif self.id == 1:
        ep = 15
        sl = 0.2

    for i in range(0, ep):

        # if i == 10:
        #     break

        self._log.info(self.id, "image.. " + str(i))
        image = torch.zeros((1,2,3,256,384))
        input = {"rgb": image}
        self._log.info(self.id, "model.. " + str(i))
        output = self.model(input)
        loss = torch.sum(output["output"][-1])
        

        self._log.info(self.id, "zerograd.. " + str(i))
        self.optimizer.zero_grad()
        self._log.info(self.id, "backward.. " + str(i))
        loss.backward()
        self._log.info(self.id, "step.. " + str(i))
        self.optimizer.step()
        print("step: " + str(self.id) + " " + str(i))
        self._log.info(self.id, "done " + str(i))

        time.sleep(sl)

    self._log.info(self.id, "barrier.. ")
    dist.barrier()
</code></pre>
<p>But as soon as the first process (with lesser iterations) hits dist.barrier(), loss.backward() is blocked in the 2nd process! How do I get around this? I could force all processes to have the same number of iterations…but thats not something i want.</p>

username: mrshenli
cooked: <p>Is this the same as <a href="https://discuss.pytorch.org/t/multiprocessing-barrier-blocks-all-processes/80345" class="inline-onebox">Multiprocessing - Barrier Blocks all Processes?</a></p>
<p>BTW, for future questions on <code>torch.distributed</code>, could you please add a “distributed” tag. People working on the distributed package are monitoring that channel.</p>
Construct quantized Tensor from int_repr()

username: v0lta
cooked: <p>Dear pytorch community,<br>
Given an integer representation and the affine transform, how do I create a corresponding torch.tensor object? The python tensor constructor seems to assume no quantization.<br>
Thanks a lot for your help,<br>
moritz</p>

username: v0lta
cooked: <p>For posterity since I was looking here for the solution, if your main interest is in the original weight approximation one solution is to approximate the original weights manually:<br>
restweights = (quant_ints - q_zero_point)*q_scale<br>
This ‘hacky’ approach works for me since I was going to call dequantize anyway.</p>

username: jerryzh168
cooked: <p>we do have some non-public API to do this: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L3862" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L3862</a> and <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L3868" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml#L3868</a></p>
<p>but they we might change the API when we officially release quantization as a stable feature.</p>
Sub-8 bit quantization

username: zlou
cooked: <p>I am trying to simulate sub-8 bit quantization. Currently, I am only doing post-training quantization, and I am doing so by creating a custom observer which is identical to the existing <code>HistogramObserver</code> except that the <code>qmin</code> and <code>qmax</code> values are changed to match the new bit width.<br>
Running this with ResNet-50, with 4 bit activations and 8 bit weights, the top 1 accuracy is around 10%, which is significantly lower than the results reported in the whitepaper by Krishnamoorthi (36%). Are there other changes which need to be made to simulate this quantization, or does this come down to some kind of TF vs. Pytorch difference?</p>

username: jerryzh168
cooked: <p>cc <a class="mention" href="/u/raghuramank100">@raghuramank100</a> <a class="mention" href="/u/hx89">@hx89</a></p>
During QAT, how to save the float32 model without fuse module?

username: onesnow123q
cooked: <p>Hi, I tried QAT with this example <a href="https://github.com/pytorch/vision/blob/master/references/classification/train_quantization.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/references/classification/train_quantization.py</a>.<br>
I would like to know whether the fp32 model can be saved during the quatization, which has no extra parameters learned by QAT?<br>
This FP32 model without fusion, just like the normal pretrained float32 model.</p>

username: jerryzh168
cooked: <p>are you asking if you can save a normal fp32 model? of course, you can find the doc for serialization here: <a href="https://pytorch.org/docs/stable/notes/serialization.html?highlight=save" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/serialization.html?highlight=save</a></p>

username: onesnow123q
cooked: <p>Thanks!<br>
I know how to save a normal fp32 model, but I don’t know how to save it during quantization ware training.Because the model which saved during QAT has some params such as scale , zero_points and so on.I want these params disappear in the model.But I don’t know how to change code to achieve it.</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="onesnow123q" data-post="3" data-topic="78505">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/ebca7d/48.png" class="avatar"> onesnow123q:</div>
<blockquote>
<p>I know how to save a normal fp32 model, but I don’t know how to save it during quantization ware training.Because the model which saved during QAT has some params such as scale , zero_points and so on.I want these params disappear in the model.But I don’t know how to change code to achieve it.</p>
</blockquote>
</aside>
<p>why do you want these params to disappear? we need to save scale/zero_point of qat model since it’s part of the state of the model</p>
Precompiled binaries if compilation is not feasible

username: Scipio
cooked: <p>Hello everyone,</p>
<p>I’m trying to run some pytorch code on a universities cluster. The GPU in question is a Tesla K20m, so according to wikipedia it supports CUDA compute capability 3.5. As I understood from other threads the corresponding binary isn’t shipped with pytorch anymore. I tried building pytorch from source but the system itself is rather ancient(Ubuntu 14.04, gcc 4.8.4) and I get</p>
<pre><code class="lang-auto">The C++ compiler does not support required functions.  This is very likely
  due to a known bug in GCC 5 (and maybe other versions) on Ubuntu 17.10 and
  newer.  For more information, see:
  https://github.com/pytorch/pytorch/issues/5229
</code></pre>
<p>As I have limited possibilities to change these circumstances I was wondering if I could precompile the binaries on another machine and move them to the cluster. Or if maybe those were available somewhere. And what files I was even looking for in my environment folder. I am new to Pytorch, CUDA and using remote systems, please let me now which additional information might be required.</p>
<p>Best regards</p>

username: albanD
cooked: <p>Hi,</p>
<p>Doing cross compilation is know to be tricky. I wouldn’t recommend it if you’re not already familiar with build systems.</p>
<p>I think your best change is to get a newer version of gcc on the cluster.<br>
You can either use conda and install a gcc there that will be recent enough that you can compile pytorch with it.<br>
Or ask the people that maintain the cluster how you can get a newer version of gcc locally. You usually can get that for your user only.</p>

username: Scipio
cooked: <p>Alright, I’ll keep my hands of cross compilation then, thanks for the warning. To get gcc inside my conda environment I would try and install it from the conda-forge channel or somewhere else?</p>

username: albanD
cooked: <p>Yes conda-forge has one, and compatible cuda and mkl maybe?<br>
You can find the full instructions for the install from source in conda in the doc: <a href="https://github.com/pytorch/pytorch#from-source">https://github.com/pytorch/pytorch#from-source</a></p>

username: Scipio
cooked: <p>I had followed the instructions mentioned by <a class="mention" href="/u/alband">@albanD</a> (<a href="https://github.com/pytorch/pytorch#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a>) before but assumed the system would have an at least somewhat recent set of compilers available. Turns out it didn’t and not only gcc but a proper c++ compiler was missing too, but</p>
<blockquote>
<p>conda install -c conda-forge gcc cxx-compiler</p>
</blockquote>
<p>Appears to have done the trick. I’m now waiting for the compilation to finish and hope my original problem which caused me to consider building pytorch from source will be solved to. Thank you for your help!</p>
Multiprocessing - One Process Broadcasting to other processes?

username: soulslicer
cooked: <p>I need to broadcast a string or some variable from one process to another. What is the correct way to do this?</p>
<p>I could use a shared memory tensor and write to it and then read it. Is there a better way?</p>

username: mrshenli
cooked: <p>An alternative is to use the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.broadcast" rel="nofollow noopener"><code>torch.distributed.broadcast</code></a> API.</p>
Custom loss not decreasing after reducing the dataset

username: maroov
cooked: <p>So, I have been working with an unsupervised model that learned from the NORB dataset (images of toys) and I wanted to create a version that learns from samples of variations of <strong>just one toy</strong> in order to figure out the angle variations, instead of using all toys for all classes as I was doing before for traditional classification.</p>
<p>The thing is, using exactly the same code but using only one toy leads me to a situation where the loss is pretty much constant up to several decimal places for <strong>any</strong> learning rate lower than approximately 1e-2, and above that it’s just crazy high. It was working just fine until I reduced the dataset to a single toy.</p>
<p>I have tried many learning rates, adding and removing layers, using a different toy…</p>
<p>My loss looks like this:</p>
<pre><code class="lang-auto">def infoNCE(zt, ztk):
  ind = np.diag_indices(b_size)
  mul = ztk @ torch.t(zt)
  num = mul[ind[0], ind[1]]
  m, _ = torch.max(mul, axis=0)
  den = torch.log(torch.sum(torch.exp(mul - m), axis=0)) + m
  val = torch.mean(-num + den)
  return val
</code></pre>
<p>My model:</p>
<pre><code class="lang-auto">class Module1(nn.Module):

  def __init__(self):
    super(Module1, self).__init__()
    self.conv1 = nn.Conv2d(1, 32, 5, 1, 2)
    self.conv2 = nn.Conv2d(32, 32, 5, 1, 2)
    self.conv3 = nn.Conv2d(32, 64, 5, 1, 2)
    self.conv4 = nn.Conv2d(64, 64, 5, 1, 2)
    self.conv5 = nn.Conv2d(64, 128, 5, 1, 2)
    self.conv6 = nn.Conv2d(128, 128, 5, 1, 2)
    
    self.pool = nn.MaxPool2d(2, 2)
    self.fc = nn.Linear(128*8*8, 3)

  def forward(self, x):
    x = self.pool(F.relu(self.conv2(F.relu(self.conv1(x)))))
    x = self.pool(F.relu(self.conv4(F.relu(self.conv3(x)))))
    x = self.pool(F.relu(self.conv6(F.relu(self.conv5(x)))))
    x = x.view(x.size(0), -1)
    x = self.fc(x)
    return x
</code></pre>
<p>My training routine:</p>
<pre><code class="lang-auto">mod1 = Module1().to(device)
optimizer = optim.Adam(mod1.parameters(), lr=lr)

for epoch in trange(epochs):
  for i, data in enumerate(trainloader):
    xt = data[0][0].to(device)
    xtk = data[0][1].to(device)
    optimizer.zero_grad()
    zt = mod1(xt)
    ztk = mod1(xtk)
    loss = infoNCE(zt, ztk)
    loss.backward()
    optimizer.step()
</code></pre>
<p>The loss is always around 2.77, because that is usually the value of the variable <code>den</code>, while <code>num</code> is kinda small (this didn’t happen when I used all of the samples).</p>
<p>EDIT: Yes, this number is suspiciously close to <code>e</code> and I’m using logarithms.</p>