akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Should compute mean and std for new dataset?

EdisonKing: <p>I am working on a cv model. I know that I should normalize the dataset by its mean and std.</p>
<p>The backbone is a pre-trained model (resnet-50), and I saw some code just use std and mean computed from Imagenet, which was used to train the backbone.</p>
<p>But now, I am working on another dataset A, other than Imagenet. Should I change std and mean from Imagenet to A?</p>
<p>sincerely thanks.</p>

ptrblck: <p>It depends a bit on your use case.<br>
E.g. if your new dataset is quite small, but originates from the same domain (“natural” images like in ImageNet), you could still use the ImageNet stats.<br>
However, if your new dataset contains images from another domain, e.g. medical CT scans, I would rather calculate the new statistics, as they might differ a lot.</p>

EdisonKing: <p>Thanks for your reply, this helps me a lot.!</p>
Implementation of Jordan RNN

abn: <p>Hi PyTorch Community!</p>
<p>I was studying RNN and curious to check the difference in performance between Elman and Jordan RNN. It appears the <a href="https://pytorch.org/docs/stable/nn.html#rnn" rel="nofollow noopener">RNN in PyTorch</a> uses Elman RNN. But, I was not able to find an implementation of Jordan RNN in PyTorch. Is there any open source implementation of Jordan RNN in PyTorch?</p>
<p>Thanks for any advice in advance!</p>
Is there SVHN format 1 dataset in torchvision?

zxgx: <p>I want to implement a digits detection system by yolo v3, and I want to train it on <a href="http://ufldl.stanford.edu/housenumbers/" rel="nofollow noopener">SVHN</a>. So, I import <code>torchvision.datasets.SVHN</code>.<br>
However, When I display an image for verification, I have found that it only contains the format 2 dataset, which only contains cropped digit for classification.<br>
So, is there format 1 svhn in torchvision?</p>

sbl1996: <p>There is no SVHN detection dataset in torchvision. However, I transformed the detections annotations in MATLAB format to coco style and packed the images and annotations to a tar file. You can check the data and scripts <a href="https://github.com/sbl1996/pytorch-hrvvi-ext/blob/master/horch/datasets/svhn.py" rel="nofollow noopener">here</a>. If you use Windows, you may need to change the suffix from ‘tar’ to ‘tar.gz’ to uncompress correctly.</p>

zxgx: <p>thank you for your nice work!<br>
But I’m trying to implement my svhn detection dataset from the source.</p>
How to crop the extra row or column of tensor?

MingHongL: <p>Hi！<br>
I want to know how to crop the tensor in Pytorch. There is a tensor with size of (16, 3,46, 46), what I want to do is to crop the tensor’s size from (16, 3, 46, 46) to (16, 3, 45, 45).<br>
The code is following:</p>
<pre><code class="lang-auto">image = torch.randn(16, 3, 45, 45)
b, c, h, w = image.size()
image_pad = torch.nn.functional.pad(image, [0,  w%2, 0,  h%2])  
</code></pre>
<p>How can I crop the image_pad (16, 3, 46, 46) to recover the original image (16, 3, 45, 45)?<br>
I used the function tensor.resize_() to do it, it did recover the size of the original image, but I found that it didn’t recover the value of the original image.<br>
I just want to crop the extra row or column of image_pad to recover the original image.<br>
Thanks!</p>

ddeeppnneett: <p>image_pad[:,:,:h,:w] ???</p>

MingHongL: <p>Thanks for your answer! It is really useful.</p>
Bigger tensor results in RunImpl error C++

Aeroxander: <p>So I’m passing an image tensor through traced C++ model, I just used a 1080 tensor for the tracing (if this has anything to do with it)+, but when I run in in C++ it gives an error if I make it bigger than ~280.</p>
<p>The error is</p>
<blockquote>
<p>Microsoft C++ exception: std::runtime_error at memory location 0x000000395E2F6710.</p>
</blockquote>
<p>at</p>
<pre><code class="lang-auto">
try {
        loadTensorsFromRegisters(inst.inputs, stack);
        size_t new_pc = pc + 1 + inst.callback(stack);
        for (int i = inst.outputs.size - 1; i &gt;= 0; --i) {
          int reg = get(inst.outputs, i);
          registers[reg] = pop(stack);
          // std::cout &lt;&lt; "pop reg[" &lt;&lt; reg &lt;&lt; "];\n" &lt;&lt; registers[reg] &lt;&lt; "\n";
        }
        pc = new_pc;
      } catch (Suspend&amp; e) {
        // wait() expects a single input
        AT_ASSERT(inst.inputs.values.size == 1);

        getOrCreateFuture();

        if (get(inst.inputs.free_flags, 0)) {
          // make sure the register is not freed once we are waked up
          registers[get(inst.inputs.values, 0)] = e.future;
        }

        // Make sure adding callback is the last step.
        // Otherwise if e.future has completed,
        // the current thread will continue running before it suspends.
        InterpreterState state(intrusive_from_this());
        e.future-&gt;addCallback([state]() {
          c10::global_work_queue().run(InterpreterContinuation(state, Stack(),
              autograd::GradMode::is_enabled()));
        });

        return true;
      }
</code></pre>
<p>So<br>
<code>at::Tensor tensor_contentimage = torch::rand({ 1, 3, 200, 200}).to(device_type).toType(at::kFloat);</code> works fine, but<br>
<code>torch::rand({ 1, 3, 720, 720}).to(device_type).toType(at::kFloat);</code><br>
fails, while the same size tensor worked fine in Python!<br>
It’s something with the .ToTensor and intrusive pointer</p>
<p>Just any tips or advice on solving the problem will be greatly appreciated, thanks!</p>

Aeroxander: <p>Suppose I need to use CUDA during the tracing…</p>
Call nn modules in Dataloader

Edgar_Medina: <p>Hello guys,</p>
<p>I came here with a particular question about Dataloader, Could I introduce cuda nn modules in <strong>call</strong>?<br>
I am having problems with CUDA initialization but I didn’t find the answer. Thank you so much in advanced. The error I am having happens when I use <code>.cuda()</code> or <code>torch.cuda.FloatTensor()</code>, I describe the problem below:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: initialization error
</code></pre>
<p>Dataloader and reading data</p>
<pre><code class="lang-auto">class SomeNetwork(object):
    def __init__(self,parama=1,paramb=2,paramc=3):
        self.net = nn.net(self,parama=1,paramb=2,paramc=3)).cuda() # CUDA
    def __call__(self, x):
       self.net(x[None,:].cuda())
        return x[0]
</code></pre>
<pre><code class="lang-auto">transformations.extend([ transforms.ToTensor(),
                         SomeNetwork(),
                         transforms.ToPILImage(),

trainset = ImageNet(root='./Database/', 
                    split='train',  
                    transform=transforms.Compose(transformations), 
                    target_transform=None,)
</code></pre>

ptrblck: <p>You’ll most likely encounter this issue if you are using multiple workers in your <code>DataLoader</code>, as each will try to initialize CUDA.<br>
Could you try to set <code>num_workers=0</code> and run your code again?</p>

Edgar_Medina: <p>Thank you so much, It ran now.</p>
<p>Unfortunately, the other traditional augmentation are not working with 16 workers anymore. I hope this can be improved in future release.</p>

ptrblck: <p>You could try to add <a href="https://github.com/pytorch/pytorch/issues/1494#issuecomment-305993854" rel="nofollow noopener">this code snippet</a> to your code.<br>
However, usually the data processing it performed on the CPU, while the data is pushed onto the device in the training loop.</p>

Edgar_Medina: <p>Thank you again <a class="mention" href="/u/ptrblck">@ptrblck</a> , I checked the link (also the function documentation) but I did not understand how it works, could you share me a link of why it should work?</p>
<p>I added it at the beginning of the main code in python3, but I did not see an speed improvement.</p>
I'm trying to work with pandas data set

mohamed_nabil: <p>iam trying to build a neural network to classify survival in titanic dataset<br>
but i keep getting this errors when i run the neural network</p>
<blockquote>
<p>Expected object of scalar type Double but got scalar type Float for argument</p>
</blockquote>
<pre><code class="lang-auto">optimizer = optim.SGD(model.parameters(), lr=0.003)
epochs = 3
print_every = 40
steps = 0
for e in range(epochs):
    running_loss = 0
    for data, labels in zip(X_train,y_train):
        steps += 1
        
        
        
        
        optimizer.zero_grad()
        
        # Forward and backward passes
        output = model.forward(data)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()
        
        running_loss += loss.item()
        
        if steps % print_every == 0:
            print("Epoch: {}/{}... ".format(e+1, epochs),
                  "Loss: {:.4f}".format(running_loss/print_every))
</code></pre>
<p>that’s the code of training<br>
and that’s my network</p>
<pre><code class="lang-auto">class Network (nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(7,5)
        self.fc2 = nn.Linear(4,3)
        self.fc3=nn.Linear(3,2)
    def forward(self,x):
        x=self.fc1(x)
        x=F.relu(x)
        x=self.fc2(x)
        x=F.relu(x)
        x=self.fc3(x)
        x=F.sigmoid(x,dim=1)
        return x
model= Network()
</code></pre>
<p>i’m playing around with the pytorch api to learn so i’m still a begginer</p>

ptrblck: <p>Most likely <code>data</code> is a <code>DoubleTensor</code> as this is the default type in numpy.<br>
Try to call <code>data = data.float()</code> before passing it to the model.</p>
How to get train/eval flg in side the model?

ddeeppnneett: <p>Hi,</p>
<p>During the training, we use</p>
<pre><code class="lang-auto">model.train() 
for i, ( imgs, labels) in enumerate(train_loader):
    prd = model(imgs)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">model.eval() 
for i, ( imgs, labels) in enumerate(val_loader):
    prd = model(imgs)
</code></pre>
<p>to swtich on/off some functions.</p>
<p>How could I get the <strong>train/eval</strong> flag in side the model? Is there a build-in parameters that I could use? Or I have to pass the flg like</p>
<pre><code class="lang-auto">prd = model(imgs, flg_trn=True) 
</code></pre>

ptrblck: <p>Inside of your model you can use the <code>self.training</code> attribute.</p>

ddeeppnneett: <p>I copy the source code of resnet.py to the folder. And add different codes in forward(self, x) for the training case and eval case.</p>
<pre><code class="lang-auto">def forward(self, x)
      x = ......
      if self.training :
            print("...........training.......")
</code></pre>
<p>This print doesnt work.</p>
<pre><code class="lang-auto">from resent import resnet50
model = resnet50()
model.train() 
for i, ( imgs, labels) in enumerate(train_loader):
    prd = model(imgs)
</code></pre>
<p>Am I missing something?</p>

ptrblck: <p>It should work. Could you add a print statement before the if condition to check, if your custom code is really used instead of the original <code>torchvision</code> implementation?</p>

ddeeppnneett: <p>You’re right.</p>
<p>I changed model.train() to model.eval() by mistake.</p>
<p>Thanks for your help.</p>
Get stuck in clip_grad and optimizer.step using multi-GPU

46a9705b0a71168fc058: <p>Hi, I meet a problem when train my model using multi-gpu on one node with nn.parallel.DistributedDataParallel. And the version of PyTorch is 0.4.1.<br>
I use the following comment to run the program</p>
<p>export NGPU=3;<br>
python -m torch.distributed.launch --nproc_per_node=$NGPU train.py</p>
<p>and it get stuck in clip_grad_norm_() and optimizer.step(). When I use nvidia-smi to check the utilization rate of the GPUs, all three of them are 100%.<br>
However, when I set NGPU=1, the program can be executed correctly.<br>
What’s the reason? How can I fix it? Thanks!</p>
Propagate gradients through (external) PCA

cmplx96: <p>Hi all,</p>
<p>I have an architecture where I put the outputs of a neural network through a PCA (Incremental PCA from scikit-learn) and then compute the loss function:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6241dd891379af79e88d87b428a7e6d737439459.png" data-download-href="https://discuss.pytorch.org/uploads/default/6241dd891379af79e88d87b428a7e6d737439459" title="Untitled%20Diagram"><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6241dd891379af79e88d87b428a7e6d737439459.png" alt="Untitled%20Diagram" data-base62-sha1="e1dUipi5OlTnN7cv6nNvdnWrLZL" width="690" height="101" data-dominant-color="EFF0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled%20Diagram</span><span class="informations">721×106 6.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Of course when I compute the PCA, the PyTorch tensors are converted to NumPy arrays and lose the gradient context information.</p>
<p>My goal is to update the weights of the neural network with respect to the loss function. Is there a straightforward way of achieving this?</p>
<p>Thanks!</p>
Could I change the weight and bias in nn.torch.Sigmoid()?

GoldD: <p>Hello.</p>
<p>I have question.</p>
<p>Could I change the weight and bias in nn.torch.Sigmoid()?</p>

ptrblck: <p>The sigmoid function does not hold any weight or bias parameters. It just applies <code>1 / (1 + e**(-x))</code> on its input <code>x</code>.</p>

GoldD: <p>Thanks for your reply.<br>
Then, If I use Sigmoid function, Couldn’t revise bias and weight??</p>

ptrblck: <p>I’m not sure I understand your use case clearly.<br>
If you would like to take a look at some parameters, you can call them directly in the corresponding module:</p>
<pre><code class="lang-python">weight = model.fc.weight
bias = model.fc.bias
</code></pre>
<p>where your model contains e.g. a linear layer assigned to <code>self.fc</code>.</p>
ImportError: No module named 'cv2'

KanZa: <p>Hi,</p>
<p>I have 2 versions of Pytorch 0.3.0 the other one is installed virtually 0.4.0. the project. I am trying to implement some code in Pytorch 0.4. how should I install OpenCV in a virtual environment?</p>

iArunava: <p>What’s wrong with<br>
<code>pip3 install opencv-python3</code><br>
?</p>
F1score for nuclei segmentation

sank: <p>I have a problem while implementing f1score at object level. I have a mask and prediction of a nucleI, each mask has 500-1000 nucleui and i need to find out the F1score. The F1score will be calculated if a nuclei in mask and prediction have IOU&gt;=.5 then true positive =1. So how can i compare all the nuclei in a mask to the prediction?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9c132f409c7687b10153e9e3ced4ea0f1dac4f95.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c132f409c7687b10153e9e3ced4ea0f1dac4f95" title="0"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c132f409c7687b10153e9e3ced4ea0f1dac4f95_2_128x128.png" alt="0" data-base62-sha1="mgHAsTyHQU8yP6w9RV4kTti5ZDn" width="128" height="128" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c132f409c7687b10153e9e3ced4ea0f1dac4f95_2_128x128.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c132f409c7687b10153e9e3ced4ea0f1dac4f95_2_192x192.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c132f409c7687b10153e9e3ced4ea0f1dac4f95_2_256x256.png 2x" data-dominant-color="4A4A4A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">0</span><span class="informations">1000×1000 34.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Oli: <p>Hello there!</p>
<p>I guess you need to loop over your ground truths/nuclei, comparing if its IOU &gt;.5 for each prediction and if one is, you mark the nuclei as found and remove both the nuclei and prediction.</p>
<p>There is a <a href="https://github.com/rafaelpadilla/Object-Detection-Metrics" rel="nofollow noopener">library</a> that does what you are looking for but this one only works for bounding boxes. Perhaps you could fork this repo? Good luck!</p>
Inconsistent operation of F.conv2d()

srikanthram: <p>I am getting inconsistent results when I tried doing the convolution operation using F.conv2d()</p>
<p>Lets say <code>h**2</code> is the input and w&gt;=0 is the filter kernel. Now, <code>F.conv2d(h**2,w,padding=1)</code> and <code>F.conv2d(h**2,w)</code> should give only non-negative results. However, to my surprise, the former does give negative numbers in the output for some w&gt;=0. Mathematically padding should not affect the sign change</p>

gslaller: <p>Seems quite impossible, since h**2 is guaranteed to be [0,inf]^Rn and w&gt;=0. Could you provide some accompanying code?</p>
Nvidia driver doubt

imtherealnaska: <p>Hey guys , I just want to update my Nvidia driver ,to install PyTorch with cuda 9.  I use Ubuntu 16.04 . Can you please direct me to some working-without-crashing steps.(I have crashed it a lot of times)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/596bf46b2e45e51a3296867390c9d2ec8a91f9da.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/596bf46b2e45e51a3296867390c9d2ec8a91f9da" title="1558153495045-1388244327"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/596bf46b2e45e51a3296867390c9d2ec8a91f9da_2_690x388.jpeg" alt="1558153495045-1388244327" data-base62-sha1="cL3MMKWCfsMSSry8oGyBysWvF8u" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/596bf46b2e45e51a3296867390c9d2ec8a91f9da_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/5/596bf46b2e45e51a3296867390c9d2ec8a91f9da_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/596bf46b2e45e51a3296867390c9d2ec8a91f9da_2_1380x776.jpeg 2x" data-dominant-color="3F414A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1558153495045-1388244327</span><span class="informations">1920×1080 272 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have cuda 9 installed<br>
My driver isn’t good enough to use it , yet .<br>
So any suggestions would be appreciated.</p>
<p>Thank you .!</p>
Reusing the embedding matrix as a sub tensor in a larger one

ac1d: <p>I am working on Language Modelling and assuming you are familiar with the topic, you already know that the first part of a Language Model is an <strong>encoder</strong> containing the <strong>Embedding Matrix</strong>. Assuming, I have an encoder:</p>
<p><code>encoder = nn.Embedding(vocab_size, emb_size)</code></p>
<p>that contains an embedding matrix:</p>
<p><code>EmbMatrix.shape = [vocab_size, emb_size]</code></p>
<p>I would like to create a tensor with dimensions:</p>
<p><code>t = [seq_length, batch_size, vocab_size, emb_size]</code></p>
<p>where for each i,j:</p>
<p><code>t[i, j] = EmbMatrix</code></p>
<p>I will then perform transformations on this tensor and use the output in a loss function. The most important thing for me is that the gradients are propagated nicely to the Embedding Matrix and the updates change both the <strong>encoder</strong> and <strong>t</strong> in the same manner, i.e. in every training iterations:</p>
<p><code>EmbMatrix == t[i, j]</code></p>
<p>Any help is appreciated! Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ac1d: <p>So far, I have the following:</p>
<pre><code class="lang-auto">encoder = nn.Embedding(vocab_size, emb_size)
l = []
for i in range(seq_length * batch_size):
    l.append(encoder.weight)
t = torch.stack(l).view(seq_length, batch_size, vocab_size, emb_size)
</code></pre>
<p>I’m not sure I can reuse the embedding matrix like this and also even if this is correct, I really feel like there has to be a better way of doing this <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
During the training phase, how to fix subset of the weights of NN?

Animesh_Kumar_Paul: <p>I want to build NN in which <strong>some of the weights always be 0</strong>, and other weights will be optimized through the training process.</p>
<p>                    <a href="https://i.stack.imgur.com/3efTO.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.stack.imgur.com/3efTO.png" width="514" height="500">
          </a>

</p>
<p>From the <a href="https://stackoverflow.com/questions/43851657/fixing-a-subset-weights-in-neural-network-during-training" rel="nofollow noopener">StackOverflow</a>, I found something closer for tensorflow.</p>
<p>But, I want to implement this in Pytorch.</p>
<p>Can anyone please help me?</p>

ptrblck: <p>You could create a mask for the constant weights and zero out their parameters after calculating the gradients with <code>loss.backward()</code>.</p>