akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
An efficient way to slice a torch tensor

username: Abe
cooked: <p>How can I efficiently implement this code? Implementing without the loop.</p>
<pre><code class="lang-auto">
import torch
torch.manual_seed(0)
tensor = torch.rand(2,4,3,3)
def sort_tensor(tensor):
        # average pooling
        polling = torch.nn.AdaptiveAvgPool2d(1)
        tensor_pooled = polling(tensor)
        # sorting the averaged pooled values and getting the indexes 
        sorted_tensor_pooled, sorted_indx = torch.sort(tensor_pooled, 1, descending=True)
        sorted_indx = sorted_indx.squeeze()
        
        #sorting the orginal tensor based on the sroting index
        for i in range(len(sorted_indx)):
            if i == 0:
                sorted_tensor = tensor[i, sorted_indx[i], :, :].unsqueeze(dim=0)
            else:
                sorted_tensor = torch.cat((sorted_tensor, tensor[i, sorted_indx[i], :, :].unsqueeze(dim=0)), dim=0)

        return sorted_tensor
print('sorted', sort_tensor(tensor))

</code></pre>

username: Kushaj
cooked: <p>There is no pytorch function that can do what you are looking for. So you have to use a for loop. Your implementation can be made more efficient by creating a new tensor and then copying the contents to it, rather than increasing the current tensor by using <code>cat</code>.</p>
<pre><code class="lang-python">def sort_tensor(tensor):
    tensor_pooled = nn.AdaptiveAvgPool2d(1)(tensor).squeeze()
    
    _, sorted_indx = torch.sort(tensor_pooled, 1, descending=True)
    
    # Create a new tensor where the result would be stored
    sorted_tensor = torch.empty(tensor.size())
    for dim_0 in range(tensor_pooled.shape[0]):
        for dim_1 in range(tensor_pooled.shape[1]):
            sorted_tensor[dim_0, dim_1] = tensor[dim_0, sorted_indx[dim_0, dim_1]]
    
    return sorted_tensor
</code></pre>

username: chetan_patil
cooked: <p>Hi <a class="mention" href="/u/abe">@Abe</a> , I have converted the loop into a vectorized format.<br>
I have verified with multiple inputs, and it works much faster.<br>
Here’s the code.</p>
<pre><code class="lang-auto">def sort_tensorV2(tensor):
        # average pooling
        polling = torch.nn.AdaptiveAvgPool2d(1)
        tensor_pooled = polling(tensor)
        print("Polling shape ", tensor_pooled.shape)
        # sorting the averaged pooled values and getting the indexes 
        # print("Tensor-pooled ", tensor_pooled)
        sorted_tensor_pooled, sorted_indx = torch.sort(tensor_pooled, 1, descending=True)
        sorted_indx = sorted_indx.squeeze()

        a, b, c, d = tensor.shape
        mults = np.arange(0,a*b,b).reshape(-1,1)
        temp_indices = (mults+sorted_indx).flatten()
        temp = tensor.reshape(a*b,c,d)
        final = temp[temp_indices,:,:]
        sorted_tensor = final.reshape(a,b,c,d)
        return sorted_tensor
</code></pre>
<p>Hope it helps.!</p>
Module 'torch.utils.data' has no attribute 'Dataloader'

username: Mohammad_Islam
cooked: <p>Here is my code snippets:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f4254c5b9525802a5eca8cae9a0a0e178b4ffd.png" data-download-href="https://discuss.pytorch.org/uploads/default/e1f4254c5b9525802a5eca8cae9a0a0e178b4ffd" title="p1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1f4254c5b9525802a5eca8cae9a0a0e178b4ffd.png" alt="p1" data-base62-sha1="weSvEIDijIv8cQtHhVSIJPEvMo5" width="690" height="224" data-dominant-color="EFF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">p1</span><span class="informations">882×287 11.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
and Here’s the error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a7894a5d08fbc1fc825e9e7716327a5800c488c.png" data-download-href="https://discuss.pytorch.org/uploads/default/9a7894a5d08fbc1fc825e9e7716327a5800c488c" title="p1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a7894a5d08fbc1fc825e9e7716327a5800c488c.png" alt="p1" data-base62-sha1="m2vS90xku1dTKAxYWkBGjqzZsKM" width="690" height="169" data-dominant-color="F5F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">p1</span><span class="informations">758×186 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I’m a beginner so I would also like to know the mistake in details…</p>

username: Caruso
cooked: <p>Its DataLoader not Dataloader, just a little typo ^^</p>

username: Mohammad_Islam
cooked: <p>Thanks a lot…How could I miss that!!!</p>
Why would model loading takes that long?(usually takes 2 seconds to complete)

username: zarzen
cooked: <p>Hi there,<br>
I am curious about why the first time job launch cost so much time in PyTorch.<br>
e.g. load the model into CPU, move the model to GPU.</p>
<p>Here is the code snippet to reproduce:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
from torchvision import models
import torch
import time

def only_import():
    """"""
    t1 = time.time()

    model = torch.hub.load('pytorch/vision:v0.4.2',
                           "resnet152",
                           pretrained=True)
    # util.set_fullname(model, MODEL_NAME)
    t2 = time.time()

    model = model.to('cuda')
    t3 = time.time()

    print("import model takes", (t2 - t1) * 1e3, 'ms')
    print("move model to cuda", (t3 - t2) * 1e3, 'ms')

def main():
    """"""

    for _ in range(10):
        proc = mp.Process(target=only_import)
        proc.start()
        proc.join()


if __name__ == "__main__":
    mp.set_start_method("spawn")
    main()
</code></pre>
<p>Base on the log, it usually takes 1.5s to 2.2s to load the model to CPU. and over 2 seconds to load the model to cuda.</p>
<p>Is this because PyTorch need to dynamically load libraries from disk, at the first time?</p>
<p>(experiment setup: aws-ec2-p3.2xlarge intance, PyTorch 1.3.0)</p>

username: Kushaj
cooked: <p>The first time to use cuda, cudnn has to allot some cache and that takes time. If you pass a dummy input to GPU like <code>torch.randn(4).to(device)</code> then after that you will normal transfer speed.</p>
Is PyTorch pip3 package for aarch64 available?

username: sakaia
cooked: <p>Hello</p>
<p>I try to use PyTorch package on aarch64.<br>
I do following commands and intstalls torch-0.1.2.post2 on AWS m6g.<br>
Is there any plan to update it?</p>
<pre><code class="lang-bash">sudo apt update
sudo apt install python3-pip
pip3 install -r requirements.txt
git clone http://github.com/pytorch/examples
cd examples/mnist
time python3 main.py --no-cuda  --epochs 1
</code></pre>
<p>outputs are follows</p>
<pre><code class="lang-auto">Collecting torch
  Downloading torch-0.1.2.post2.tar.gz (128 kB)
</code></pre>
Removing words from a string

username: cevvalu
cooked: <p>I am trying to remove specific words from my string to have a clear output. Words are adding one by one in a loop:</p>
<p>text += ’ ’ + word</p>
<p>text looks like this for example:</p>
<p>start this is a sentence end<br>
(need to remove start and end)</p>

username: albanD
cooked: <p>If guess you can add</p>
<pre><code class="lang-auto">if word in ["start", "end"]:
  continue
</code></pre>
<p>To skip the words you don’t want?</p>

username: cevvalu
cooked: <p>this could work but im generating the text so i need start and end as flags to understand it begins and ends. I want to remove them after i have a string like text = start this is a sentence end<br>
so it will be more clear</p>

username: albanD
cooked: <p>If you’re doing this after the fact, you can use the regular python string builtin to replace a string with another: <code>sentence.replace("start", "")</code>.<br>
Or if you always want to remove the begining and end, you can do <code>sentence[len("start"):-len("end")-1]</code>.</p>

username: dunefox
cooked: <p>Your first suggestion replaces every occurrence in the string with “”.</p>

username: cevvalu
cooked: <p>thanks, second solutions worked well</p>
How long should it take to train the mobilenet model on a 1080Ti?

username: aditya-hari
cooked: <p>So I’m training a mobilenet model available in torchvision using a 1080Ti and CUDA 9. The total size of my dataset is roughly ~4000 images</p>
<p>With minibatches of 16, it’s taking me ~4 minutes for training and ~1 minute for validation per epoch. Even with minibatches of size 64, I only see a marginal improvement in the time.</p>
<p>Is this an expected amount of time? It feels rather slow to me for a 1080Ti.</p>

username: Caruso
cooked: <p>Whats your image size? 256x256?</p>
<p>And do you use any DataLoader or do some data processing in the loop?</p>
Gradient clipping in Pytorch c++ libtorch

username: Aaditya_Chandrasekha
cooked: <p>I am trying to invoke a gradient clipping in C++ similar to this line of code in Python :</p>
<pre><code class="lang-auto">torch.nn.utils.clip_grad_norm_(Net.parameters(),0.2)
</code></pre>
<p>In C++ I  am calling</p>
<pre><code class="lang-auto">torch::nn::utils::clip_grad_norm_(Net-&gt;parameters(), 0.2);
</code></pre>
<p>Can anyone kindly verify that this is the right call ?</p>

username: ptrblck
cooked: <p>The usage look correct and is also used in this way in <a href="https://github.com/pytorch/pytorch/blob/0dded4026e627dd5c4cdd94ccc9068ca35819fc1/test/cpp/api/nn_utils.cpp#L67">this test</a>.</p>
Companies using TorchScript in Production

username: Param
cooked: <p>I am looking at using pytorch and TorchScript as the DL framework for my company.  One of the datapoint we were looking for was to check if there are companies (other than FB) which have used TorchScript/C++ already in production.  Could not find info with searching on the internet.  Any inputs would be appreciated.</p>
<p>Thanks</p>

username: yf225
cooked: <p><a class="mention" href="/u/michael_suo">@Michael_Suo</a> Would you like to chime in on this one? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: soldierofhell
cooked: <p>If you want to use C++ then you’ve got no choice, TF C++ API is dead. If you mean pure inference TF is better choice.</p>

username: Param
cooked: <p>Selecting framework would mean fx should be good at (1) Model development (2) Production serving.</p>
<p>For model development,  I think python is a reasonable choice (inspite of GIL issue it has).</p>
<p>For production inference TorchScript is a solution to productionize the model,  so wondered if some companies/people already tried it out and what the findings were.</p>

username: sahni
cooked: <p>Hi <a class="mention" href="/u/param">@Param</a>, I am curious as to how you finally proceeded and what was the deciding factor? I am in a similar situation right now and could use some advice.</p>
Nan value in convolution output

username: zeus
cooked: <p>I am getting <strong>nan</strong> value in the convolution output like this <code>[ 6.6468e+00,  7.1471e+00, -7.2259e+00,         nan]</code>.</p>
<p>I am using efficient net. Please help me.</p>

username: Isaac_Kargar
cooked: <p>Hi, Can you please put the code here for reproduction?</p>

username: zeus
cooked: <p>It is not the output of the convolution that  produces nan but the input to the loss function. It is due to divisibility by negligible amount I think.</p>

username: zeus
cooked: <p>log of a negative decimal value is resulting nan value. Do you know how to solve this?. Should take absolute difference or make relu?</p>
How to implement action detection with RNN

username: thxkew
cooked: <p>I tried to implement the action detection model using LSTM follow this figure</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e164fbca86db29ad3429053208019c9a4c991fbc.png" alt="image" data-base62-sha1="w9VMNILheqq1XgM1591CMIud4Qs" width="322" height="232"></p>
<p>where x is the sequence feature extracted from a CNN model, in this case, I used Resnet50, and y is a class predicted in each time step.<br>
I use batch size 1 and Lr is 0.00001, and the target of each sequence looks like</p>
<pre><code class="lang-auto">[32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,  
0,  0,  0,  0, 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  
0,  0,  0,  0, 7,  7,  7,  7,  7,  0,  0,  0,  0,  0,  0,  0,  0,  0,  
0,  0,  0,  0, 0,  0,  0,  0,  0,  0,  0, 36, 36, 36, 36, 36]
</code></pre>
<p>I computed loss function in every time step using Cross-Entropy and average them.</p>
<p>I found the loss graph looks unusual.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05818358df6e855f232c2ff7e15f08e2cb0cfd3c.png" data-download-href="https://discuss.pytorch.org/uploads/default/05818358df6e855f232c2ff7e15f08e2cb0cfd3c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05818358df6e855f232c2ff7e15f08e2cb0cfd3c_2_690x276.png" alt="image" data-base62-sha1="MHRNXsXrgynb5dSA6l8SHxpP8o" width="690" height="276" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05818358df6e855f232c2ff7e15f08e2cb0cfd3c_2_690x276.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05818358df6e855f232c2ff7e15f08e2cb0cfd3c_2_1035x414.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/5/05818358df6e855f232c2ff7e15f08e2cb0cfd3c_2_1380x552.png 2x" data-dominant-color="DBE8F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">4000×1600 540 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>do you have any idea what happened?<br>
and I’m not clear about my loss function computation.<br>
what is the correct way to compute loss in this case?<br>
is it possible that the problem come from loss function computation?</p>
<p>thank you,</p>

username: Isaac_Kargar
cooked: <p>Hi, Can you explain more about your training data? Is it one single frame? or a sequence of frames? How is your RNN model? Is it many to one or many to many?</p>

username: thxkew
cooked: <p>thank you for your reply,<br>
the training data is sequence of frames and the RNN model is many to many type.</p>
Compute accuracy for 4d torch tensors

username: spadel
cooked: <p>I want to compute the classification accuracy, while my targets and predictions have the shape (batch_size, nb_classes, x_size_image, y_size_image). The accuracy is the number of correctly classified pixels divided by the total number of pixels. Is there an easy way to do that?</p>

username: Rohan_Kumar
cooked: <p>I am assuming that the target and prediction batches are arranged like this -  that there are <code>x_size_image*y_size_image pixels</code>, and that each pixel corresponds to one of the n classes which are in a one hot representation. if not then correct me.<br>
so basically just convert it into (n of pixels , no. classes) by doing</p>
<pre><code class="lang-auto">src = src.view(-1, src.shape[1])
trg = trg.view(-1, src.shape[1])
</code></pre>
<p>to get the most likely output use argmax or softmax as you need,</p>
<pre><code class="lang-auto">trg = trg.argmax(1) 
src = src.argmax(1)
</code></pre>
<p>and just sum the indices where these 2 are equal and divide by the length of either.</p>
Finding groups of pixels

username: harrycb
cooked: <p>I’m trying to find groups of pixels within a mask as shown here:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/027997a7c045da52b602bcc0b64100ed05710bb7.png" data-download-href="https://discuss.pytorch.org/uploads/default/027997a7c045da52b602bcc0b64100ed05710bb7" title="Screenshot from 2020-05-22 22-01-18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/027997a7c045da52b602bcc0b64100ed05710bb7_2_118x500.png" alt="Screenshot from 2020-05-22 22-01-18" data-base62-sha1="lTsRf3e5WEnAUiKuLnbIDbfIhh" width="118" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/027997a7c045da52b602bcc0b64100ed05710bb7_2_118x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/027997a7c045da52b602bcc0b64100ed05710bb7_2_177x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/2/027997a7c045da52b602bcc0b64100ed05710bb7.png 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-22 22-01-18</span><span class="informations">197×831 5.93 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Circles are initially generated with the code:</p>
<pre data-code-wrap="py"><code class="lang-py">t = torch.zeros(3,200,200)
im = to_pil_image(t)
draw = ImageDraw.Draw(im)

for _ in range(3):
    s, e = rint(180),  rint(180)
    
    draw.ellipse((s, e, s+20, e+20), fill = 'blue')
t = to_tensor(im)
</code></pre>
<p>Here is how I’m getting those groups of pixels:</p>
<pre data-code-wrap="py"><code class="lang-py">def get_all_pixels_near(group, pixels, distance=2):
    stacked = group.repeat(1, pixels.shape[0]).view(-1, pixels.shape[0], 2)
    ds = stacked.sub(pixels).pow(2).sum(2).float().sqrt()
    for i in ds:
        group = torch.cat([pixels[i&lt;distance], group])
    return group.unique(dim=0)

def find_clusters(mask, masks=[], distance=2):
    mask = mask.clone()
    ungrouped_pixels = mask.nonzero()[:, 1:]
    
    if len(ungrouped_pixels) &lt; 1: 
        return masks
    
    p = ungrouped_pixels[0]

    prev_len = len(p)
    near = get_all_pixels_near(p[None], ungrouped_pixels, distance)

    while prev_len != len(near):
        prev_len = len(near)
        near = get_all_pixels_near(near, ungrouped_pixels)
    
    m2 = mask.clone()
    for i in near:
        m2[0,i[0],i[1]] = .5
        mask[0,i[0],i[1]] = 0
    m2[m2!=.5] = 0 
    m2[m2==.5] = 1
    return find_clusters(mask, masks+[m2])
</code></pre>
<p>Is there is a more efficient way to do this?</p>
Splitting data with percentage information

username: pastelDiplo
cooked: <p>Hello, everyone!</p>
<p>The data I use to train model are intertwined. For example, an image belogs to the A folder with a rate of 15% and 85% to the B folder. There is no clear distinction. How can I separate my data in this case? How should I create folders for train and val so that I can use percentage information in the data?</p>
<p>Thank you!</p>

username: Kushaj
cooked: <p>If you have two classes 0, 1. Now if an image belongs 15% to 0 and 85% to 1, you can interpolate the output class as <code>0*.15 + 1*.85</code> which is same as using the probability of it belonging to class 1 as the class label.</p>

username: pastelDiplo
cooked: <p>Thank you for your reply. Still, there is an unclear part for me. Let’s say I’ve two classes: A and B. Could you please tell me if I should name folders differently (e.g.: First folder: A15B85, Second folder: A30B70) to interpret this percentage information or simply name the folders as “A” and “B”?</p>

username: Kushaj
cooked: <p>For this task you cannot use that style of dataset. Instead store all the images in a single folder. Now how you get the information that it belongs to 85% folder B depends on you.</p>
<p>You can have this info in image file names, or create a separate csv file matching each filename to prob.</p>
<p>Then you would have to create a new dataset class.</p>
Sliding window over image

username: elad1
cooked: <p>Hi, i’m trying to compute a dot product between a sliding window to an image(shaped (1024,2048)), in a way that may be resemble to convolution operation.<br>
My goal is that each pixel score will be dependent on it environment scores. currently i came up with two ways of doing it:</p>
<pre><code class="lang-python">import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

kernel_size = 3
pmap = torch.rand((1024, 2048)).to(device)
kernel = torch.rand((3, 3)).to(device)

padding = tuple([kernel_size//2] * 4)
padded_pmap = torch.nn.functional.pad(pmap, padding)

# method1
patches = padded_pmap.unfold(0,kernel_size,step=1).unfold(1,kernel_size,step=1)
kerenelized_pmap = (patches * kernel).sum((2,3))

# method2
kerenelized_pmap2 = torch.zeros_like(pmap)
for i in range(padded_pmap.shape[0] - kernel_size):
    for j in range(padded_pmap.shape[1] - kernel_size):
        kerenelized_pmap2[i,j] = (padded_pmap[i:i+kernel_size,j:j+kernel_size] * kernel).sum((0,1))
</code></pre>
<p>the problem is that method1 uses a lot of memory and if i want to use large kernels(50+), i will exceed my device memory(and even with smaller kernels)<br>
and method2 is very slow. My next step will be to try split the image and then use method1,but i wanted to see maybe someone already faced similar issues and has better solutions.<br>
Thanks</p>
Why deployment is favored in Tensorflow?

username: ilyes
cooked: <p>Helloo,<br>
I have a question of bit general. I am wondering why people say TensorFlow is better than PyTorch  in deployment? Because I see a lot of companies integrating their deep learning models in their base code using Pytorch!<br>
Because at the end all you need is an input, and you shall give back an output for further processing. Botch frameworks support multi-gpo/cpu computing. So the question is why, or what do people actually mean by deployment? it would be better If someone gives a concrete example with technical details.<br>
Thank you</p>

username: vladium
cooked: <p>A few reasons that I am aware of:</p>
<ol>
<li>pytorch <a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">APIs for optimizing</a> and otherwise preparing a model for production are still fairly new and rough around the edges</li>
<li>FB described themselves as <a href="https://research.fb.com/wp-content/uploads/2017/12/hpca-2018-facebook.pdf" rel="noopener nofollow ugc">relying on caffe for production inference</a> – but caffe seems to have been “merged” into pytorch in 2018 and the result for us the end users seems to be a somewhat confused mix of possible runtimes (or “backends”): python, JIT traced, JIT scripted, ONNX. If you quantize a model you might not be able to export it to ONNX. Or even save it. And so on – things just don’t feel properly cooked yet.</li>
</ol>
<blockquote>
<p>Because at the end all you need is an input, and you shall give back an output for further processing.</p>
</blockquote>
<p>Hmm, a lot of people want more. Can you run the above on mobile? How about a production environment that doesn’t have python at all?</p>
Confusion matrix with one single image to test

username: julioeu99
cooked: <p>Is it possible, using opencv, to test one single image in a confusion matrix with a model that is already trained? If is possible, can someone give me a code example?<br>
Thank you</p>

username: julioeu99
cooked: <p>Any ideas? <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a> ?</p>

username: Kushaj
cooked: <p>What do you mean by testing one single image in a confusion matrix?</p>

username: julioeu99
cooked: <p>After train a cnn with mnist, make a confusion matrix with the classes from mnist and a single image, to test the prediction of the model. Do you understand now?</p>

username: Kushaj
cooked: <p>To create a confusion matrix you can use</p>
<pre><code class="lang-python">def get_preds(model, data_loader):
    with torch.no_grad():
        preds = torch.tensor([]) # It will hold tuple (predicted_label, real_label)
        for batch in loader:
            images, labels = batch # labels are already real_labels (0,1,2, ...)

            pred = model(images) # pred.shape = [batch_size, num_classes]
            pred = torch.argmax(-1) # this will give you index of the predicted class
            preds = torch.cat((pred, labels), dim=0)
    return preds # shape is [total_images, 2]

# Now you can use a for loop to get the confusion matrix
num_classes = 4
c_mat = torch.zeros(num_classes, num_classes, dtype=torch.int64)

for p in preds:
    i,j = p[0], p[1] # Add code here to convert indices to int
    c_mat[i,j] += 1
</code></pre>
<p>I still don’t understand what you mean by <code>and a single image</code>.</p>

username: Nikronic
cooked: <p>Can you explain the purpose of using a single image in confusion matrix?</p>
<p>MNIST has 10 classes so confusion matrix will be 10x10. By testing only one image, you get a single <code>1</code> in <code>10x10</code> of zeros. You cannot calculate recall, precision, f1, etc on such a matrix.</p>

username: julioeu99
cooked: <p>Of course, i will try to explain better. I want to test one image, to check in wich class my model predict that image. I can use a confusion matrix for that, or not?</p>

username: Nikronic
cooked: <p>You do not need confusion matrix for a single image. Confusion matrix is used for large number of inputs to get statistical measurement of model’s behavior. For instance, in training stage, you may have report Accuracy. This optained from confusion matrix, so for single image, it is 0 or 100% accuracy.</p>
<p>For any number of input images, depends on your model, you get a result like <code>[samples, classes]</code>. For a single image it is like <code>pred=[1, classes]= [[0.1, 0.9, ..., 0.3]]</code>. By taking argmax over this tensor, you will get the index of predicted class which in our example it would be <code>torch.argmax(pred, dim=-1)</code>. It will return <code>1</code> as the index of maximum. So, there is no need to define confusion matrix for a single image.<br>
Your model predict class <code>1</code> and you already have true label.</p>
Speeding up the training process and utilising the hardware optimally

username: bing
cooked: <p>Hi ,<br>
I am trying to speed up the training process and use the resources optimally. My GPU utilization is low but my GPU memory is full, I am really not able to understand, where is the bottleneck and how to approach it.<br>
Below are my system results -<br>
<img src="/uploads/default/original/3X/7/3/73404cba4879522cdcd20b93831254b8ae3f1d12.png" alt="image" data-base62-sha1="gryFzPmxY3K8o8iNRQKZsyQfy26" width="690" height="336"><br>
<img src="/uploads/default/original/3X/5/8/587fd3affc62b7aca872a572cbc77ae3becffc9e.png" alt="image" data-base62-sha1="cCTST0DLShJT9KeMWDY7Ewp3Dxc" width="690" height="338"><br>
<img src="/uploads/default/original/3X/3/4/34aad315747f999afd653d0da791d5670c7251ce.png" alt="image" data-base62-sha1="7vUOJNpEoDcR8TBoQaIEzvclomy" width="689" height="340"></p>
<p>I am training the 10k images which are resized from size 2048,2048 to size 512,512.<br>
I want to train it further on larger resolution too but it is maxing out at 1024,1024. But, my GPU utilisation is really low.<br>
Batch size is 20.<br>
Is there any way to allocate less memory and maximise GPU utilisation.<br>
Also, any other suggestions to optimise it is hugely appreciated.</p>

username: Kushaj
cooked: <p>If GPU memory is getting full, then one reason might be, you are accumulating history somewhere. Like you do when you do <code>total_loss += loss</code> rather than doing <code>total_loss += float(loss)</code>.</p>

username: bing
cooked: <p>Oh, I never knew about it. Yes, I am doing it -<br>
<code> train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))</code><br>
and valid loss<br>
<code>valid_loss +=  valid_loss + ((1 / (batch_idx + 1)) * (loss.data - valid_loss))</code>.<br>
I’ll change it , are there any more such issues which results in accumulating history.<br>
Thanks.</p>

username: Kushaj
cooked: <p>Mainly this. And for the computation for which you do not want gradient use <code>with torch.no_grad():</code>.</p>
CUDA version mismatch

username: harshildarji
cooked: <p>When I run my scripts on this remote GPU, it takes more than double the time than it normally takes on Google Colab.</p>
<p>The CUDA version of remote GPU is <code>9.1.x</code> while the PyTorch I installed requires <code>10.2</code>.</p>
<p>Is it possible this version mismatch is causing increased timing?</p>
<p>Is it possible to upgrade CUDA on the remote server?</p>

username: Kushaj
cooked: <p>Remove pytorch from the server. Then you can use <code>!conda install pytorch torchvision cudatoolkit=10.2 -c pytorch</code> in jupyter to download pytorch with cuda 10.2. If you use pip then use <code>!pip install torch torchvision</code>.</p>

username: harshildarji
cooked: <p>I already did that, still the output of <code>cat /usr/local/cuda/version.txt</code> shows <code>CUDA Version 9.1.85</code>.</p>

username: Kushaj
cooked: <p>What is the output of <code>torch.version.cuda</code>?</p>

username: harshildarji
cooked: <p>The output of <code>torch.version.cuda</code> is <code>10.2</code>.</p>

username: Kushaj
cooked: <p>So pytorch is using cuda 10.2</p>

username: Kushaj
cooked: <p>Also, conda install cuda in anaconda directory where all conda packages are stored. So checking cuda version of local machine will not matter.</p>

username: harshildarji
cooked: <p>Well, I agree but why isn’t it showing under the directory <code>/usr/local/</code> as similar to <code>9.1.x</code>?<br>
Also, time it takes for training is also way higher compared to Google Colab.</p>

username: Kushaj
cooked: <p>If you are using conda, then it is not installed in <code>/usr/local/</code>. It is in <code>~anaconda/envs/{something}</code>.</p>

username: Kushaj
cooked: <p>There may be other reasons for the slowdown. Slow CPU, worse GPU, slow storage type.</p>
CNN: random padding instead of random cropping?

username: FaultyBagnose
cooked: <p>In order to square-ify images, it is common to crop them, rather than use padding, which could be a problem if images have key information at the edge. An argument for cropping seems to be that you can randomize it for training images, which has the nice side effect of some low-cost data augmentation.</p>
<p>Could you not do the same with padding? I.e. for validation pictures add equal amount of padding left and right (top/bottom for landscape), and for training images randomize the amount of padding on either side?<br>
Anyone know if this is done, and if not, why?</p>

username: Kushaj
cooked: <p>Padding is not used as it is just wasted computation. The network is not learning anything from those pixels.</p>
<p>Also, when we have to resize our images. There are 3 different approaches</p>
<ol>
<li>Squish them:- In real life we do not see squish images, so working with these does not make much sense. As images are distorted in various ways.</li>
<li>Use padding:- usually try to avoid it, as you can do much more by using a small image size</li>
<li>Random Crop:- This is the preferred appraoch. This allows your network to learn and generalize better.</li>
</ol>
<p>If you have a dog image. With random cropping, you can use different regions of an image to train your neural network, which forces the network to generalize. Also, as you said it is a good data augmentation technique.</p>
What(): cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:51

username: Abdul_Basit
cooked: <p>I start to run TensorFlow application on terminal. At the same time, when I start Pytorch application on another terminal I get error</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=3 : initialization error
terminate called after throwing an instance of 'std::runtime_error'
  what():  cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:51

</code></pre>
<p>both Pytorch TensorFlow and running on different virtualenvs</p>
<p><strong>My Environment</strong><br>
Ubuntu 18.04<br>
GPU GeForce GTX 1060<br>
Pytorch env (torch==1.1.0, torchvision==0.2.0)<br>
Tensorflow env (tensorflow-gpu==1.15.0)</p>
<p>Pytorch application was running smoothly before starting tensorflow application</p>

username: Abdul_Basit
cooked: <p>I stop the TensorFlow application but error not goes back</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.device_count()
0
&gt;&gt;&gt; torch.cuda.is_available()
False


</code></pre>

username: Kushaj
cooked: <p>Starting tensorflow could not have done anything, I think. If you restart machine and if the pytorch env works then it might have been due to tensorflow. Also, try upgrading to latest pytorch version, maybe it was a bug that was fixed.</p>
How to save certain weight of model

username: yolearn
cooked: <p>I have a question about saving model.</p>
<p>I have 2 tasks which output is not the same.<br>
The first one is a classifier.<br>
The second one is a NER.</p>
<p>I would like to fine-tune a dataset via Roberta for task 1.<br>
I want to save the weight of model, and just save the weight origin Roberta has.<br>
And them use this weight to add down-stream layer and fine-tune second model.</p>
<p>Can model.save_state do that?</p>
<p>Or I should load all layer and trunct the layer I don’t need.</p>

username: Pranavan_Theivendira
cooked: <p>Hi,</p>
<pre><code class="lang-auto">torch.save(model.state_dict(), path)
</code></pre>
<p>Using this you can save a model.</p>
<p>If you want to load the model, you can do the following.</p>
<pre><code class="lang-auto">model = torch.load(path)
</code></pre>
<p>This topic is widely discussed in this forum. You can find other useful threads as well.</p>
<p>Thanks<br>
regards<br>
Pranavan</p>

username: yolearn
cooked: <p>Hi I know this function. But I just want to load the certain layer instead of all.<br>
Assuming I use base-bert and connect 2 layers for downstream task.<br>
The bert’s layers are 12 and plus 2 layers I connect, therefore total is 14.</p>
<p>And I just want the original 12 layers which already fine-tuned.</p>

username: dunefox
cooked: <p>You can define a model with only the first 12 layers, then load the state_dict into it with strict=false. That should work (assuming the layers have the original names). Otherwise you could iterate over the model parameters like so:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    ... do something with each parameter
</code></pre>
<p>I think the first version is better.</p>
Gradient is None when using torch.sum() or torch.abs()

username: cijerezg
cooked: <p>I have the following code:</p>
<pre><code class="lang-auto">for count, params in enumerate(agent.actor.parameters()):
     print(params.grad)
</code></pre>
<p>where agent is a class that has the neural network: actor. If I run that code, it prints the gradients, which is a tensor of size 4x5. Now, if I do something like this:</p>
<pre><code class="lang-auto">for count, params in enumerate(agent.actor.parameters()):
     print(torch.sum(params.grad))
</code></pre>
<p>or even this</p>
<pre><code class="lang-auto">for count, params in enumerate(agent.actor.parameters()):
     print(torch.abs(params.grad))
</code></pre>
<p>Then I get the error: <code>TypeError: sum(): argument 'input' (position 1) must be Tensor, not NoneType</code></p>
<p>The error is clear, but then why do the gradient become None when I try to do something with them? and how do I correct that? This is a pretty weird error. Moreover, I had done something similar, when the neural net was not part of a class, but declared explicitly, that is:</p>
<pre><code class="lang-auto">for count, params in enumerate(actor.parameters()):
     print(torch.abs(params.grad))
</code></pre>
<p>where again, actor is a neural net that was explicitly declared (it’s coming from an imported class)</p>
<p>I’d appreciate any help with this! I know this is very weird. I can provide more details about the agent class if necessary, but I don’t think that’s the problem, because it works when I just print the gradients themselves.</p>

username: ptrblck
cooked: <p>The operations such as <code>torch.sum</code> and <code>torch.abs</code> shouldn’t change any behavior and I assume you might have used the code snippets in different parts of your original code.</p>
<p>Before the first <code>backward</code> call, all <code>.grad</code> attributes are set to <code>None</code>.<br>
After the gradients were calculated for the very first time or after you’ve called <code>.zero_grad()</code> on the model or optimizer, the <code>.grad</code> attributes will be filled with values or zeros, respectively.</p>
<p>Let me know, if you get stuck and feel free to post a reproducible code snippet.</p>

username: cijerezg
cooked: <p>Thanks! I just realized what the problem was. Your answer was pretty helpful. Basically, during my first iteration, I don’t call <code>.backward()</code>, so the gradients are None, but I call <code>backward()</code> after the second iteration and on, that’s why it prints gradients, but can’t do <code>torch.sum()</code>.</p>
Finetuning RoBERTa on a custom Entailment data

username: holoholger
cooked: <p>Hi there,</p>
<p>I want to use fairseq / RoBERTa as a sentence-pair classification task (entailment), similar to the demo code here: <a href="https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md" rel="nofollow noopener">https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md</a></p>
<pre><code class="lang-auto">Use RoBERTa for sentence-pair classification tasks:
roberta = torch.hub.load('pytorch/fairseq', 'roberta.large.mnli')
tokens = roberta.encode('Roberta is a heavily optimized version of BERT.', 'Roberta is not very optimized.')
roberta.predict('mnli', tokens).argmax()  # 0: contradiction

</code></pre>
<p>Two questions:</p>
<ol>
<li>
<p>How do I fine tune the RoBERTa model with my own dataset for entailment?<br>
I’ve seen and understood the IMDB example for a custom classification task. But I’m not sure how I have to pre-process the data for entailment tasks rather than 1/0 classification tasks on one single text rather than a text pair</p>
</li>
<li>
<p>I have a dataset which is a bit different from the MultiNLI dataset in the following way:</p>
</li>
</ol>
<ul>
<li>My dataset uses a lot of technical terms (think “deoxyribonucleic acid” and words like that). Will finetuning do the job to work on these words / my custom dictionary?</li>
<li>I only have “neutral” and “entailment” training data, but I also don’t really need “contradiction”. Will this work anyway?</li>
</ul>
<p>I’d highly appreciate some help or hints</p>
Loading weights from a previous epoch doesn't seem to do anything

username: alx
cooked: <p>I’m saving my model state_dict at every epoch.</p>
<p>When overfitting, if I load back my state_dict from a previous epoch it seems like the train accuracy continues to go up as if nothing happened. However, if I stop the training and restart from that previous state_dict it works.</p>
<p><strong>While training:</strong><br>
Epoch 30 - Train Acc 70%<br>
…<br>
Epoch 40 - Train Acc 85%<br>
<em>load epoch 30 weights</em><br>
Epoch 41 - Train Acc 86%</p>
<p><strong>If I restart the notebook:</strong><br>
<em>load epoch 30 weights</em><br>
Epoch 41 - Train Acc 71%</p>
<p>How can I completely go back to the state of a certain epoch while training without interrupting training. I just want the weights from the previous epoch, not the momentum etc…</p>

username: ptrblck
cooked: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/how-do-you-properly-load-weights-while-training/81871/4">here</a> or is this another unrelated issue?</p>

username: alx
cooked: <p>Yes, sorry, completely forgot. Please delete</p>
How to take max not over padding tokens

username: Kevin_Shen
cooked: <p>I have the output of a Bert model that is H = [batch_size, sequence_length, hidden_size]. I have another tensor indicating the sequence length of each sample in the batch, L = [batch_size, 1].</p>
<p>How would I compute the max of H over the sequence length (dim=1) without including the padding tokens (i.e. only including the positions less than that specificied by L)?</p>
<p>I have a solution for taking the mean,</p>
<pre><code>mask = torch.arange(H.shape[1]).repeat(H.shape[0], 1)
mask = (mask &lt; L).float()
mask[:, 0] = 0
masked_h = H * mask.unsqueeze(2)
mean_emb = (masked_h.sum(dim=1)/L)</code></pre>

username: ptrblck
cooked: <p>Your approach looks, but it would calculate the mean, wouldn’t it?</p>

username: Kevin_Shen
cooked: <p>Ya sorry, maybe that part of the post is a bit irrelevant. I meant to say I know how to calcualte the mean but calculating the max seems trickier and I have yet to figure it out.</p>

username: ptrblck
cooked: <p>You could use a similar approach using a mask.<br>
I don’t know what values your padding tokens have, but you could set the mask values for all padding values to a large negative value (<code>-Inf</code> should work), multiply <code>H</code> with this mask, and apply <code>max()</code> on it.</p>
<p>Let me know, if that would work.</p>

username: Kevin_Shen
cooked: <p>This is also the solution I thought of but the problem is a hidden vector is float32 in general and can contain negative values. Multiplying by -inf gets +inf. The padding tokens can be assumed to be 0 for simplicity.</p>
Understanding Linear layer batch size

username: Siyovush_Kadyrov
cooked: <p>Hello,<br>
I have been struggling with determining how the batching of the Dataloader works with nn.Module. As far as I understand any last 2 dimensions of the given tensor are accepted as an input however I cannot see this to be the case on the following piece of code:</p>
<pre><code class="lang-auto">class FNet(nn.Module):
    def __init__(self, **kwargs): 
        super(FNet, self).__init__()
        self.fc1        = nn.Linear(32678, 1024)
        self.fc2        = nn.Linear(1024, 256)
        self.fc3        = nn.Linear(256, 1)
     
    def forward(self, X): 
        X = F.relu( self.fc1( F.relu(X) ) )
        X = F.relu( self.fc2(X) )
        
        return F.sigmoid( self.fc1(X) )
</code></pre>
<pre><code class="lang-auto">train_dataloader = DataLoader(train_dataset, batch_size=train_batch, shuffle=True)

model.train()
for batch_idx, (data, labels) in enumerate(loader):
        optimizer.zero_grad()
        print(data.shape, data.dtype)
        data = data.to(device)
        labels = labels.to(device)
        
        # Forward pass: Compute predicted y by passing x to the model
        data = data.reshape(train_batch, 1, 32768).float()
        labels_pred = model(data)

        loss = criterion(labels_pred, labels.long())
        loss.backward()
        optimizer.step()
</code></pre>
<p>On the code given above, I try to resize the input to get a size of (batch_size, 1, 32768) so that 1x32768 should be gotten as input dimensions. However I get the following error:</p>
<p>RuntimeError: size mismatch, m1: [4 x 32768], m2: [32678 x 1024]</p>
<p>I will be grateful for any response. Thank you.</p>

username: Abdulrahman
cooked: <p>Your input in nn.Linear is <code>32678</code> but you reshape data into `32768’. You replaced 7 with 6 in your code.</p>

username: Siyovush_Kadyrov
cooked: <p>Thank you so much, didn’t see that error</p>
Variable causing GPU meomory full

username: Jaideep_Valani
cooked: <p>Below loss_func returns variable for whom require grad has to be explicitly made requires grad=True  or else i get an error message during .backward saying there is grad fn for this.<br>
But   few epochs this variable results cuda meomory error,i keep very low batch size.</p>
<pre><code class="lang-auto">CustomLossfunc(nn.Module):

def forward 
loss=torch.tensor(loss_func(torch.round(inputs), targets,  )-1).requires_grad_(True).cuda()
        return  -loss
</code></pre>

username: ptrblck
cooked: <p>Are you storing the <code>loss</code> tensor in a list or another container?<br>
This would increase the memory usage. However, the tensor should not be attached to the computation graph based on your description, which seems to be another unrelated issue.</p>
<p>Could you remove the new tensor creation and just try to use:</p>
<pre><code class="lang-python">loss = loss_func(torch.round(inputs), targets) - 1
</code></pre>
<p>Also, I’m not sure about the derivative of round, but I assume it should be zeros almost everywhere?</p>

username: Jaideep_Valani
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks  I was doing it the way  u suggested but the thing is variable returned by function which is sckcit metric function I think is not having require grad as true, so loss.backward  threw exception saying no grad  or grad fn existing, so that is why I had to add this ,it resolved the error but is now always resulting into GPU meomory error with just 2 to 3 iterations of epoch .</p>

username: ptrblck
cooked: <p>If you are using functions of other libraries, such as numpy or <code>sklearn</code>, you would have to implement the backward method manually, since Autograd cannot track these operations.<br>
Wrapping the loss into a new tensor with <code>requires_grad=True</code> creates a new computation graph from this point, so that all preceding operations are detached (basically your model’s forward pass) and your model won’t get any valid gradients.</p>
<p>You could write a custom <code>autograd.Function</code> with the manual backward method as explained <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a> or you would have to use PyTorch methods only.</p>

username: Jaideep_Valani
cooked: <p>is this same for every function or specific to Relu ? i want to call cohenkappa in function of skcit. in fwd</p>
<p><code> grad_input[input &lt; 0] = 0</code></p>

username: ptrblck
cooked: <p>The <code>MyReLU</code> implementation is just an example in the tutorial and you could of course use the built-in <code>nn.ReLU()</code> module.<br>
You would need to implement a custom <code>autograd.Function</code> e.g. if you are leaving PyTorch.<br>
If you want to call the <code>kappa</code> method from <code>sklearn</code>, you would have to write the backward pass manaully.<br>
However, if I’m not mistaken Cohen’s Kappa score is not contiguous and not differentiable, is it?<br>
(There might be approximations which are differentiable, so you should check out the current research.)</p>