akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Large memory allocated of nn.Conv2d when kernel_size=7

username: 4d81ebbc4d3e703a2917
cooked: <p>When I run <code>conv_mem(kernel_size=7, padding=3)</code> in the following demo, I find the allocated max_memory is 1591.0751953125M while <code>conv_mem(kernel_size=5, padding=2)</code> is 69.2568359375M,  is it normal?</p>
<pre><code class="lang-auto"># -*- coding: utf-8 -*-
import torch
import torch.nn as nn


def memory_usage_stats(gpu_id=0):
    allocated_mem = torch.cuda.memory_allocated(gpu_id) / 1024 / 1024
    max_allocated_mem = torch.cuda.max_memory_allocated(gpu_id) / 1024 / 1024
    return allocated_mem, max_allocated_mem


def conv_mem(kernel_size, padding):
    conv3d = nn.Conv2d(256, 256, kernel_size=kernel_size, padding=padding, dilation=1, groups=1, bias=False).cuda()
    x = torch.rand(1, 256, 132, 240, device='cuda')
    y = conv3d(x)
    allocated_mem, max_allocated_mem = memory_usage_stats(0)
    print("allocated: {}M".format(allocated_mem))
    print("allocated max_memory: {}M".format(max_allocated_mem))


if __name__ == '__main__':
    # conv_mem(kernel_size=1, padding=0)
    # result:
    # allocated: 63.0751953125M
    # allocated max_memory: 63.2568359375M

    # conv_mem(kernel_size=3, padding=1)
    # result:
    # allocated: 65.0751953125M
    # allocated max_memory: 71.4501953125M

    # conv_mem(kernel_size=5, padding=2)
    # result:
    # allocated: 69.0751953125M
    # allocated max_memory: 69.2568359375M

    conv_mem(kernel_size=7, padding=3)
    # result:
    # allocated: 75.0751953125M
    # allocated max_memory: 1591.0751953125M

</code></pre>
Pytorch Random Number Generation Algorithm

username: MaybeAustin
cooked: <p>I was trying to find out what Pytorch is using for random number generation. I think I see in  “THCTensorRandom.cpp” that Pytorch uses Mtgp32 from curand. Anyone know if this is correct? Thanks.</p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/pytorch/pytorch/pull/21301" rel="nofollow noopener">this recent PR</a> the <code>curandStateMTGP32</code> usage was removed and instead <code>curandStatePhilox</code> seems to be used now.</p>

username: tom
cooked: <p>One might add that the Philox genrator has been used before, too, but now it’s the only one (for CUDA on master/nightlies).</p>
Controlling requires_grad

username: daniel0
cooked: <p>When i have a tensor with requires_grad=False, some operations appear to set requires_grad=True on the output.</p>
<p>For example, this is the case for torch.conv2d, but not the case for torch.relu.</p>
<p>The documentation for the operators does not seem to indicate which behavior will be used.</p>
<p>Is there documentation for which operators exhibit this behavior? Is there some way to control it, other than disabling requires_grad on the output?</p>
<p>Thanks.</p>
Torch Dataset: Generate examples directly on GPU

username: Andreas_Bloch
cooked: <p>I’m training a network which takes triples of indices as inputs: (u,i,j).</p>
<p>Now, “u” and “i” are predefined. The number “j” is drawn uniformly at random from a large set.</p>
<p>I know how I could pre-store “u” and “i” on the GPU (as done for example in: <a href="https://discuss.pytorch.org/t/how-to-put-datasets-created-by-torchvision-datasets-in-gpu-in-one-operation/34587" class="inline-onebox">How to put datasets created by torchvision.datasets in GPU in one operation?</a>) However, it would be prohibitively expensive to pre-generate and store samples of “j” on the GPU. Further, I’d like “j” to vary a lot with each epoch, so generating it randomly would be much better.</p>
<p>For now my approach is to generate “j” on the fly (at random), storing it in a CPU tensor, and then transferring the tensor to the GPU. Is there some way to generate “j” directly on the GPU (without having to transfer it)?</p>
Poor minima while backpropagating affine_grid?

username: kekday
cooked: <p>Hi,<br>
I was just trying an experiment where I try to compute the affine transformation matrix from a given pair of images (original and transformed image). For this example I just use a small 5x5 grid with a straight line as the original image and the line tilted at 45 degrees as the transformed output. For some reason, it seems the loss comes down and the gradients become smaller and smaller (obviously). But the solution it converges to seems to be way off (totally does not look like a straight line).</p>
<p>Apologies for the formatting (the code is from a notebook)</p>
<pre><code class="lang-auto">import numpy as np
import matplotlib.pyplot as plt
import torch.optim as optim
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(989)

# source_image = torch.tensor([[0,1,0],[0,1,0],[0,1,0]])
source_image = torch.tensor([[0,0,1,0,0],[0,0,1,0,0],[0,0,1,0,0],[0,0,1,0,0],[0,0,1,0,0]])

plt.imshow(source_image)

# transformed_image = torch.eye(3)
transformed_image = torch.eye(5)

plt.imshow(transformed_image)

source_image = source_image.reshape(1, 1, source_image.shape[0], source_image.shape[1])
transformed_image = transformed_image.reshape(1, 1, transformed_image.shape[0], transformed_image.shape[1])
source_image = source_image.type(torch.FloatTensor)
transformed_image = transformed_image.type(torch.FloatTensor)

class AffineNet(nn.Module):
    def __init__(self):
        super(AffineNet, self).__init__()
        self.M = torch.nn.Parameter(torch.randn(1, 2, 3))
    def forward(self, im):
        flow_grid = F.affine_grid(self.M, transformed_image.size())
        transformed_flow_image = F.grid_sample(transformed_image, flow_grid, padding_mode="border")
        return transformed_flow_image

affineNet = AffineNet()
optimizer = optim.SGD(affineNet.parameters(), lr=0.01)
criterion = nn.MSELoss()

for i in range(1000):
    optimizer.zero_grad()
    output = affineNet(transformed_image)
    loss = criterion(output, source_image)
    loss.backward()
    if(i%10==0):
        print(i, loss.item(), affineNet.M.grad)
    optimizer.step()

print(affineNet.M)

printme = output.detach().reshape(output.shape[2], output.shape[3])
plt.imshow(printme.cpu())
</code></pre>
<p>This is what the end result looks like: <img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a1be5847cab65f3a342c8f92221b6adea0cf6e16.png" alt="Screenshot%20from%202019-06-14%2016-31-24" data-base62-sha1="n4QGrXu311d4oQZ7iWVzq0ZB3Tg" width="240" height="246"></p>
<p>PS: It does seem to work fine if you mess around with the commented lines and use a 3x3 grid rather than a 5x5. Can someone help me understand why this is happening?<br>
Please let me know if a gist would be more helpful.</p>
How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?

username: pamessina
cooked: <p>In order to not repeat myself again, here is a link to the full detailed question on DataScience Stackexchange: <a href="https://datascience.stackexchange.com/questions/53814/pytorch-how-to-implement-nested-transformers-a-character-level-transformer-for" rel="nofollow noopener">link</a></p>
Why do sequences need to get sorted for packing?

username: josmi9966
cooked: <p>In order to make an LSTM deal with variable length sequences in a batch, I have to use pack_padded_sequence first. But why do the sequences for a batch have to be sorted in decreasing order of length? In order to match each of the sequences with the correct training signal, I then have to unsort the output of the unpacked sequences again. I do not understand why this reordering is necessary, it makes everything a whole lot more complex in the code.</p>
<p>What I also do not understand is why there is no option to instead pass a mask on to the LSTM so that the LSTM only runs over those elements where the mask is true/1. That way, we could achieve the same as with packing/unpacking but for the original padded sequences.</p>
<p>One final question: if I use a uni-directional LSTM and I know the lengths of the sequences but still run the LSTM on the padded sequences, then if I am just interested in the final output/hidden state for the sequence, I could just manually pick the proper element from the output, right? However for bidirectional LSTMs there is no way to simulated the processing of the packed sequence with just the padded sequences, because the LSTM will always run over the pad elements as well which will change the hidden states?</p>

username: Brando_Miranda
cooked: <p>Crossposted: <a href="https://www.quora.com/unanswered/Why-do-sequences-need-to-get-sorted-for-packing" rel="nofollow noopener">https://www.quora.com/unanswered/Why-do-sequences-need-to-get-sorted-for-packing</a></p>
<p>hopefully we’ll get an answer some day.</p>
What's the meaning of this 'state' in optimizer?

username: kangkang_li
cooked: <h2><a name="image624x397uploadu8wgxmeof4qzuesainadkcbvzzupng-1" class="anchor" href="#image624x397uploadu8wgxmeof4qzuesainadkcbvzzupng-1"></a><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d337c9f7a24047573e2476b71f8776c5bf97bfa2.png" alt="image" data-base62-sha1="u8wgXmeof4QzUESAINaDkCBvzZU" width="624" height="397"></h2>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9a245edc81e7196b1c680fe66f31553d09fb995e.png" alt="image" data-base62-sha1="lZBs9aodIbRAwo7G38TCrG9UrM2" width="632" height="157"></p>
<p>7974 is the loop until finishing train. And the shape of exp_avg and exp_avg_sq are both 512.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/fd6316f4dc70d2ebbfed0adec2882bc514620e01.png" alt="image" data-base62-sha1="A9z67hXF8s8G1Yi1lQangPS5cLn" width="486" height="186"><br>
gt and gt2 ?If the net is overfit, should I build a new optimizer for the purpose of cleaning the exp_avg and exp_avg_sq? Or just modify the lr.</p>

username: kangkang_li
cooked: <p>anyone can help? Can I clean these states?</p>

username: ptrblck
cooked: <p>I’m not sure if manipulating the internal optimizer states are related to counter overfitting.<br>
What would be the idea behind it?</p>
Efficient way to calculate angles of normals between to tensors

username: aleurs
cooked: <p>I have two images of normals. The target and the prediction (size: 3 x 224 x 224). An image is 224 * 224 pixels and each pixels has a x-, y- and z-component.</p>
<p>Now I want to compute the angle between each normal of the prediction and the target. So I go through the image pixel by pixel and calculate the angle between the normals at each pixel.</p>
<p>First I convert the image 3x224x224 -&gt; 3 x 50176, so that I have a row of normals</p>
<p>Here’s my code:</p>
<pre><code>for j in range(target.size()[1]):
   	angle = math.fabs(angle_between(prediction[:, j].data, target[:, j].data))

def angle_between(v1, v2):
    return np.arccos((v1[0] * v2[0] + v1[1] * v2[1] + v1[2] * v2[2]) / 
                 (math.sqrt(v1[0]*v1[0] + v1[1]*v1[1] + v1[2]*v1[2]) * 
                  math.sqrt(v2[0]*v2[0] + v2[1]*v2[1] + v2[2]*v2[2])))
</code></pre>
<p>This code works fine but is very slow. I heard that for-loops are quite slow in python.</p>
<p>I couldn’t find a solution so far to get rid of the for loop.</p>
<p>Do you have an idea how to increase the performance here? At the moment each image takes 13 sec. Which means for my 142200 images it would take ~21 days.</p>

username: chenchr
cooked: <p>suppose variable <code>a</code> and <code>b</code> both with shape n * 3. then:</p>
<pre><code class="lang-auto">inner_product = (a * b).sum(dim=1)
a_norm = a.pow(2).sum(dim=1).pow(0.5)
b_norm = b.pow(2).sum(dim=1).pow(0.5)
cos = inner_product / (2 * a_norm * b_norm)
angle = torch.acos(cos)
</code></pre>
<p>Is it what you want ?</p>

username: aleurs
cooked: <p>Thanks. That’s what I needed.</p>

username: edgarriba
cooked: <p>Use F.cosine_similarity and specify the axis where you have your normals. Will return 1 for similar, 0 different, -1 opposite</p>
Multiplication of 1D and 2D Tensor?

username: zapnuk
cooked: <p>Hi,<br>
<strong>in short</strong>:<br>
My problem is that I have a mask/tensor of shape [B] and want to multiply it with a tensor of shape [B, 3] like this:</p>
<blockquote>
<p>[1, 0] * [  [1,2,3] , [4, 5, 6] ] = [  [1 * 1, 1 * 2, 1 * 3], [0 * 4, 0 *5 , 0 * 6] ] = [ [1, 2, 3], [0, 0, 0] ]</p>
</blockquote>
<p>How do I multiply such a mask of shape [Batchsize] with my positions of shape [Batchsize, 3]?</p>
<p>In my implementation, the multiplication is understood as dot product, which raises a “RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 1” (Batchsize=2)</p>
<p><strong>in long:</strong><br>
I’m trying to implement a custom loss function that combines classification and (conditional) regression loss similar to this post <a href="https://discuss.pytorch.org/t/loss-function-conditional-on-two-outputs/8296/5" class="inline-onebox">Loss function conditional on two outputs - #5 by vabh</a>.</p>
<p>In my case, the model has 5 outputs. Two should refer to the class and the other three refer to a position.<br>
I want to add the regression loss only if the target has a specific class.</p>
<p>I tried to adopt his approach to use <code>(target_class == 1).float()</code> as a mask. Which results in the error mentioned above.<br>
`</p>
<pre><code>class LocateLoss(nn.Module):
    def __init__(self):
        super(LocateLoss, self).__init__()
        self.cross_entropy_loss = nn.CrossEntropyLoss()
        self.b_cross_entropy_loss = nn.BCEWithLogitsLoss()
        self.soft_max = nn.Softmax(dim=1)


    def forward(self, input, target):
        input_class, input_loc = torch.split(input, [2, 3], dim=1)
        input_class = self.soft_max(input_class)

        target_class, target_loc = torch.split(target, [1, 3], dim=1)
        target_class = target_class.reshape(1, -1).squeeze()
    
        input_loc = input_loc * (target_class == 1).float()

        l1 = self.cross_entropy_loss(input_class, target_class)
        l2 = self.b_cross_entropy_loss(input_loc, target_loc)
        return l1 + l2
</code></pre>
<p>`</p>

username: a3VonG
cooked: <p><strong>In short:</strong></p>
<pre><code class="lang-auto">A = torch.tensor([1,0])
B = torch.tensor( [ [1,2,3] , [4, 5, 6] ] )
A = A.unsqueeze(1)
C = A * B
print(C)
</code></pre>
<p>Which gives as output:</p>
<pre><code class="lang-auto">C = tensor([[1, 2, 3],
        [0, 0, 0]])
</code></pre>
<p>You can use <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="nofollow noopener">broadcasting</a> for these type of problems.</p>
Gradients through sampling operation?

username: DuttaAbhigyan
cooked: <p>I was wanting to know whether gradients are propagated through sampling layer. For example let’s say I have defined my own new custom Dropout layer where I can define the probability of the dropout for each node in a layer. The sampling operation is done through an external function. So wil lPyTorch raise an error or is everything ok?</p>
<p>NOTE: I have defined the dropout probabilities as a function of weights from previous layer.</p>

username: InnovArul
cooked: <p>As long as you handle passing back the gradients correctly to the appropriate connections in the custom layer (in your case, a new dropout layer), pytorch will have no issues AFAIK.</p>

username: DuttaAbhigyan
cooked: <p>Hello! So is there any way to check whether gradients are being passed or not? Also, what I meant through gradient passing through sampling layer is that my dropout layer has different probability for each nodes and the probabilities are trainable parameters. Check the “Adaptive Dropout” by Ba et al for details, but in summary we use special trainable weights and inputs from previous layer to create dropout probabilities for the successive layer. Of course this is done by going through a sampling operation.</p>
<p>Now, I want to know whether the gradients are flowing through this dropout mask  and training the weights with which we create the dropout probabilities?</p>
<p>NOTE: I do not really know whether what I am talking is mathematically not possible, but as far as what I want is that when 1 is sampled then I would like to backpropagate through these special weights of this dropout layer. You know something similar to ReLu, if positive backpropagate else 0 gradient.</p>
Solve linear system with Variable matrix

username: spaceinvader
cooked: <p>Hello!<br>
I have to solve linear system with Variable matrix and then to compute gradients with respect to it.<br>
Is there any way to perform this operation?</p>

username: richard
cooked: <p>I think you’re looking for <a href="http://pytorch.org/docs/master/torch.html?highlight=gesv#torch.gesv"><code>torch.gesv</code></a></p>

username: spaceinvader
cooked: <p>Thanks a lot, it was rather hard to find this function!</p>

username: D-X-Y
cooked: <p>gesv can only support the matrix A being square. Is it possible to solve linear system for non-square A in PyTorch?</p>

username: JuanFMontesinos
cooked: <p>Linear system for non-square is an overparametrized linar system which is solved by minimization, typically svd. Im not sure if it’s differentiable</p>

username: isarandi
cooked: <p>For non-square A, you should look at gels, not gesv. <a href="https://pytorch.org/docs/stable/torch.html#torch.gels" rel="nofollow noopener">https://pytorch.org/docs/stable/torch.html#torch.gels</a></p>
Accessing nn.Module mode

username: barthelemymp
cooked: <p>Hello,</p>
<p>While making a very custom NN, I had to take into account in the forward the fact of being in train mode or not.<br>
I discovered that I could use the function model.train(True) , but I don’t know how to access this mode.</p>
<p>I need to get boolean tensor in order to make an if statement in the forward:</p>
<p>def forward(self, x):<br>
is_train = self.mystery_fn()<br>
if (is_train):<br>
…<br>
else:<br>
…</p>
<p>so far I have build my own is_train parameter, in the init and the functions to change it, but I guess using the proper Pytorch approved one would be better.</p>
<p>Best regards</p>
<p>Barth</p>

username: ptrblck
cooked: <p>You can access this flag via <code>self.training</code>.</p>

username: barthelemymp
cooked: <p>Thank you very much !</p>
How to change the output dimension of nn.Linear according to the batch data?

username: Yingqiang_Gao
cooked: <p>I’m doing machine translation with pytorch. Because I use cnn and maxpooling in the decoder, the decoder reduces the total time steps of the target sentence. Now I want to do a upsampling to reconstruct the time steps, so it can have the same length as the original target sentence. I find it hard to implement since the target sentences have different length. Here is my model class:</p>
<pre><code class="lang-auto">class Archi(nn.Module):
def __init__(self, src_vocab_size, ref_vocab_size, d_model=512, num_block=3, heads=8, dropout=0.1):
    super().__init__()
    self.encoder = Encoder(src_vocab_size=src_vocab_size, d_model=d_model, num_block=num_block, heads=heads, dropout=dropout)
    self.decoder = Decoder(ref_vocab_size=ref_vocab_size, d_model=d_model, num_block=num_block, heads=heads, dropout=dropout)
    self.to_vocab = nn.Linear(d_model, ref_vocab_size)
    self.to_original_length = nn.Linear(????????)

def forward(self, src, ref, src_mask, ref_mask):
    enc_outputs = self.encoder(src=src, mask=src_mask)
    dec_outputs = self.decoder(ref=ref, enc_outputs=enc_outputs, mask=ref_mask)
    output = self.to_vocab(dec_outputs) 
    output = self.to_original_length(output)
    return output
</code></pre>
<p>Let’s say I have to target sentence, the first one has length of 10 and the second one has length of 15. But decoder generates two representations of length 5 and 7 correspondingly, so I want to use a Linear layer to project the dimension back to 10 and 15, but since the target sentences have different length, I can’t define the linear layer in the init function, otherwise for the coming sentences there will be dimension errors. Is there a way to deal with this without introduce the target sentence into the init function?</p>
One hot encoding of label map

username: che85
cooked: <p>Let’s say I have a label map with 0: background and 1: label</p>
<p>I am implementing an autoencoder that takes the input label and extends it to be a multi-label where each label is converted into a separate channel.</p>
<p>Now my question: When using the <code>one hot encoder</code> with a label map that has only one label (1), would the number of classes that I put into the <code>on hot encoder</code> be C=1 or C=2?</p>
<p>I am not sure if the <code>one hot encoder</code> converts the 0 values in its own channel. Would it?</p>
<p>Thanks a lot for your response.</p>

username: che85
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Maybe you have an answer for this? I hope my question is clear enough.</p>
Retrieving embedding gradients

username: GeoChrys
cooked: <p>Hi,</p>
<p>I’m implementing a model for a binary NLP classification task with a bi-RNN and an Attention mechanism on top and I would like to get the gradients of the embeddings with respect to the predominant predicted class. My issue is I found various approaches to obtain the gradient and they yield various results. The approaches I tried are:</p>
<ol>
<li>
<p>torch.autograd.grad( loss, model.embedding.weight ) and retrieve element 0.</p>
</li>
<li>
<p>To add retain_grad() on the embeddings  and retrieve the gradient after calling loss.backward() by doing embedding.grad</p>
</li>
<li>
<p>To backward through the output by y_predicted.backward(  (y_predicted &gt; 0.5 ).float()  ) where the (y_predicted &gt; 0.5) will return a mask of equal size to y_predicted with 1’s occupying the predominant class. Following this backward I then retrieve the gradient with embedding.grad.</p>
</li>
</ol>
<p>Can anyone advise on which of these three (if any) is the correct approach? If not is there a more robust way to obtain the embedding gradients starting from the predominant class?</p>
<p>Thanks!</p>
<p>George</p>
Anaconda install pytorch from local directory

username: wxystudio
cooked: <p>I use anaconda prompt to install pytorch in the following command:<br>
conda install pytorch torchvision cudatoolkit=10.0 -c pytorch<br>
But because of the bad network I got a HTTP ERROR, so I download the four required packages:<br>
cudatoolkit-10.0.130-0.tar<br>
ninja-1.9.0-py37h74a9793_0.tar<br>
pytorch-1.1.0-py3.7_cuda100_cudnn7_1.tar<br>
torchvision-0.3.0-py37_cu100_1.tar<br>
And I save these packages in the C:\Users\user.conda\pkgs. then I want to run the conda install command to install the pytorch, how can I install these local packages and avoid re-downloading them?</p>
<p>question 2: I also try to use pip install to install pytorch, and I try to download the “whl” file in pytorch website, but I can only get the torch1.0.0 in the previous version, how can I  get the newest version torch1.1.0 “whl” file in the website?</p>
<p>Please help me thx</p>

username: ptrblck
cooked: <ol>
<li>
<p>Did you try to call <code>conda install PATH_TO_TAR</code>?</p>
</li>
<li>
<p>This URL might work: <code>https://download.pytorch.org/whl/cu100/torch-1.1.0-cp37-cp37m-linux_x86_64.whl</code></p>
</li>
</ol>
Where is CuDNN installed?

username: throwaway-acct
cooked: <p>I installed PyTorch along with CUDA toolkit and (presumably) CuDNN. The install appears to work well: <code>torch.backends.cudnn.version()</code> returns something reasonable, and CUDA tensors are well-behaved.</p>
<p>Now I’m trying to install some other DL packages, and I’d like to set my <code>LD_LIBRARY_PATH</code> so that those packages can use the same CuDNN as PyTorch. I’m trying to avoid installing conflicting copies of CuDNN and breaking things. However, I’ve found that PyTorch’s CUDA install (at <code>/usr/local/cuda-10.0</code>) doesn’t actually include CuDNN in its <code>lib64</code> directory – or anywhere else, it appears. Calling <code>find /usr/local/cuda-10.0 -name "*cudnn*"</code> returns nothing at all.</p>
<p>So I’m wondering, where does PyTorch install CuDNN?</p>

username: JuanFMontesinos
cooked: <p>Does PyTorch install cuda?</p>
<p>CuDNN is another nvidia library and I’d say you should install it your self. It’s typically copied in cuda folder but if you want a system with several pairs cuda/cudnn you may save it in a different one.</p>
<p>Anyway im not an expert so consider this as a weak insight</p>

username: ptrblck
cooked: <p>The PyTorch binaries will not install a complete CUDA and cudnn library on your system.<br>
I’m not familiar with the build process of PyTorch, but e.g. if you’ve used conda to install it, you’ll find the <code>torchlib.so</code> in <code>~/anaconda3/envs/YOUR_ENV_NAME/lib/python.3.7/site-packages/torch</code>.<br>
Running <code>ldd</code> on it gives:</p>
<pre><code class="lang-auto">	libcudart.so.10.0 =&gt; ...
        ...
	libcusparse.so.10.0 =&gt; ...
	libcurand.so.10.0 =&gt; ...
	libcufft.so.10.0 =&gt; ...
</code></pre>
<p>I’m not sure, if cudnn is linked in <code>libcudart.so</code> or separately.</p>
<p>That being said, if you would like to build other libraries with CUDA (using <code>nvcc</code>), you should install CUDA (and cudnn) directly on your system as <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> said.<br>
If possible, try to use the same CUDA version which is used in your PyTorch install, as this might avoid some incompatibility issues.</p>
Not able to run torch on mac

username: Shivank_Yadav
cooked: <p>I have downloaded torch as described on this link.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/get-started/locally/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">Start Locally</a></h3>

  <p>Start Locally</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I have no GPU and therefore I selected Cuda as none.<br>
Then I tried to import torch on Jupyter notebook.<br>
I got this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/75ec98e16d3eb15165dc62d618cf6b3e3e793a4c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/75ec98e16d3eb15165dc62d618cf6b3e3e793a4c" title="38%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/75ec98e16d3eb15165dc62d618cf6b3e3e793a4c_2_690x431.jpeg" alt="38%20PM" data-base62-sha1="gPcLOqFPswGN5ngIH6UBd3yQWao" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/75ec98e16d3eb15165dc62d618cf6b3e3e793a4c_2_690x431.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/7/75ec98e16d3eb15165dc62d618cf6b3e3e793a4c_2_1035x646.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/75ec98e16d3eb15165dc62d618cf6b3e3e793a4c_2_1380x862.jpeg 2x" data-dominant-color="DADADA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">38%20PM</span><span class="informations">2880×1800 773 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have tried both pip and anaconda instructions. Result is the same.<br>
What should I do?</p>

username: ptrblck
cooked: <p>Could you try to install <code>libomp</code> via:</p>
<pre><code class="lang-auto">brew install libomp 
</code></pre>
<p>as mentioned in <a href="https://github.com/pytorch/pytorch/issues/20030" rel="nofollow noopener">this issue</a>.</p>
How to map RGB label map to grey label map?

username: MariosOreo
cooked: <p>Hello there,</p>
<p>If I have some label maps in RGB format, I want to convert them to Grey format and feed them to the networks. <strong>Is it necessary to convert the 3 channels label map to single channel one?</strong></p>
<p>For the single channel map, just using <code>mask[mask == pixel_value] = class_value</code> can achieve it.</p>
<p>Is there a convenient method (like single channel map with one line code) to achieve this goal?<br>
e.g.</p>
<pre><code class="lang-auto">mask_mapping = {
    [255,255,255] : 0,
    [255,255,0]   : 1,
    [255,0,0]     : 2,
    ...
}
</code></pre>
<p>Thanks in advance!</p>

username: MariosOreo
cooked: <p>Okay, I have solve it by myself, but I don’t know whether it is the <strong>efficient one</strong>.<br>
If you have other ideas, I will also appreciate it!</p>
<pre><code class="lang-auto">mask_mapping = {
    (255, 255, 0):   0,
    (0, 255, 255):   1,
    (255, 255, 255): 2,
}
for k in mask_mapping:
    label[(label == k).all(axis=2)] = mask_mapping[k]
</code></pre>
How to load pictures in C++ with Libtorch?

username: RuFAI
cooked: <p>Hi, the LibTorch library does not support any preprocessing functions for loading and resizing pictures etc. as torchvision does in python. What ist the best practice to do so in C++? I am aiming for an openCV approach. I this best practice or is there a better way?</p>
<p>Thank you</p>

username: RuFAI
cooked: <p>Push. Nobody here who can answer the question?</p>

username: dambo
cooked: <aside class="quote quote-modified" data-post="6" data-topic="47701">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dambo/48/29088_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/libtorch-c-convert-a-tensor-to-cv-mat-single-channel/47701/6">Libtorch C++ convert a Tensor to cv:Mat (single channel</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    From cv2 to TORCH: 
 cv::cvtColor(frame, frame, CV_BGR2RGB);
 frame.convertTo(frame, CV_32FC3, 1.0f / 255.0f);
 auto input_tensor = torch::from_blob(frame.data, {1, frame_h, frame_w, kCHANNELS});
 input_tensor = input_tensor.permute({0, 3, 1, 2});

From TORCh to cv2: 
input_tensor = input_tensor.to(at::kCUDA);
                torch::Tensor out_tensor = module-&gt;forward({input_tensor}).toTensor();

                out_tensor = out_tensor.squeeze().detach().permute({1, 2, 0});
                out_t…
  </blockquote>
</aside>
Error in glow examples

username: Sharath_R
cooked: <p>I built glow from github and when I ran the examples (./bin/mnist) , I’m getting errors as follows<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3747b02e65299d64a5191f33624fcc56be626a88.png" data-download-href="https://discuss.pytorch.org/uploads/default/3747b02e65299d64a5191f33624fcc56be626a88" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3747b02e65299d64a5191f33624fcc56be626a88_2_690x226.png" alt="image" data-base62-sha1="7T1QPFDVCt8y8XOH186EaT4hGLC" width="690" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3747b02e65299d64a5191f33624fcc56be626a88_2_690x226.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3747b02e65299d64a5191f33624fcc56be626a88_2_1035x339.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/3747b02e65299d64a5191f33624fcc56be626a88.png 2x" data-dominant-color="39152D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1150×377 64.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Any idea on how to solve this?</p>

username: jfix
cooked: <p>I believe there is a PR up to fix this issue <a href="https://github.com/pytorch/glow/pull/3097" rel="nofollow noopener">here</a>. It should land tomorrow.</p>
Compute the whole gradient of a mini-batch using the accumulated gradient of small mini-batches

username: Dhorka
cooked: <p>Hi,</p>
<p>My doubt is related to the possibility of compute the whole gradient of a mini-batch using the accumulated gradient of small mini-batches(I named this technique batch partition I do not know if it is the common name ).  As is explained <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">here</a> Methods 1 and 2 should give the same results, right? I did some tests that you can find in this <a href="https://colab.research.google.com/drive/1ZoC-_ox_XfkKEgIGf18EAHudlvX03eF9" rel="nofollow noopener">colab</a>. As you can see the results are different. In addition, I tested a modification version that averages the accumulated gradient by the number of partitions of the mini-batch, but it does not help.</p>
<p>It is expected this behaviour? If it is not, can you please me give some hint what am I doing wrong? If it is expected, can you help me to understand why is it expected? As far as I understood the results should be the same…</p>
<p>Thanks,</p>

username: jerinphilip
cooked: <p>The default loss reduction seems to be <code>mean</code> [<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html" rel="nofollow noopener">docs</a>]. Perhaps set it to <code>sum</code> and divide by total batch size in both cases will get  you the same gradient?</p>

username: Dhorka
cooked: <p>It should be the same, right? I mean, if you have the numbers 4 7 12 15. The mean of all of them is the same if you compute the sum of all of them and divide by the number of elements or if you compute first the mean of the first numbers, following the mean of the last numbers and the mean of both.</p>
<p>In addition, I am accumulating the gradients, the loss is used in each small mini-batch. As is explained <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">here</a>.</p>

username: Dhorka
cooked: <p>Sorry for insist, but I would like to know if this behavior is expected and if not how I can solve the issue. I hope someone can help me. Thanks.</p>
Read a pytorch project and get confused?

username: Steve_Hu
cooked: <pre><code class="lang-auto">    def forward(self, input):
        #TODO confused
        input = input.view((-1, ) + input.size()[-3:])
        if self._representation in ['mv', 'residual']:
            input = self.data_bn(input)

        base_out = self.base_model(input)
        return base_out

</code></pre>
<p>anyone can tell me this line function?</p>
<pre><code class="lang-auto">input = input.view((-1, ) + input.size()[-3:])
</code></pre>

username: god_sp33d
cooked: <p>Hi,</p>
<p>“.view” reshapes the tensor to a new size. It takes in, the new size to which we want our tensor to be reshaped. Now, in your code, what you are trying to achieve is keep the last 3 dimensions the same (because of [-3:]) and combine the initial dimensions.</p>
<p>For ex:</p>
<pre><code class="lang-auto">a = torch.rand((1,2,3,4,5,6))
a  = input.view((-1,) + input.size()[-3:])
</code></pre>
<p>Initial shape of the tensor = (1,2,3,4,5,6)<br>
after resizing the new shape = (6,4,5,6) because we kept the last 3 dimensions the same and combined the rest which is 1x2x3.</p>

username: Steve_Hu
cooked: <p>thanks a lot!  it’s a clear and concise explanation!</p>