akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gradiant jacobian matrix output image with respect to input image

username: Sam3
cooked: <p>Hi,<br>
i try to compute jacobian gradient output image with respect to input image(for each pixels) but i have some errors .<br>
does anyone have an idea about the right code?</p>
<p>Thanks</p>

username: Sam3
cooked: <p>image = Image.open(’/Users/lionardo/Desktop/3.png’)<br>
x = TF.to_tensor(image)<br>
x = Variable(x, requires_grad = True)</p>
<p>image = Image.open(’/Users/lionardo/Desktop/4.png’)<br>
y = TF.to_tensor(image)<br>
y = Variable(y, requires_grad = True)<br>
def jacobian(y, x, create_graph=False):</p>
<pre><code>jac = []                                                                                          
flat_y = y.reshape(-1)                                                                            
grad_y = torch.zeros_like(flat_y)
flat_x = x.reshape(-1)
grad_x = torch.zeros_like(flat_x)

for i in range(len(flat_y)) :                                                                    

for j in range (len(flat_x)) : 
    grad_y[i] = 1. 
    grad_x[j]=1.
    grad_x, = torch.autograd.grad(flat_y, flat_x, grad_y,grad_x, retain_graph=True, create_graph=create_graph)

    jac.append(grad_x.reshape(x.shape))
    jac.append(grad_y.reshape(y.shape)) 
    grad_y[i] = 0.  
    grad_x[j] =0.
return torch.stack(jac).reshape(y.shape + x.shape)</code></pre>

username: albanD
cooked: <p>You can find correct code in <a href="https://gist.github.com/apaszke/226abdf867c4e9d6698bd198f3b45fb7">this gist</a> to get the jacobian of the output y wrt an input x.</p>

username: Sam3
cooked: <p>thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>
`torch.cdist` cannot handle large matrix on GPU

username: Zhaoyi-Yan
cooked: <pre><code class="lang-python">import torch


m = 2500
c = 256

# works fine for cpu
x = torch.randn(1, m, c)
# raise an error: RuntimeError: CUDA error: invalid configuration argument
# x = torch.randn(1, m, c).to("cuda:0")

x.requires_grad = True

dist = torch.cdist(x, x, p=2)

dist.sum().backward()

</code></pre>
<p>It’s not OOM. I found some similar links posted in pytorch github. <a href="https://github.com/pytorch/pytorch/pull/31593" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/31593</a><br>
<a href="https://github.com/pytorch/pytorch/pull/31167" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/31167</a><br>
I am not sure if these get merged, whether it can deal with my case.</p>
<p>Is there some workaround for this ?</p>

username: Eta_C
cooked: <p>I see <a class="mention" href="/u/ptrblck">@ptrblck</a> in the first link.<img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: ptrblck
cooked: <p>My PR should fix your issue. I’ll continue working on it this week. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Zhaoyi-Yan
cooked: <p>It will be very nice. Hope it can be solved ASAP, since I really need this to push my work forward.</p>
Change batch size based on loss

username: vainaijr
cooked: <p>How do I increase the batch size when loss decreases and decrease the batch size when loss increases, that is dynamic batch size based on loss during training and evaluation?</p>

username: Eta_C
cooked: <p>I have no idea. Maybe</p>
<pre><code class="lang-python">class Dataset:
    def __init__(self, data_path):
        self.data = load_data(data_path)
        self.data_len = len(self.data)
        self.indices = np.arange(0, self.data_len)
        self.current_index =  0

    def reset(self):
        random.shuffle(self.indices)
        self.current_index = 0

    def get_next(batch_size):
        end_index = min(current_index + batch_size, self.data_len)
        indices = self.indices[self.current_index:end_index]
        x, y = get_batch_data(indices)
        self.current_index = end_index
        return x, y

batch_size = 64
last_loss = 50
dataset = Dataste(data_path)
for epoch in range(0, max_epoch):
    dataset.reset()
    while dataset.current_index &lt; dataset.data_len:
        opt.zero_grad()

        x, y = dataset.get_next(batch_size)
        y_pred = model(x)
        loss = loss_function(y, y_pred)
        loss.backward()

        batch_size = batch_size + 5 if loss.item() &lt; last_loss else batch_size - 5
        last_loss = loss.item()

        opt.step()
    scheduler.step()
</code></pre>
<p>Or you can just redifine <code>dataloader</code> each iteration, but I don’t know if this operation will affect the efficiency.</p>
RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[1, 512, 512, 3] to have 3 channels, but got 512 channels instead

username: 111215
cooked: <p>Experts, is this a problem to convert [1, 512, 512, 3] into [1,3, 512, 512]? If so, where do I need to modify it?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/4868c6e92a6a03ee706f8586b9c0895320139e63.png" data-download-href="https://discuss.pytorch.org/uploads/default/4868c6e92a6a03ee706f8586b9c0895320139e63" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/4868c6e92a6a03ee706f8586b9c0895320139e63_2_690x273.png" alt="image" data-base62-sha1="akyRpNn2vs8ryjxOE3t9vaW2rXt" width="690" height="273" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/4868c6e92a6a03ee706f8586b9c0895320139e63_2_690x273.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/4868c6e92a6a03ee706f8586b9c0895320139e63_2_1035x409.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/4868c6e92a6a03ee706f8586b9c0895320139e63_2_1380x546.png 2x" data-dominant-color="F2EFF8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2531×1004 347 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Eta_C
cooked: <p>Use (batch_size, channels, height, width) as your input, not (batch_size, height, width, channels).</p>

username: 111215
cooked: <p>Thank you for your confirmation. The problem has been solved</p>
Can't pickle local object 'dataset.__init__.<locals>.identify_file

username: FenryrMKIII
cooked: <p>I am trying to train a yoloV2 model. As part of this, I am using a custom <code>dataset</code> class that defines a <code>identify_file</code> function and acts as a loader.</p>
<p>Then, during training, I need to <code>enumerate(dataset)</code> but Python raises the error  :<br>
<code>Can't pickle local object 'dataset.__init__.&lt;locals&gt;.identify_file</code></p>
<p>I checked, this is not due to lambda function (and therefore this <a href="https://discuss.pytorch.org/t/cant-pickle-local-object-dataloader-init-locals-lambda/31857/7">answer</a>) since  <code>identify_file</code> is defined the following way :</p>
<pre><code class="lang-auto">def identify_file(img_id):
            root = Path(r"some\root\Path")
            return Path.joinpath(root, img_id + '.jpg')
</code></pre>
<p>Any idea what could be going on ? I am on Windows.</p>

username: FenryrMKIII
cooked: <p>Ok, I found the <a href="https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function" rel="nofollow noopener">solution</a>.</p>
<p>The function, in order to be pickable, needs to be defined at the top of a module which was not the case in my program.</p>
Train nn.embedding

username: Rohan_Kumar
cooked: <p>I want to initialize nn.embedding, then train it as well. but when i check the model.embedding.weight before and after the training, it doesn’t change. how do i make it so that the weights change?</p>

username: ptrblck
cooked: <p>Double post from <a href="https://discuss.pytorch.org/t/no-embbeding-grad-data/65940/2">here</a>.<br>
Let’s continue the discussion in the other thread, please.</p>
Softmax normalized weights in ELMO

username: rustagiadi95
cooked: <p>I was going through the ELMo paper, here is the link :- <a href="https://arxiv.org/pdf/1802.05365.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1802.05365.pdf</a><br>
It has softmax normalized weights multiplied as<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/7/370b66a56a50a51d73a385a6311c9f41a7ed119e.png" alt="Untitled" data-base62-sha1="7QWGC46Qk9iHE4xju5om3KU1UOG" width="136" height="120"></p>
<p>here s is the softmax normalized weights and h is the output of BiLSTM jth layer and for kth token.<br>
as this ‘s’ is task-specific it is to be learned, my question is that what is the most pytorchy way to do it, if we want to implement it from scratch?<br>
Thanks in advance.</p>
About using Libtorch directly, compile and build

username: luocheng610
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23674e36951895d6e13a04a7abc3a422a1523900.png" data-download-href="https://discuss.pytorch.org/uploads/default/23674e36951895d6e13a04a7abc3a422a1523900" title="%E6%8D%95%E8%8E%B7"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23674e36951895d6e13a04a7abc3a422a1523900.png" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="53c2Dsg8uvj7e0eg6xGS1K1k8ow" width="690" height="322" data-dominant-color="F7DFDA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%95%E8%8E%B7</span><span class="informations">808×378 24 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Hello everyone, I have some problems in the process of using Libtorch, I hope to get the help of the masters<br>
OS:win10<br>
IDE:VS2017 Pro （v15.9.18 toolset 14.16.27023)<br>
PS:With the help of Github, I changed the toolset to 14.11.25503 in a special way.<br>
I created a new project with VS2017 using normal operations. The additional include directories and library directories have been configured. However, this operation does not seem to compile properly, and many errors have occurred, most of which are syntax errors. As a result, I cannot use it normally. It seems that only cmake can be used to build the project (I tried this method, and it does work)<br>
Is it only possible to build projects with cmake from the command line? Has anyone encountered this situation? Hope that can help me solve this problem</p>
Tensor Type for torch._C.Node

username: flynntax
cooked: <p>I am taking in a trace of a model and would like to check what type the tensor is supposed to be. Is there a way to grab this info from torch._C.Node (ie: get float, double, from <a href="https://pytorch.org/docs/stable/tensors.html" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html</a>)? Or in general any way to grab that info starting from the trace?</p>

username: driazati
cooked: <p>You can grab it in this way:</p>
<pre><code class="lang-python">def some_fn(x):
	return x + x

some_input = torch.randn(2, 2)
my_traced_function = torch.jit.trace(some_fn, [some_input])

for input in my_traced_function.graph.inputs():
	traced_tensor_type = input.type()
	# Prints "Float"
	print(traced_tensor_type.scalarType())

# However, note that the interpreter will still run with differently typed
# tensors
my_traced_function(torch.ones(2, 2, dtype=torch.long))
</code></pre>

username: flynntax
cooked: <p>I see that this is getting it for inputs but what about intermediate nodes in the graph? It seems inputs are of type torch._C.Value which has a decorated type field but torch._C.Node doesn’t.</p>

username: driazati
cooked: <p><code>Node</code> represents the whole operation (some inputs, an operation, and some outputs). The inputs and outputs are represented as <code>Value</code>s, you can get them like so:</p>
<pre><code class="lang-python">def some_fn(x):
	return x + x

some_input = torch.randn(2, 2)
my_traced_function = torch.jit.trace(some_fn, [some_input])
print(my_traced_function.graph)

for node in my_traced_function.graph.nodes():
    for node_input in node.inputs():
        if isinstance(node_input.type(), torch._C.TensorType):
            print(node_input.type().scalarType())
</code></pre>
<p>You can read more about the internal representations <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/docs/OVERVIEW.md#node">here</a>.</p>

username: flynntax
cooked: <p>Thanks for the link, really helpful, somehow didn’t find it in my initial search. And got it, this makes a lot of sense now.</p>
Matplotlib doesn't work in distributed training

username: Alex_Luya
cooked: <p>I am training a model in a 4-GPUs machine with torch.distributed,and I want the ONLY rank_0 process be responsible for plotting,so I wrote code like this:</p>
<pre><code class="lang-auto"> if is_distributed() and distributed.get_rank()!=0:
             print('Only rank_0 will do plotting,this is rank_{}'.format(distributed.get_rank()))
             return# in parallel context,single plot is enough
print('this is rank_0 and  will do plotting')
plotAccuracyAndLoss()
 .....
</code></pre>
<p>So,if the process rank is not 0,it should print out:</p>
<pre><code class="lang-auto">    Only rank_0 will do plotting,this is rank_x
</code></pre>
<p><strong>and I do get 3 printings of this type</strong><br>
if the process rank is 0,it should print out:</p>
<pre><code class="lang-auto">      this is rank_0 and  will do plotting
</code></pre>
<p><strong>and I never got this type of printing,and meanwhile all processes hanging and no exception got thrown out</strong></p>
<p><code>watch -n0.1 nvidia-smi</code> tell that before these code all, all GPU will have memory usage &gt; 10341MB,when hitting these lines,the first GPU’s memory usage drops to 2387MB,others remain,More strangely,if changing code to<br>
<code> if is_distributed() and distributed.get_rank()!=1:</code><br>
which let the second GPU to be responsible for plotting,when comes to these lines,1st,3rd,4th GPU’s memory usage still &gt; 10341MB,but the 2nd GPU’s memory usage drop to 1073MB,training hangs,no exception got thrown out.<br>
With same code in non-distributed training,the plotting works fine,would you please tell me how to make plotting work?</p>

username: Alex_Luya
cooked: <p>After adding:<br>
<code>distributed.barriere()</code><br>
before any rank_x specific operation,everything goes fine,silly me</p>
Training hangs if any specific rank process start an other process to do anything

username: Alex_Luya
cooked: <p>Hello,a similar question has been asked here:<a href="https://discuss.pytorch.org/t/matplotlib-doesnt-work-in-distributed-training/65724,but">https://discuss.pytorch.org/t/matplotlib-doesnt-work-in-distributed-training/65724,but</a> no answer,Question can be briefed blow:</p>
<p>In a 4-GPU machine,all gpus are used for training,there is some some code like this:</p>
<pre><code class="lang-auto"> if is_distributed() and distributed.get_rank()!=0:
             print('Only rank_0 will do plotting,this is rank_{}'.format(distributed.get_rank()))
             return# in parallel context,single plot is enough
print('this is rank_0 and  it will do plotting')
plotAccuracyAndLoss()
</code></pre>
<p>When comes to these code,three:<br>
<code> Only rank_0 will do plotting,this is rank_x</code></p>
<p>got printed out,but</p>
<p><code>print('this is rank_0 and  will do plotting')</code></p>
<p>never got printed out,and all 4 processes hanged and NO exception got thrown out</p>
<p><code>watch -n0.1 nvidia-smi</code>  tell that</p>
<pre><code class="lang-auto">before these code all, all GPU will have memory usage &gt; 10341MB,
when hitting these lines,the first GPU’s memory usage drops to 2387MB,others remain
</code></pre>
<p>previously,I thought that it is the matplotlib which caused this hanging,but right now I found any rank_0-only operation(ploting/checkpointing…) will cause hanging,further more,any rank_x-only operation will cause hanging,so,How to solve this problem?</p>

username: Alex_Luya
cooked: <p>After adding:<br>
<code>distributed.barriere()</code><br>
before any rank_x specific operation,everything goes fine.</p>
Help with tensorboard

username: Faolian
cooked: <p>I’m not sure if this issue comes from pytorch or not, but I figured people might have some advice.</p>
<p>When I run tensorboard i can’t connect to it, getting messages like “[02/Nov/2019 01:04:58] code 400, message Bad request version (‘lèÆ,\x00"êê\x13\x01\x13\x02\x13\x03À+À/À,À0Ì©Ì¨À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x00’)”. The generated log file looks similarly odd "      £K"	ø?I‹5o×A<br>
brain.Event:2‰n£C       £K"	¨<code>I‹5o×A*  y_2x    ñxC       ›Ø-	¨</code>I‹5o×A*<br>
<br>
y_2x   @°1oå       ›Ø-	¨<code>I‹5o×A*  y_2x  €@‡|       ›Ø-	¨</code>I‹5o×A*<br>
<br>
y_2x  À@pwƒ       ›Ø-	¨<code>I‹5o×A*  y_2x   Aö´D©       ›Ø-	¨</code>I‹5o×A*<br>
<br>
y_2x   Aé»»}       ›Ø-	¨`I‹5o×A*"</p>
<p>any ideas whats going on here or how to resolve this - note that this is from a very simple test code (the one in the docs for add scalars)</p>

username: victorhcm
cooked: <p>I had this same issue. Mine was due to my browser forcing HTTPS because of the HTTPS Everywhere extension. If you have the same issue, just disable it for this website and it will be good to go.</p>
WeightedRandomSampler not equally sampling

username: lfolle
cooked: <p>Why does the <code>torch.utils.data.WeightedRandomSampler</code> sample theses values only approximately equal with equal weights?</p>
<pre><code class="lang-auto">list(WeightedRandomSampler([0.3, 0.3, 0.3], 1000, replacement=True)).count(2)
Out[15]: 346
list(WeightedRandomSampler([0.3, 0.3, 0.3], 1000, replacement=True)).count(1)
Out[16]: 339
list(WeightedRandomSampler([0.3, 0.3, 0.3], 1000, replacement=True)).count(0)
Out[17]: 319
</code></pre>
<p>Shouldn’t this yield 333, 333, 333 ?</p>

username: ptrblck
cooked: <p>The passed weights are used in <code>torch.multinomial</code> as seen <a href="https://github.com/pytorch/pytorch/blob/492ca46e71b9f5071a090bc78601c6621f400c9c/torch/utils/data/sampler.py#L160">here</a>, which is a random function, so you cannot expect a perfectly sampled distribution.</p>
<p>The sample count will approximate the expected number, the more samples you are drawing.</p>

username: lfolle
cooked: <p>Thanks for the reply. I understand that due to the randomness the sampling is not perfectly uniform.<br>
But I would not expect these results:</p>
<pre><code class="lang-auto">list(WeightedRandomSampler([0.3, 0.3, 0.3], 100000, replacement=True)).count(0)
Out[5]: 33548
list(WeightedRandomSampler([0.3, 0.3, 0.3], 100000, replacement=True)).count(1)
Out[6]: 33101
list(WeightedRandomSampler([0.3, 0.3, 0.3], 100000, replacement=True)).count(2)
Out[7]: 33176
</code></pre>
<p>Is there a way to make this more uniform?</p>

username: ptrblck
cooked: <p>These numbers would represent a ~0.6% error, wouldn’t they?</p>
<p>Do you need an exact amount of class samples during your sampling?<br>
If so, I think writing a custom sampler with a specified expected class count might be the better approach.</p>

username: lfolle
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="65920">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>These numbers would represent a ~0.6% error, wouldn’t they?</p>
</blockquote>
</aside>
<p>Yes you are right. I guess in case one want to generate a uniform distribution over imbalanced classes with low number of samples the <code>WeightedRandomSampler</code> might not be the best solution (as for <code>num_samples=100</code> we have a deviation of up to 5 %).</p>
<p>Thanks again for your help!</p>
Adding mask to one-hot encoded batch text sequence

username: Dylan_Yung
cooked: <p>I have this code from the GRU with attention PyTorch tutorial</p>
<blockquote>
<p>hidden_cat = torch.cat((hidden[0], hidden[1]), dim=2)</p>
<pre><code>    attn_weights = F.softmax(self.attn(torch.cat((input, hidden_cat), 2)), dim=2)

    attn_applied = torch.bmm(attn_weights.transpose(0,1),encoder_outputs.transpose(0,1)).transpose(0,1)

    attn_output = torch.cat((input, attn_applied), 2)

    attn_output = F.relu(self.attn_combine(attn_output))
</code></pre>
</blockquote>
<p>atten_weights is 1 x batch size x sequence length. So I have to change every index in the third dimension(seq length) was a pad character weights to negative infinity. What’s a good way to create this mask? The input is 1 x batch size x number of words in bag.</p>
Dataiter.next taking indefinite time

username: neonbulb
cooked: <p>I was following <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">this</a> tutorial to train on CIFAR10 dataset. Everything works fine but <code>dataiter.next()</code> is taking indefinite time.</p>
<p>The dataset is already present in the folder. I am using 4.19.88-1-MANJARO with Python3.6 and PyTorch v1.3.1. I don’t have any GPU related libraries installed.</p>
<p>Also, does it loads the whole data into memory when <code>trainloader</code> is initialized or it picks in batches at the <code>dataiter.next</code> step?</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]
)

trainset = torchvision.datasets.CIFAR10(root='../data', train=True,
                                        download=False, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=2,
                                          shuffle=True, num_workers=4)

testset = torchvision.datasets.CIFAR10(root='../data', train=False,
                                       download=False, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

import matplotlib.pyplot as plt
import numpy as np

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()


# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()   &lt;-- taking indefinite time
</code></pre>

username: ptrblck
cooked: <p>Could you try to set <code>num_workers=0</code> in your <code>DataLoader</code> and check, if you get a valid batch?</p>

username: neonbulb
cooked: <p>Yes, it worked. Thank you very much. I am new to PyTorch. Could you please explain me what happened?</p>

username: ptrblck
cooked: <p>This was just a debugging step to narrow down the potential root cause of the issue.<br>
It seems that mutliprocessing causes the hang. I’m not familiar with Manjaro, but could you check the amount of shared memory on your system?</p>
Universal way to loop through parameters of a model

username: pinocchio
cooked: <p>Is the following the universal pytorch way to loop through model params:</p>
<pre><code class="lang-auto">for name, W in net.named_parameters():
    do(name,W)
</code></pre>
<p>are there other ways or thats the universal way to do it? Are all models most commonly just a <code>net</code> object instance?</p>

username: albanD
cooked: <p>Hi,</p>
<p>This will return all the <code>nn.Parameter()</code> stored in you model (and all it’s “child” nn.Module). This is the right way to go through all the parameters (with <code>net.parameters()</code>).<br>
Yes most models are a single <code>nn.Module</code> that will contain many other <code>nn.Module</code>s.</p>
Best way to iterate over weights of model

username: tejus-gupta
cooked: <p>I want to compute L1 regularization. I need to iterate over my network parameters and ignore the bias for computing the L1 loss. Is there a function for checking if a parameter is a weight or bias?</p>
<p>Here is my code. Is this the recommended way? I am afraid that the string comparison in every loop may slow down training.</p>
<pre><code class="lang-auto">l1_reg = None
for name, W in net.named_parameters():
    if name[-6:] != 'weight':
        continue
    if l1_reg is None:
        l1_reg = W.norm(1)
    else:
        l1_reg = l1_reg + W.norm(1)
</code></pre>
<p>Thanks in advance :D.</p>

username: justusschock
cooked: <p>I think you almost did it.<br>
A string compare should need almost no computational cost. If you initialize the l1_reg with zero you do not need the second if-closure which also saves a minimal amount of computation time. AFAIK there is no more efficient way to do so.</p>
<pre><code class="lang-auto">l1_reg = 0
for name, W in net.named_parameters():
    if 'weight' in name:
        l1_reg = l1_reg + W.norm(1)
</code></pre>

username: Brando_Miranda
cooked: <p>the solution you accepted does NOT match the title of the question. Please modify the title or make the body of the question match the title. Having mismatching body and title makes search online harder. Don’t do that.</p>
SWA wasn't applied to param {}; skipping it".format(p))

username: Murtaza_Basu
cooked: <p>Hello, I tried to follow the Stochastic Weight Average tutorial and I tried to implement in my code but I get the following error,</p>
<pre><code class="lang-auto">/home/murtaza/.local/lib/python2.7/site-packages/torchcontrib/optim/swa.py:191: UserWarning: SWA wasn't applied to param Parameter containing:
tensor([[-0.0556,  0.1067,  0.0519, -0.1137,  0.0632, -0.0402,  0.0576, -0.0704,
         -0.0888, -0.1129, -0.0102,  0.0503,  0.0469, -0.0822, -0.1028, -0.0354,
         -0.0007,  0.0863, -0.0221, -0.1036,  0.0431,  0.0164,  0.0004, -0.1106,
          0.0466, -0.0283, -0.0954, -0.1001, -0.0113,  0.0089,  0.0471, -0.0335,
          0.0501,  0.0773,  0.1195, -0.0987,  0.0455, -0.0468, -0.0520, -0.1011,
         -0.0373, -0.0642,  0.0105,  0.0455,  0.0452, -0.0569, -0.0551, -0.1137,
         -0.0057,  0.0203,  0.0088,  0.0077,  0.0917, -0.1203,  0.0266,  0.0904,
         -0.0180,  0.0097, -0.0717, -0.0547, -0.0954,  0.1197,  0.0836,  0.0938]],
       requires_grad=True); skipping it
  "SWA wasn't applied to param {}; skipping it".format(p))
</code></pre>
<p>i have no clue where I am going wrong here is the code snippet where I am implementing SWA</p>
<pre><code class="lang-auto">                for _ in range(self.K_epochs):

                    # Evaluating old actions and values :
                    logprobs, values, dist_entropy = self.policy.evaluate(old_states, old_actions)
                    # Finding the ratio (pi_theta / pi_theta__old):
                    advantages = self.calculate_advantages(reward_batch, values.detach())
                    ratios = torch.exp(logprobs - old_logprobs)

                    # Finding Surrogate Loss:
                    surr1 = ratios * advantages
                    surr2 = torch.clamp(ratios, 1-self.eps_clip, 1+self.eps_clip) * advantages
                    actor_loss = -torch.min(surr1, surr2)
                    critic_loss = 0.5*self.MseLoss(values, returns)
                    loss = actor_loss + critic_loss + 0.5*self.MseLoss(values, returns) - 0.01*dist_entropy

                    # take gradient step
                    self.optimizer.zero_grad()
                    loss.mean().backward()
                   self.SWAoptim.step()
                self.SWAoptim.swap_swa_sgd()            
                # Copy new weights into old policy:
                self.policy_old.load_state_dict(self.policy.state_dict())
                self.loSS.append(loss.mean().item())

</code></pre>
Does Python3.8 support torch and torch vision installation?

username: crissallan
cooked: <p>Hi, I tried to install torch and torchvision in Python3.8. But I met an Error ```<br>
ModuleNotFoundError: No module named ‘tools.nnwrap’</p>
<pre><code class="lang-auto">
How should I solve this problem?

Many thanks!</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>You should be able to compile from source for python 3.8<br>
How did you tried to install? Did you get all the submodules of the git repo recursively properly?</p>
How to know the memory usage of a module in Pytorch?

username: PiggySensen
cooked: <p>It’s clear that in model training, the memory usage of module parameters like BN layer or Conv layer is not a big problem in GCN or CNN. But when the input data is of great size such as 5M(size of one sample), the memory usage is hard to bear because of the intermediate result to be saved to calculate the gradient for BP. I want to know the appropriately how much memory will be added after the input go through a function such as view(), contiguous(), einsum(), stack(), mul() and many functions in Pytorch. I go to find this in official doc only to find how to use this function. I truly want to know whether there are websites or docs that can help, or what I can do is to indicate that from the doc? Thanks a lot for helping!</p>

username: PiggySensen
cooked: <p>For example,</p>
<p>fv = fv.permute(0, 4, 3, 1, 2).contiguous().view(N, M * V_node * C, T)<br>
fv = self.data_bn_v(fv)</p>
<p>if the size of fv is 100M, after these two function, how much memory will be used?</p>
GPU will crash when run multi threads by libtorch

username: Ryan95
cooked: <p>Hi, i have same question in libtorch, i trace bert model to libtorch and run it with multi thread on GPU. after infering some images, program will freeze and not print anythings in console, but if i run on CPU , it’s fine, and if i use single thread to run model on GPU, it’s also no problem.<br>
this question only appear in Tesla P40, and on Tesla K80 everything is ok.</p>
<p>my environments is CUDA9.0 + cuDNN7.5 + GCC 4.9.4 + libtorch 1.1.0.</p>
<p>it’s screenshot when program has no progress<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/f663bc23615ddc7ec4945c91ccfd787341ec3f84.png" alt="nvidia" data-base62-sha1="z9F8BxWtC2eKOrke88GkZU53ADa" width="572" height="296"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/23f2e96488242fe8268b9829ec8a7b8a37cd59ce.png" alt="run" data-base62-sha1="58198ss2YqFJbXFYITkGTzoY8vI" width="492" height="433"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c9188f4c6c3bb7f52c78b6fe05dcfaebcb095a7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/c9188f4c6c3bb7f52c78b6fe05dcfaebcb095a7f" title="CPU"><img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c9188f4c6c3bb7f52c78b6fe05dcfaebcb095a7f.png" alt="CPU" data-base62-sha1="sGYAChfTfX8lTGxZ6SqxP20juUf" width="690" height="213" data-dominant-color="304A48"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">CPU</span><span class="informations">714×221 39 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: AlbertVan
cooked: <p>Hi, I hava same problem as you, do you resolve it now ?</p>
Getting some error while using pytorch lib

username: Surya92
cooked: <p>I got the below error while using pytorch. How to rectify this error? Is something wrong with the installation?</p>
<p>print(torch.<em>version</em>)<br>
Traceback (most recent call last):</p>
<p>File “”, line 1, in <br>
print(torch.<em>version</em>)</p>
<p>AttributeError: module ‘torch’ has no attribute ‘<em>version</em>’</p>
<p>a=torch.tensor([10,10,10])</p>
<p>a<br>
Out[8]: tensor([10, 10, 10])</p>
<p>a.mean()<br>
Traceback (most recent call last):</p>
<p>File “”, line 1, in <br>
a.mean()</p>
<p>RuntimeError: Can only calculate the mean of floating types. Got Long instead.</p>

username: Eta_C
cooked: <pre><code class="lang-python">import torch
print(torch.__version__)
a = torch.tensor([10, 10, 10]) # type: Long
print(a.float().mean())
b = torch.tensor([10., 10., 10.])  # type: Float
print(a.mean())
</code></pre>

username: Eta_C
cooked: <p>Use <code>__version__</code> insteal of <code>_version_</code>.<br>
<strong>Double underline</strong>.</p>

username: Surya92
cooked: <p>Thanks for your response. Is it possible to calculate mean for Long data type without converting to Float?</p>
Precision, recall and mIOU for binary segmentation

username: Jack_sp
cooked: <p>I am doing a binary segmentation of brain tumors from CT scan images. Following is the code I am using for precision and recall</p>
<pre><code class="lang-auto">def get_precision(PR,GT,threshold=0.5):
    PR = PR &gt; threshold
    GT = GT == torch.max(GT)
    TP = ((PR==1)&amp;(GT==1))
    FP = ((PR==1)&amp;(GT==0))
    Precision = float(torch.sum(TP))/(float(torch.sum(TP)+torch.sum(FP)) + 1e-6)
    return Precision
</code></pre>
<pre><code class="lang-auto">def get_recall(PR,GT,threshold=0.5):
    PR = PR &gt; threshold
    GT = GT == torch.max(GT)
    TP = ((PR==1)&amp;(GT==1))
    FN = ((PR==0)&amp;(GT==0))
    Recall = float(torch.sum(TP))/(float(torch.sum(TP)+torch.sum(FN)) + 1e-6)
    return Recall
</code></pre>
<p>PR is the predicted probabilities of the model from sigmoid layer and GT is the ground truth images divided by 255.Is this the right implementation of the metrics for binary segmentation as I am getting both values in high 90’s. Also can I follow same approach for calculating mIOU.</p>
When using CrossEntropyLoss for segmentation,How to encode labels into single channel target image if value <= C is a must-be?

username: Alex_Luya
cooked: <p>CrossEntropyLoss doc say the target value must be:<br>
<code> 0≤targets[i]≤C−1</code><br>
And I have 4 type of objects which must be segmented,in this case C==4,and if I want to use single channel target image where all values must be &lt;=4,if 0 for background,1 for the first type of objects…and How to encode the FOURTH type of objects?Or should outputs must be N+1(treating background as a class),in my case,the output channels will be 5(4+background)?</p>

username: ptrblck
cooked: <p>You would treat the background as a separate class, thus your target mask will have values in the range <code>[0, 4]</code>.</p>
Pytorch Training code flow from github

username: abhilash_pinky
cooked: <p>Hi how to understand any code obtained from github like yolov2, maskrcnn training and inference pipeline by flow diagram ? any tool available to view the flow</p>

username: ptrblck
cooked: <p>Are you looking for a tool to visualize the computation graph (model architecture)?<br>
If so, have a look at <a href="https://discuss.pytorch.org/t/graph-visualization/1558">this post</a>, where some tools were suggested.</p>
<p>If not, could you explain what you mean by flow a bit more, please? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Eta_C
cooked: <p><a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html" rel="nofollow noopener">VISUALIZING MODELS, DATA, AND TRAINING WITH TENSORBOARD</a></p>

username: abhilash_pinky
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> i am able to visualize the made by the link <a class="mention" href="/u/eta_c">@Eta_C</a>, i wanted to understand the training pipeline code which is either written by some one or which is available in github. Eg dataloading , preprocessing and training modules which functions are getting called and all</p>
Polygamma(n,x) is not implemented for n>=2 at /pytorch/aten/src/THC/generic/THCTensorMathPointwise.cu:364

username: KylinA1
cooked: <p>As document state here: <a href="https://pytorch.org/docs/master/torch.html" rel="nofollow noopener">https://pytorch.org/docs/master/torch.html</a><br>
<code>torch.polygamma(n, x)</code> only works when n = 0, 1<br>
in my forward neural network,  <code>torch.polygamma(1, x)</code> works normal.<br>
But, come to <code>loss.backward()</code><br>
it raise the error said <code>polygamma(n,x)</code> is not implemented for n&gt;=2.<br>
I do understand the derivative of<code> polygamma(1, x)</code> is <code>polygamma (2, x)</code>. how should I get this function derivative.</p>
[Hessian Vector Product] RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior

username: fang150
cooked: <p>I am trying to implement hessian vector product but I am facing this problem.</p>
<p>Here’s the code:</p>
<pre><code>    for p in model.parameters():
        grads_fun.append(p.grad)
        grads_data.append(p.grad.data)

    for gg in grads_fun:
        gg.requires_grad=True

    zs = [0.0*p.data for p in model.parameters()]
    rs = [g for g in grads_data]
    ds = [-1.0*g for g in grads_data]

    Hd = torch.autograd.grad(grads_fun, list(model.parameters()), grad_outputs=ds, only_inputs=True, retain_graph=True,allow_unused=False)
</code></pre>
<p>“”""""</p>
<p>But I got the errors:</p>
<p>“”"""<br>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.<br>
“”"""</p>
<p>And if I set allow_unused=True, Hd will be all None.</p>
<p>Could any one point out what’s the problem with the codes and how do I get around this problem.</p>
<p>Thanks</p>

username: tom
cooked: <p>You don’t show how you got the gradients. Did you use <code>build_graph=True</code>?<br>
Also, for linear functions the gradient will not depend on the parameters, this will also look similar to your error message.</p>
<p>Best regards</p>
<p>Thomas</p>

username: fang150
cooked: <p>Thanks Tomas. I am using loss.backward() to accumulate gradients, I also try to let retain_graph =True.</p>
<p>As to the dependency on the parameters, I tried to not using ds but grads_data to compute hessian vector product. I still get the same error message.</p>
<p>“Hd = torch.autograd.grad(grads_fun, params, grad_outputs=grads_data, only_inputs=True, retain_graph=True,allow_unused=False)<br>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.”</p>
<p>Cheers</p>

username: tom
cooked: <aside class="quote no-group" data-username="fang150" data-post="3" data-topic="50490">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/977dab/48.png" class="avatar"> fang150:</div>
<blockquote>
<p>Thanks Tomas. I am using loss.backward() to accumulate gradients, I also try to let retain_graph =True.</p>
</blockquote>
</aside>
<p>Yeah, but if you want to take derivatives of quantities involving the gradients, you need <a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.backward" rel="noopener nofollow ugc"><code>create_graph=True</code></a>, too.</p>
<p>Best regards</p>
<p>Thomas</p>

username: fang150
cooked: <p>This solves the problem.</p>
<p>Thank you so much Thomas,</p>
<p>Best Regards,</p>
<p>chih-hao</p>

username: Dhawgupta
cooked: <p>Can you explain the reason as to why for linear functions the gradients will no longer depend on parameters, and also cant they be just set to zero if they do not in that case ?</p>
Get the matmul operations in a net?

username: solvingPuzzles
cooked: <p>There’s code such as [1] which walks through a net’s nn.Modules, checks their type, and calculates their FLOPs.</p>
<p>However, some functions in pytorch are defined only in forward() and not in <strong>init</strong>(). For example, using the methodology of [1], we would get the FLOPs of torch.nn.Linear, but we would miss any instances of torch.matmul. Does anyone have an idea for how to write a function that takes a net and figures out the FLOPs used by torch.matmul operations?</p>
<p>[1] <a href="https://github.com/Lyken17/pytorch-OpCounter" rel="nofollow noopener">https://github.com/Lyken17/pytorch-OpCounter</a></p>

username: solvingPuzzles
cooked: <p>…I ended up making nn.Module wrappers for anonymous functions such as MatMul for my net. Not the most elegant solution, but it worked for my use-case.</p>

username: Eta_C
cooked: <p>I had seen a code aiming to convert PyTorch to Caffe. It looks like</p>
<pre><code class="lang-python">raw_mul = torch.Tensor.__mul__
def mul(input, *args):
    x = raw_mul(input, *args)
    do something
torch.Tensor.__mul__ = mul
</code></pre>
<p>Not the most elegant solution.</p>

username: solvingPuzzles
cooked: <p>That makes sense too.</p>
Mapping one tensor of values and another tensor of indices to a third tensor by index

username: tobogranyte
cooked: <p>I have one_tensor (3,1) of values, say:</p>
<p>one_tensor = torch.tensor([[24],[36],[12]])</p>
<p>and index_tensor (3,1)  of indices, say:</p>
<p>index_tensor = torch.tensor([[0],[3],[2]])</p>
<p>I’d like to map the values of one_tensor to a third_tensor (3,4) of zeros:</p>
<p>third_tensor = torch.zeros(3,4)</p>
<p>using the indices of index_tensor so that third tensor ends up like this:</p>
<p>third_tensor = tensor([[24, 0, 0, 0], [0, 0, 0, 36], [0, 0, 12, 0]])</p>
<p>Is there a fast, efficient way to do this without manually iterating over each row?</p>

username: ptrblck
cooked: <p><code>.scatter_</code> should work:</p>
<pre><code class="lang-python">third_tensor.scatter_(1, index_tensor, one_tensor)
print(third_tensor)
&gt; tensor([[24.,  0.,  0.,  0.],
        [ 0.,  0.,  0., 36.],
        [ 0.,  0., 12.,  0.]])
</code></pre>
Autograd of scatter

username: as754770178
cooked: <p>I recently want to change <code>Scatter</code>. The original code as below, <code>Scatter</code> is the subclass of <code>Function</code>. But when I exec my code, I find that the <code>backward</code> function in <code>Scatter</code> not exec, whether I use <code>DataParallel</code> or <code>DistributedDataParallel</code>.</p>
<pre><code class="lang-auto">class Scatter(Function):

    @staticmethod
    def forward(ctx, target_gpus, chunk_sizes, dim, input):
        target_gpus = list(map(lambda x: _get_device_index(x, True), target_gpus))
        ctx.dim = dim
        ctx.input_device = input.get_device() if input.is_cuda else -1
        streams = None
        if ctx.input_device == -1:
            # Perform CPU to GPU copies in a background stream
            streams = [_get_stream(device) for device in target_gpus]
        outputs = comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams)
        # Synchronize with the copy stream
        if streams is not None:
            for i, output in enumerate(outputs):
                with torch.cuda.device(target_gpus[i]):
                    main_stream = torch.cuda.current_stream()
                    main_stream.wait_stream(streams[i])
                    output.record_stream(main_stream)
        return outputs

    @staticmethod
    def backward(ctx, *grad_output):
        return None, None, None, Gather.apply(ctx.input_device, ctx.dim, *grad_output)
</code></pre>
<p>Questions：</p>
<ol>
<li>The <code>backward</code> function in <code>Scatter</code>, when to trigger execution?</li>
<li>Pytorch how to compute gradient on multi gpus(use DataParallel) and multi node(DistributedDataParallel)?</li>
</ol>

username: as754770178
cooked: <p>My code first exec model’s forward, then use model’s output calculate loss on GPU0. Such as</p>
<pre><code class="lang-auto">import torch.nn as nn
from torchvision.models import resnet50

def classify_loss_fn(output, label):
  loss = torch.nn.CrossEntropyLoss()(output, label.long())
  return loss

model = resnet50()
model = nn.DataParallel(model, device_ids=[1, 2, 3, 4]) # or
model = nn.parallel.DistributedDataParallel(model, device_ids=[1, 2, 3, 4],
                                            find_unused_parameters=True)

for (img, label) in dataloader:
  output = model(img)
  loss = classify_loss_fn(output, label)
  loss.backward()
</code></pre>
Question in python 60min tutorial - Training a Classifier

username: Yolkandwhite
cooked: <p>Hi I’m an newbie in pytorch<br>
I’m studying torch on this tutoral<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py</a></p>
<h1></h1>
<h1></h1>
<h1></h1>
<hr>
<p>import torch<br>
import torchvision<br>
import torchvision.transforms as transforms</p>
<p>transform = transforms.Compose(<br>
[transforms.ToTensor(),<br>
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])</p>
<p>trainset = torchvision.datasets.CIFAR10(root=’./data’, train=True,<br>
download=True, transform=transform)<br>
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,<br>
shuffle=True, num_workers=2)</p>
<p>testset = torchvision.datasets.CIFAR10(root=’./data’, train=False,<br>
download=True, transform=transform)<br>
testloader = torch.utils.data.DataLoader(testset, batch_size=4,<br>
shuffle=False, num_workers=2)</p>
<p>classes = (‘plane’, ‘car’, ‘bird’, ‘cat’,<br>
‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’)</p>
<hr>
<h1></h1>
<h1></h1>
<hr>
<p>import matplotlib.pyplot as plt<br>
import numpy as np</p>
<h1>functions to show an image</h1>
<p>def imshow(img):<br>
img = img * 0.5 + 0.5     # unnormalize  std = 0.5, mean = 0.5<br>
npimg = img.numpy()<br>
plt.imshow(np.transpose(npimg, (1, 2, 0)))<br>
plt.show()</p>
<h1>get some random training images</h1>
<p>dataiter = iter(trainloader)<br>
images, labels = dataiter.next()</p>
<h1>show images</h1>
<p>imshow(torchvision.utils.make_grid(images))</p>
<h1>print labels</h1>
<p>print(’ ‘.join(’%5s’ % classes[labels[j]] for j in range(4)))</p>
<hr>
<h1></h1>
<p>and here’s my question.<br>
I don’t understand this line.</p>
<h1></h1>
<hr>
<p>images, labels = dataiter.next()</p>
<hr>
<h1></h1>
<p>What I thought was dataiter.next() goes to images and labels each which is an same data,<br>
but when I printed out the images and labels right after that line just like below.</p>
<h1></h1>
<hr>
<p>print(images, labels)</p>
<hr>
<h1></h1>
<p>The output wasn’t same.<br>
images got img file in array and labels got label numbers<br>
and I don’t know why.<br>
<strong>How did the image get the image data in arrays and labels get label numbers respectively??</strong></p>
<p>Thank you for your answers<br>
regards.</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> calls into the passed <code>Dataset</code>'s <code>__getitem__</code> method to load and process the next sample.<br>
Have a look at <a href="https://github.com/pytorch/vision/blob/07cbb46aba8569f0fac95667d57421391e6d36e9/torchvision/datasets/cifar.py#L110-L130">these lines of code</a> to see, how the <code>CIFAR10</code> datasets loads each sample.</p>

username: Yolkandwhite
cooked: <p>Thank you.<br>
I got it!</p>
How to mask tensor with boolean numpy array

username: RTC
cooked: <p>I have a  <code>84x84</code>  pytorch tensor named  <code>target</code> . I need to mask it with an  <code>84x84</code>  boolean numpy array which consists of  <code>True</code>  and  <code>False</code> . This mask array is called <code>mask</code>.</p>
<p>When I do  <code>target = target[mask]</code> , I get the error  <code>TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8.</code></p>
<p>Surprisingly, I get this error only when running on a GPU. When running on a CPU, everything works fine. How can I fix this?</p>

username: ptrblck
cooked: <p>Could you try to index with a tensor via:</p>
<pre><code class="lang-python">target = target[torch.from_numpy(mask)]
</code></pre>