akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
3-d tensor gather with 2-d indices

username: dl_noob
cooked: <p>Hi,<br>
I have a tensor <strong>y</strong> which has the size 1024 x 10 x 10, basically I have 1024 matrices of 10x10<br>
I also have 2 indices tensors <strong>a</strong> and <strong>b</strong> size 1024 x 15 and 1024 x 15, to access elements of each 1024 10x10 matrices, I guess this is some form a gather function<br>
Thus, my final output <strong>z</strong> should be 1024 x 15<br>
Currently I do this with loop enumerations of the first index .<br>
Could you suggest me a more vectorized version or is there a better way? Thank you.<br>
Here is the snippet:</p>
<pre><code class="lang-auto">z =  [y[i][a[i],b[i]].sum() for i,_ in enumerate(y)]
z = torch.tensor(z, dtype=torch.float).to(y.device)
</code></pre>

username: ptrblck
cooked: <p>The creation of <code>z</code> is not working using the provided shapes and you should probably use <code>torch.stack</code> instead.</p>
<p>Anyway, this code should yield the same result:</p>
<pre><code class="lang-python">N = 100
y = torch.randn(N, 10, 10)
a = torch.randint(0, 10, (N, 15))
b = torch.randint(0, 10, (N, 15))

z = y[torch.arange(N).unsqueeze(1), a, b]
</code></pre>

username: dl_noob
cooked: <p>You are right, I forgot that there was a summation as well, edited it. Will try that thank you.</p>
How to detect faces in real time or in a single image

username: hunar
cooked: <p>how to detect faces in real time (video) or in a single image with pytorch (neural network)<br>
thanks<br>
regards …</p>

username: ptrblck
cooked: <p>You could just search for an open source implementation for face detecion (I just checked and there seem to be multiple implementations) and use one of these as a starter code.</p>
<p>If you want to implement it by yourself, you would need a dataset, create an appropriate model architecture, and train the model as described in our tutorials.</p>

username: hunar
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> how to prepare the dataset <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79c1727029b7d8cd99ab3c11636cec74c48d8cfc.png" data-download-href="https://discuss.pytorch.org/uploads/default/79c1727029b7d8cd99ab3c11636cec74c48d8cfc" title="d"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79c1727029b7d8cd99ab3c11636cec74c48d8cfc.png" alt="d" data-base62-sha1="hn6eoOtoaU99h2x2XfTddLDfYC0" width="690" height="165" data-dominant-color="0C130D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">d</span><span class="informations">952×229 6.03 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>this folder contains multi folders and each of them include 10 images of one person (10 image for one identify)<br>
but the problem is i dont know how separate them to train , and test ? and how to provide one label for 10 images ? thanks for your advice</p>

username: ptrblck
cooked: <p>If each folder contains images from a separate class, you could use <code>torchvision.datasets.ImageFolder</code>.<br>
To split the dataset to a training, validation, and test set you could use <code>Subset</code> or <code>SubsetRandomSampler</code>.</p>

username: hunar
cooked: <p>and what if want to convert the dataset to csv format ?</p>

username: ptrblck
cooked: <p>Do you mean how to load data from a csv file?<br>
If your data is directly stored in the csv, you could write a custom <code>Dataset</code>, open the file in the <code>__init__</code> method, and load each sample in <code>__getitem__</code>.</p>
<p>This approach would also work, if the csv stores file paths to the samples.</p>
<p>Alternatively, you could also preload the data and use a <code>TensorDataset</code>. It depends on your use case, so a bit more information would be needed.</p>
Using Pytorch 0.5.0 with Torchvision 0.4.2

username: Gaurav_Jain
cooked: <p>have been trying to get PyTorch 0.5 working with Torchvision. I know its pretty old but that’s what I am supposed to work with due to using GPGPUSim. The torchvision version I have is 0.4.2.</p>
<p>When I try to do import torchvision, I get the following error:</p>
<p>File “/usr/local/lib/python3.5/site-packages/torch/jit/frontend.py”, line 422, in build_Subscript<br>
raise NotSupportedError(base.range(), “slicing multiple dimensions at the same time isn’t supported yet”)<br>
torch.jit.frontend.NotSupportedError: slicing multiple dimensions at the same time isn’t supported yet</p>
<p>Is there a way to get around the version mismatch issue?</p>

username: ptrblck
cooked: <p>If you really need these particular versions, you could</p>
<ul>
<li>try to remove/comment all related (jit) errors</li>
<li>use a compatible <code>torchvision</code> version and add the missing pieces you need</li>
</ul>
<p>I’m not sure, how much work the first approach would be, so the latter one might be easier e.g. if you just need one specific model or functionality.</p>

username: Daniel_Kurniadi
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, how may I know the compatible <code>torchvision</code> version for a corresponding torch version? Thank you in advance.</p>

username: ptrblck
cooked: <p>You could have a look at the <a href="https://github.com/pytorch/vision/releases">vision release notes</a>, where the PyTorch dependencies are mentioned.</p>

username: Gaurav_Jain
cooked: <p>Thanks a lot for the response.<br>
The issue in removing the related (jit) errors is that every change I make is leading to it breaking somewhere else.</p>
<p>Anyways, one of my colleagues says that torchvision 0.4.0 should work fine. Trying that.</p>
Whether to use detach or not while cloning intermediate tensors during training?

username: Preyas
cooked: <p>The task at hand, is to classify a given “set” of images as positive or negative (binary classification using “sets” of images as input). For this purpose, I am training a  VGG16 architecture to generate a representation for each image, combining representation from multiple images, and using cross-entropy loss to predict the set label as positive or negative.</p>
<p>For the training data, I only have positive sets of images available. Therefore, I decided to create negative sets by combining randomly sampled images from different positive sets to create a negative set. This procedure of creating negative sets from randomly sampled images from positive sets is done in each mini-batch (online generation of negative sets while training).</p>
<p>Procedure for generating negative sets:</p>
<ol>
<li>positive_items= vgg16(positive_images)</li>
<li>positive_items_copy= positive_items.clone()</li>
<li>negative_items = items_copy[torch.randperm(positive_items_copy.shape[0])]</li>
</ol>
<p>Finally, I just use the positive and negative items to calculate the loss (binary cross-entropy in this case), basically predicting which set of images is positive and negative.</p>
<p><em>My question is:</em><br>
<strong>Am I correct to simply clone the representation of positive images (in step 2 in the code above), to finally randomly shuffle them and get a negative set of representations, without using .detach()</strong>. I expect the loss function to back-propagate gradients from both positive and negative examples to the weights of VGG16, so I think this is the right way to do it.</p>
<p>Another way (which I think is wrong), would be to first clone the positive items while using detach, and then set requires_grad=True (like mentioned below), but I think doing this will not allow the errors from the negative examples to update the weights of the VGG16 network.</p>
<ol>
<li>positive_items= vgg16(positive_images)</li>
<li>positive_items_copy= positive_items.detach().clone()<br>
3.positive_items_copy.requires_grad=True<br>
4.negative_items = items_copy[torch.randperm(positive_items_copy.shape[0])]</li>
</ol>
<p>I know this is a much longer than usual question, but any help would be really appreciated. Thanks in advance!</p>
Translate pytorch code to libtorch is ~6x slower

username: Esmaeil_Farhang
cooked: <p>Hi, I translate code from PyTorch to libtorch, in the main code used some Numpy and Librosa functions and I convert these methods to cpp libtorch, really in my new code all the thing for calculating is Tensor, but the performance of new code in c++ is slower than main PyTorch code. (two codes test in ubuntu 16.04 and CPU),<br>
why did this happen? need i’m used some blace or Lapack or Ttb for optimized my code.<br>
convert Numpy to tensor can be the reason for slow performance?</p>
RuntimeError about loss.backward()

username: XiaoAHeng
cooked: <pre><code class="lang-auto"> File "/data/ch/CAMUS/skip_clstm/train_motion_warp.py", line 116, in train_net
    loss.backward()
  File "/data/ch/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/data/ch/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/data/ch/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/data/ch/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/utils/checkpoint.py", line 99, in backward
    torch.autograd.backward(outputs, args)
  File "/data/ch/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: vector::_M_range_check: __n (which is 1923827846) &gt;= this-&gt;size() (which is 5)
</code></pre>
<p>Anyone can help me ? Thanks in acvance !</p>
Integrate original image shape information in CNN for image classification

username: Mdhvince
cooked: <p>Hi everyone,</p>
<p>I have a question about one of my project and I hope you can help me to solve it <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> . I have to classify images across 15 classes and I get the image from a detector, then I use ResNet18 to classify them.</p>
<p>But I know that the original shape of the detected object (the cropped object from the detector) give me a lot of information about how should this cropped image should be classified.</p>
<p>Since I have to resize all image to (224x224) for ResNet18, I’m losing the shape information. So My question is, is it possible to add this information in the CNN ? How can I do it in Pytorch ? Is it a common practice in the Computer Vision Area ?</p>
<p>Thank you all</p>

username: ptrblck
cooked: <p>Since resnet18 uses an adaptive pooling layer before the flatten operation, the input shapes are flexible.<br>
However, this might of course degrade your performance and the architecture might not make sense, if your inputs are too small.</p>
<p>Your use case reminds me a bit of <a href="https://eng.uber.com/coordconv/">CoordConv</a> by Uber, which provided a coordinate system to their inputs. Would something like this make sense for your use case?</p>

username: ebarsoum
cooked: <p>What you want to do with this information?</p>

username: Mdhvince
cooked: <p>This is an very interesting work, thanks for sharing <a class="mention" href="/u/ptrblck">@ptrblck</a> . So if I understand, one of the solution could be to add an additional channel to my image (and one more to the cnn input).<br>
But since the coordinates I want to add is 1x4 and images 224x224, do you think I have to repeat these coordinates to reach a shape of 224x224 and stack them to the current image ?</p>

username: Mdhvince
cooked: <p>Hi <a class="mention" href="/u/ebarsoum">@ebarsoum</a>,  with these additional data, I want to give the cnn more ability to classify images, because they are very difficult to dissociate as a human, but juste the original shape can sometimes help me classify them.</p>
<p>i.e : I know that one of my category always have rectangular shape.</p>
<p>But I cannot base decision only on the shape, that why I’m using CNN.</p>
How to specify the install path when build libtorch?no use cmake-gui

username: LinGeLin
cooked: <p>As the title。<br>
thanks</p>

username: LinGeLin
cooked: <p>It makes my head big</p>

username: ptrblck
cooked: <p>You can move to a specific folder to and build it there as described <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst">here</a>.<br>
Once you’ve built libtorch, you can specify the path with</p>
<pre><code class="lang-auto">cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..
</code></pre>
<p>when building your project, as described <a href="https://pytorch.org/cppdocs/installing.html">here</a>.</p>

username: LinGeLin
cooked: <p>You mean like this？</p>
<pre><code class="lang-auto">python setup.py build

cd build

cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..

cd ..

python setup.py install
</code></pre>
<p>This is very troublesome，and when I exec python setup.py install，It also compiles some files</p>
Compute gradient of bitwise OR

username: oezguensi
cooked: <p>I have to calculate the Intersection over Union (IoU) for two segmentations. For that, at some point I have to calculate the bitwise OR of two tensors. I am doing this with <code>(a + b) &gt; 0</code>.<br>
Unfortunately this isn’t differentiable. Does someone know how I can calculate the bitwise OR and still be able to backprop?</p>

username: InnovArul
cooked: <p>What if you consider them as float and use multiplications, addition  to calculate the IoU?  That’s how its done in most codes if I remember correctly</p>

username: oezguensi
cooked: <p>I can’t think of a function that only uses non-logical operators<br>
EDIT:<br>
Actually, I just used torch.max(a, b)</p>

username: InnovArul
cooked: <p>What I meant is,  to calculate the IoU,  we can use something like below.</p>
<p>Consider A,  B are two 2D tensors made up of 0 and 1s.</p>
<p>Intersection = (A * B).sum()<br>
Union = (A+B - A*B).sum()</p>
<p>IoU = intersection / Union</p>
Confused about target_label when using nn.CrossEntropyLoss

username: Audrey
cooked: <p>Hello! I am a little confused when trying to calculate cross entropy. So here is my situation:<br>
For example, the data has batch_size which is 2, and 3 classes to predict.</p>
<pre><code class="lang-auto">#               c1   c2   c3
pred_label = [[0.2, 0.5, 0.3]
              [0.3, 0.6, 0.1]]  # shape:[2,3]
</code></pre>
<p>The target label for the 1st sample in the batch is <code>c1</code>, and for the 2nd sample is <code>c3</code>.<br>
The question is: <strong>What’s the correct form for target label tot use?</strong>  In TensorFlow, it requires to use one-hot encoding to represent target value like this:</p>
<pre><code class="lang-auto">target_label = [[1, 0, 0]
                [0, 0, 1]
tf.nn.softmax_cross_entropy_with_logits_v2(pred_label, target_label)
</code></pre>
<p>I tried one-hot encoding in PyTorch, but got error.<br>
It seems that PyTorch does not need one-hot encoding. Maybe I can use the original target label value directly?</p>
<pre><code class="lang-auto">target_label = [0,2] # class1 and class3 
torch.nn.CrossEntropyLoss(pred_label, target_label)
</code></pre>
<p>I am not if I understand right. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: beaupreda
cooked: <p>Yes, you can use the class labels directly in PyTorch.</p>

username: Audrey
cooked: <p>Yes, it’s very convenient! Thanks.</p>
Is there anything wrong with setting default tensor type to cuda?

username: Simon_Mackenzie
cooked: <p>The docs say you should pass the device parameter around and create your tensors with parameter device=device or use .to(device) to move them to the gpu; and to apply .cuda() to the model.</p>
<p>However typically I want to use CPU or GPU for everything. Therefore if I want GPU it seems easier to just at the start do:</p>
<blockquote>
<p>torch.set_default_tensor_type(torch.cuda.FloatTensor)</p>
</blockquote>
<p>This avoids the need to pass around a device parameter and loads of .to(device) calls where one can easily be forgotten by mistake. Is there anything wrong with this?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think there is anything wrong with this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Simon_Mackenzie
cooked: <p>I have found a problem. It fails with multiprocessing. For example a dataloader with workers=0 works fine. If I set workers&gt;0 then it fails with cuda initialization error. It fails even if the dataloader is already created. As soon as you create an iterator it fails. For example if you have a training loop that does “for x in dl” then it fails.</p>

username: Red-Eyed
cooked: <p>Hello Simon,<br>
My workaround is:</p>
<pre><code class="lang-auto">@contextmanager
def set_default_tensor_type(tensor_type):
    if torch.tensor(0).is_cuda:
        old_tensor_type = torch.cuda.FloatTensor
    else:
        old_tensor_type = torch.FloatTensor
        
    torch.set_default_tensor_type(tensor_type)
    yield
    torch.set_default_tensor_type(old_tensor_type)
...
for data in data_loader:
    with set_default_tensor_type(torch.cuda.FloatTensor):
        run_model(data)
</code></pre>

username: Zayd
cooked: <p>Some libraries may break if you change the default tensor type.</p>
<p>I ran into a problem with <code>allennlp</code>'s <code>ElmoEncoder</code> when I had set the default tensor to <code>cuda.FloatTensor</code>.</p>
<p>In case anyone else runs into that error, the first error you get is:</p>
<p><code>RuntimeError: 'lengths' argument should be a 1D CPU int64 tensor</code></p>
<p>which then leads to a second error:</p>
<p><code>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same</code></p>
How to use vgg19 for a segmentation task?

username: I_am_you
cooked: <p>How to use vgg19 for a segmentation task? The output layer would be just be 2D tensor from vgg whereas my input has 4 channels which is RGB and an extra channel.</p>
<p>Th error which is thrown currently is<br>
NotImplementedError: Input Error: Only 3D, 4D and 5D input Tensors supported (got 2D) for the modes: nearest | linear | bilinear | trilinear (got UpsamplingNearest2d)</p>
<p>This is because of the difference in shape of the tensor between output from vgg and the input to the network. How to handle the above situation.</p>
Concatenate each row of a tensor with other rows (not itself)

username: M_Tu
cooked: <p>Hi there,</p>
<p>Is there a way to concatenate each row of tensor with other rows (not itself), for example, I have a N by D tensor and I want to output a N*(N-1) by 2D tensor. A more concrete example:<br>
input: [[1,2,3], [4,5,6], [7,8,9]]<br>
output: [[1,2,3,4,5,6], [1,2,3,7,8,9], [4,5,6,1,2,3], [4,5,6,7,8,9], [7,8,9,1,2,3],[7,8,9,4,5,6]]</p>
<p>Thanks</p>
Memory issue of using nn.DataParallel

username: ginobilinie
cooked: <p>Hi guys,</p>
<p>I’m currently using nn.DataParallel for mutli-gpu (8-gpu) training in a single node. However, if I put the data and model to devices[0], I found the memory on GPU 0 will be huge and make the program exits (cuda out of memory) at the begining of training. Can anyone help?</p>
<p>BTW, I find  if I use DistributedDataParallel, the memory is fine.</p>
<p>Environment:<br>
pytorch 1.0.1<br>
cuda9.0</p>

username: ptrblck
cooked: <p>This effect is described by <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a>.</p>
<p>We generally recommend using DDP. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: ginobilinie
cooked: <p>Thanks. Which syncBatchNormalization do you recommend when using DDP? I’m not sure if the default nn.BatchNorm2d considers multi-gpu ops?</p>
Set up different learning rate for different module when using multiple GPU

username: zhangxixi0904
cooked: <p>Recently I met a problem when using torch.nn.DataParallel.<br>
First I define a model</p>
<pre><code class="lang-python">class ft_net(nn.Module):
    def __init__(self, class_num):
        super(ft_net, self).__init__()
        model_ft = models.resnet50(pretrained=True)
        # avg pooling to global pooling
        model_ft.avgpool = nn.AdaptiveAvgPool2d((1, 1))

        self.backbone = model_ft
        self.classifier = ClassBlock(2048, class_num)

    def forward(self, x):
        feature = self.backbone(x)
        x = self.classifier(feature)
        return x, feature
</code></pre>
<p>Now I want to set two different learning rate for backbone and classifier module. And I want to use multiple GPU so I need to wrap the model with DataParallel. What should I do?  Which option is correct below ?</p>
<pre><code class="lang-python"># option 1
optimizer = torch.optim.SGD([{'params': model.backbone.parameters(), 'lr': 1e-2},
                             {'params': model.classifier.parameters(), 'lr': 1e-1}], 
                             lr=1e-1, weight_decay=1e-5, momentum=0.9)
model = torch.nn.DataParallel(model.cuda(),device_ids=[0,1])

# option 2
model = torch.nn.DataParallel(model.cuda(),device_ids=[0,1])
optimizer = torch.optim.SGD([{'params': model.module.backbone.parameters(), 'lr': 1e-2}, 
                            {'params': model.module.classifier.parameters(), 'lr': 1e-1}], 
                             lr=1e-1, weight_decay=1e-5, momentum=0.9)
</code></pre>

username: ptrblck
cooked: <p>Are you seeing different behavior?<br>
I would assume both should work.<br>
Personally, I would stick to the first one, as I’m usually wrapping the model in DataParallel or DDP as the last step.</p>
Torch.max with tuple axes

username: spacemeerkat
cooked: <p>I think this has come up a few times but never been directly addressed in a topic, but is there a way to get the maximum values of a tensor along one axis using tuple indexing for the axes?</p>
<p>(If this has come up as a full topic before, please let me know and I’ll edit the topic accordingly)</p>
<p>e.g. using numpy one can do the following to get the maximum values in each 3x3 slice of a datacube called “a”</p>
<pre><code class="lang-auto">import numpy as np

a = np.random.uniform(0,1,(5,1,3,3))
max_vals = a.max(axis=(1,2,3))
</code></pre>
<p>However, in PyTorch this will break with an exit error line:</p>
<pre><code class="lang-auto">import torch 

at = torch.tensor(a)
max_vals = a.max(axis=(1,2,3))
</code></pre>
<p>Have I missed a very simple workaround for this problem or does PyTorch not support one-line outputting of maximum values in this way?</p>
<p>Many thanks in advanced!</p>

username: spacemeerkat
cooked: <p>Not sure if this is the best way to solve the problem but one can do it recursively:</p>
<pre><code class="lang-auto">max_vals = a.max(1)[0].max(1)[0].max(1)[0]
</code></pre>

username: paganpasta
cooked: <p>I dont think pytorch.max function supports the tuple functionality. It only works along a single dimension as per the <a href="https://pytorch.org/docs/stable/torch.html#torch.max" rel="nofollow noopener">documentation</a>. Might be a good idea to request for this feature.</p>

username: spacemeerkat
cooked: <p>Really silly question but is there a channel through which I could make a request for a feature like this? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Sure! Feel free to open a feature request on the <a href="https://github.com/pytorch/pytorch/issues">Github issues</a> and explain the use case with your sample code a bit. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Why maxpool2d takes float as a input

username: dato_nefaridze
cooked: <p>i don’t understand why maxpool2d takes float as a input, that caused me bug, i spend 1 hour to fix it</p>

username: albanD
cooked: <p>Hi,</p>
<p>What would you expect it to take as input?</p>

username: dato_nefaridze
cooked: <p>just long would be ok, i mean why not long? ok i  get it now.</p>

username: albanD
cooked: <p>It does not support long because we usually do neural networks, and since long tensors are not differentiable, most neural network functions are not implemented for them (it’s the same for conv for example).</p>
AttributeError: 'bool' object has no attribute 'rfind'

username: mahmoodn
cooked: <p>Hi<br>
Any idea on how to fix this error?</p>
<pre><code class="lang-auto">$ python setup.py clean
Building wheel torch-1.4.0a0+f2a35db
Traceback (most recent call last):
  File "setup.py", line 753, in &lt;module&gt;
    extensions, cmdclass, packages, entry_points = configure_extension_build()
  File "setup.py", line 634, in configure_extension_build
    os.path.dirname(cmake_cache_vars['CUDA_CUDA_LIB']))
  File "/usr/lib64/python2.7/posixpath.py", line 129, in dirname
    i = p.rfind('/') + 1
AttributeError: 'bool' object has no attribute 'rfind'
</code></pre>
<p>Additional info:</p>
<pre><code class="lang-auto">$ python --version
Python 2.7.5
$ git describe --tags
v1.4.0a0-734-gf2a35db
</code></pre>

username: ptrblck
cooked: <p>How did you define the CUDA arguments?<br>
It seems Python has problems splitting the path.</p>

username: mahmoodn
cooked: <p>I have multiple versions of cuda (messy!).<br>
How can I check which version was  fed to setup.py?</p>

username: ptrblck
cooked: <p>Are you passing some arguments as e.g. <code>CUDA_HOME</code>?<br>
Do you have the output of your last build?</p>
<p>Also, since you want to clean your build, could you alternatively remove all files defined in <code>.gitignore</code> manually (<a href="https://github.com/pytorch/pytorch/blob/005146711870a291f4a7a4cc82d97e1e586dcaff/setup.py#L519">reference code</a>).</p>

username: mahmoodn
cooked: <blockquote>
<p>Also, since you want to clean your build, could you alternatively remove all files defined in <code>.gitignore</code> manually (<a href="https://github.com/pytorch/pytorch/blob/005146711870a291f4a7a4cc82d97e1e586dcaff/setup.py#L519" rel="noopener nofollow ugc">reference code</a>).</p>
</blockquote>
<p>To many files there…<br>
I am sure that I set a CUDA path, but can not find that</p>
<pre><code class="lang-auto">$ grep -ri cuda build
build/CMakeCache.txt://Statically link CUDA libraries
build/CMakeCache.txt:CAFFE2_STATIC_LINK_CUDA:BOOL=OFF
build/CMakeCache.txt://Use CUDA
build/CMakeCache.txt:USE_CUDA:BOOL=True
build/CMakeCache.txt://Use NVRTC. Only available if USE_CUDA is on.
</code></pre>
<p>Am I looking into the right folder?</p>
Can not create a tensor in torchscript loaded on mobile

username: l4zyf9x
cooked: <p>I tried to create tensor follow below code:</p>
<pre><code class="lang-auto">...
punc_character_torch = torch.tensor([191, 193, 194, 195, 196], dtype=token.dtype, device=device)
</code></pre>
<p>Convert this code to Torchscript and run it on my pc normally. But when i loaded torchscript to mobile, i got below issue:</p>
<pre><code class="lang-auto">2019-12-06 10:24:21.652 15260-15260/org.pytorch.demo E/AndroidRuntime: FATAL EXCEPTION: main
    Process: org.pytorch.demo, PID: 15260
    java.lang.RuntimeException: is_variable() || !defined() CHECK FAILED at ../torch/csrc/autograd/variable.h
    The above operation failed in interpreter, with the following stack trace:
    at code/__torch__/torchscript/Model.py:1686:27
          ops.prim.RaiseException("Exception")
        if torch.gt(alpha, 0):
          pass
        else:
          ops.prim.RaiseException("Exception")
        device = ops.prim.device(encoder_outputs)
        out_dtype = ops.prim.dtype(encoder_outputs)
        encoder_output_hidden_size = torch.size(encoder_outputs, -1)
        max_encoder_length = torch.size(encoder_outputs, 1)
        punc_character_torch = torch.tensor([191, 193, 194, 195, 196], dtype=ops.prim.dtype(token), device=device, requires_grad=False)
                               ~~~~~~~~~~~~ &lt;--- HERE
</code></pre>
<p>It seem that torchscript still does not support create tensor on mobile device. Any idea to tackle this problem?</p>

username: David_Reiss
cooked: <p>Can you try this with the latest nightly (1.4.0-SNAPSHOT)?  Instructions are in the first half of this post: <a href="https://discuss.pytorch.org/t/android-somethings-went-wrong-with-pytorch-android-1-4-0-snapshot/61009" class="inline-onebox">[Android] Something's went wrong with pytorch_android-1.4.0-SNAPSHOT</a></p>
<p>This part of the code was refactored and the issue should either be fixed or at least have a clearer error message.</p>
_VF.LSTM Implementation

username: yitianz
cooked: <p>Hello!</p>
<p>I’m trying to dig into the implementation of <code>torch.nn.LSTM</code>.</p>
<p>First I look at <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py" rel="nofollow noopener">this file</a> and see that there is a <code>rnn_impls</code> on line 197. Then I see it defined on lines 14-19. And then I go to <code>_VF.py</code> and see <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/_VF.py" rel="nofollow noopener">this</a>.</p>
<p>Perhaps this is due to lack of understanding of <code>types</code> or <code>VariableFunctions</code>, but I’m confused as to where to go next to find where the actual functionality of LSTM is implemented.</p>

username: kabbachokk
cooked: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/49777e67303f608987ec0948c7fd8f46f6d3ca83/torch/csrc/api/src/nn/modules/rnn.cpp" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/49777e67303f608987ec0948c7fd8f46f6d3ca83/torch/csrc/api/src/nn/modules/rnn.cpp" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/49777e67303f608987ec0948c7fd8f46f6d3ca83/torch/csrc/api/src/nn/modules/rnn.cpp</a></h4>
<pre><code class="lang-cpp">#include &lt;torch/nn/modules/rnn.h&gt;

#include &lt;torch/nn/modules/dropout.h&gt;
#include &lt;torch/types.h&gt;
#include &lt;torch/utils.h&gt;

#include &lt;c10/util/Exception.h&gt;

#include &lt;array&gt;
#include &lt;cmath&gt;
#include &lt;cstdint&gt;
#include &lt;functional&gt;
#include &lt;memory&gt;
#include &lt;string&gt;
#include &lt;tuple&gt;
#include &lt;unordered_set&gt;
#include &lt;utility&gt;
#include &lt;vector&gt;

namespace torch {
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/49777e67303f608987ec0948c7fd8f46f6d3ca83/torch/csrc/api/src/nn/modules/rnn.cpp" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Pure Pytorch MultiLabel Confusion Matrix?

username: brookisme
cooked: <p>Hi everyone -</p>
<p>Is there a pure-pytorch ( ie without converting tensors to numpy array ) implementation of a multilabel confusion matrix as with <a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html" rel="nofollow noopener">sklearn.metrics.confusion_matrix</a> ?</p>

username: brookisme
cooked: <p>Just remembered this question was dangling.  My solution is below.  It works on tensors or np.arrays without any changes.  I’ve also added methods for precision/recall and all-of-the-above.</p>
<p>Note: Two divide by zero’s can occur when calculating precision and recall.  (1) when you have 100% true-negatives, or (2) when there are some non-true-negative values.  Note that (1) is good and (2) is bad - so I am returning 1.0 and False respectively, but you can change the config to handle this however you wish.</p>
<p>Here’s a gist:  <a href="https://gist.github.com/brookisme/8f9f06286251af02bb9372fc35bb7fd8" rel="nofollow noopener">https://gist.github.com/brookisme/8f9f06286251af02bb9372fc35bb7fd8</a></p>
<p>Here is some  code:</p>
<pre><code class="lang-auto"># CONFIG
#
BETA=2
RETURN_CMATRIX=True
INVALID_ZERO_DIVISON=False
VALID_ZERO_DIVISON=1.0



#
# METHODS
#     
def confusion_matrix(target,prediction,value,ignore_value=None):
    true=(target==prediction)
    false=(~true)
    pos=(target==value)
    neg=(~pos)
    keep=(target!=ignore_value)
    tp=(true*pos).sum()
    fp=(false*pos*keep).sum()
    fn=(false*neg*keep).sum()
    tn=(true*neg).sum()
    return _get_items(tp, fp, fn, tn)


def precision(tp,fp,fn):
    return _precision_recall(tp,fp,fn)


def recall(tp,fn,fp):
    return _precision_recall(tp,fn,fp)


def fbeta(p,r,beta=BETA):
    if p is None: p=precision(tp,fp)
    if r is None: r=recall(tp,fn)
    beta_sq=beta**2
    numerator=(beta_sq*p + r)
    if numerator:
        return (1+beta_sq)*(p*r)/numerator
    else:
        return 0

      
def stats(
        target,
        prediction,
        value,
        ignore_value=None,
        beta=BETA,
        return_cmatrix=RETURN_CMATRIX):
    tp, fp, fn, tn=confusion_matrix(
        target,
        prediction,
        value,
        ignore_value=ignore_value)
    p=precision(tp,fp,fn)
    r=recall(tp,fn,fp)
    stat_values=[p,r]
    if not _is_false(beta):
        stat_values.append(fbeta(p,r,beta=beta))
    if return_cmatrix:
        stat_values+=[tp, fp, fn, tn]
    return stat_values


#
# INTERNAL
#
def _precision_recall(a,b,c):
    if (a+b):
        return a/(a+b)
    else:
        if c:
            return INVALID_ZERO_DIVISON
        else:
            return VALID_ZERO_DIVISON


def _is_false(value):
    return value in [False,None]


def _get_items(*args):
    try:
        return list(map(lambda s: s.item(),args))
    except:
        return args
</code></pre>
Does torch.autograd.profiler.profile require a GPU?

username: Phil_Tomson
cooked: <p>I ran my model with GPU turned off (CUDA_VISIBLE_DEVICES="") and I see this error:</p>
<p>RuntimeError: /opt/conda/conda-bld/pytorch-nightly_1543051141017/work/torch/csrc/autograd/profiler.cpp:131: no CUDA-capable device is detected</p>
<p>Does torch.autograd.profiler.profile require that a GPU be enabled? I wanted to get profiling info running on CPU only in addition to running with both.</p>

username: albanD
cooked: <p>Hi,</p>
<p>No it should not.<br>
Make sure to set <code>use_cuda=False</code> when creating the profiler.</p>

username: Dale_Song
cooked: <p>Is there a tool to profile the memory usage of tensors generated during network forward&amp;backward?</p>

username: albanD
cooked: <p>For CPU, you can use your prefered python memory profiler like (memory-profiler) to do it.<br>
For GPU, you can see functions like <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.max_memory_allocated">this</a> that will give you the GPU memory used by Tensors.</p>
Is it possible to get per channel mean and variance for images in pytorch?

username: alabijesujoba
cooked: <p>Is it possible to get per channel mean and variance and use on images in pytorch?</p>
<p>I want to center(by subtracting the mean) and normalizing(by dividing by the standard deviation) of an image with 3 channels(RGB) in Pytorch.<br>
Is my approach below correct?</p>
<p><code>centered_images = images - images.mean()</code><br>
<code>normalized_images = images/images.std()</code></p>
<p>images.mean() returns only a value for the batch. Is this correct?</p>

username: sdv4
cooked: <aside class="quote no-group" data-username="alabijesujoba" data-post="1" data-topic="63188">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alabijesujoba/48/18493_2.png" class="avatar"> alabijesujoba:</div>
<blockquote>
<p>centered_images = images - images.mean()</p>
</blockquote>
</aside>
<p>Calling images.mean() (or std) like this will take the mean of the entire tensor, producing a single value, not the per channel mean (or std) that you would like.</p>
<p>One way to get the mean for each channel, you could do the following (assuming your image is shaped like (L, W, 3)):</p>
<pre><code class="lang-auto">mean_c1 = images[:, :, :, 0].mean()
mean_c2 = images[:, :, :, 1].mean()
mean_c3 = images[:, :, :, 2].mean()
</code></pre>
<p>Then to centre the image you could do:</p>
<pre><code class="lang-auto">
centred_images[:, :, :, 0] = images[:, :, :, 0] - mean_c1
centred_images[:, :, :, 1] = images[:, :, :, 1] - mean_c2
centred_images[:, :, :, 2] = images[:, :, :, 2] - mean_c3
</code></pre>
<p>Alternatively, you can search for <code>torchvision.transforms.</code> <code>Normalize</code> <a href="https://pytorch.org/docs/stable/torchvision/transforms.html" rel="noopener nofollow ugc">here</a></p>

username: alabijesujoba
cooked: <aside class="quote no-group quote-modified" data-username="sdv4" data-post="2" data-topic="63188">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sdv4/48/12124_2.png" class="avatar"> sdv4:</div>
<blockquote>
<pre><code class="lang-auto">mean_c1 = image[:, :, 0]
mean_c2 = image[:, :, 1]
mean_c3 = image[:, :, 2]
</code></pre>
</blockquote>
</aside>
<p>Thank you <a class="mention" href="/u/sdv4">@sdv4</a> , do I apply the same approach for standard deviation.  Lastly, images is a batch. would the code snippet given above work?</p>

username: sdv4
cooked: <p>I modified the above answer to include the batch index (and fixed a typo). You can use the same approach for standard deviation.</p>
<p>Lastly, recall that you should calculate the mean and standard deviation from your training set only. You then apply those values to your validation and test set.</p>
Negative Examples for Image Segmentation

username: maaft
cooked: <p>Can anyone point me in the right direction for actively incorporating negative class examples during training of a segmentation network?</p>
<p>Most papers that I’m aware of are just labeling negative classes as background (i.e. network output = 0) but I am wondering if there are any more pro-active approaches to it.</p>

username: travis-harper
cooked: <p>It may be beneficial to perform binary classification on the image to better determine if the object you are interested in segmenting is in it or not. If you are using a U-Net architecture, you can do both classification and segmentation at once by passing the output of the middle layer (which contains the context of the input) into a single output dense layer with sigmoid activation. Any positive pixels in the mask generated by the second half of the U-Net can be disregarded if the binary classification deems that the object is not in the image. I used this technique in a pneumothorax segmentation competition hosted on Kaggle, and it improved my score.</p>

username: maaft
cooked: <p>I discovered this also myself and am already using an additional classifier. Although this does indeed increase the performance, it may fail for cases where Class A and B are present on the image and both get labeled to be Class A.</p>
<p>By the way, it seems like that the additional classification loss alone improves the segmentation results even when the classification result is not used for calculating the final output. I think it just helps to learn better filters in the encoder.</p>
Embedding with very small value generate nan gradient

username: sh0416
cooked: <p>Is there anyone get nan value during training with nn.Embedding?</p>
<p>I recently got nan gradient in embedding layer. I saw nan issue in the nn.Embedding model, but I don’t know whether that issue is resolved or not.</p>
<p>My use case is this, I try to use nn.Embedding as weight for linear layer and during forward, some selected row from nn.Embedding is used for calculating forward. Also, I apply weight_decay in optimizer.</p>
<p>I used to investigate when the nan gradient is generated and I found the nan is generated in the embedding model.</p>
<p>I saw some issue when embedding goes to zero, then nan is generated for gradient. I use pdb to check the row vector which gradient is nan, and I found that some values are very small like 1e-41.</p>
<p>Is there anyone encounter this kind of situation? How to resolve it? I will replace the nn.Embedding to nn.Parameter trying to avoid this kind of nan issue, but I want to know why this kind of situation is generated.</p>

username: sxbio
cooked: <p>I seem to have a very similar issue… Going to run it by disabling the weight decay on the weights of nn.Embedding (similar as <a href="https://github.com/pytorch/pytorch/issues/1402" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/1402</a>).</p>

username: sxbio
cooked: <p>Just to confirm that the weight decay may have been the culprit… Added the following snippet and things are training smoothly…</p>
<p>def group_weight(module):<br>
group_decay = []<br>
group_no_decay = []<br>
for n,m in module.named_parameters():<br>
if ‘input_embedding’ in n:<br>
group_no_decay.append(m)<br>
else:<br>
group_decay.append(m)<br>
assert len(list(module.parameters())) == len(group_decay) + len(group_no_decay)<br>
groups = [dict(params=group_decay), dict(params=group_no_decay, weight_decay=.0)]<br>
return groups</p>

username: sh0416
cooked: <p>I see… Your point is that the weight decay is not stable when the weight is extremely small.</p>
<p>Thanks, I will apply this in my code.</p>

username: sxbio
cooked: <p>Exactly, in my case, my guess is the following:</p>
<ul>
<li>
<p>I have some embedding tokens that are very very uncommon in the training data (they may even not be present in the whole training dataset).</p>
</li>
<li>
<p>In this case, the weight decay essentially updates these embedding weights by<br>
w(t+1) =  w(t) - 2  * decayfactor * w(t),<br>
which slowly approaches zero.</p>
</li>
<li>
<p>When training long enough, these weights become so small that there is some numerical underflow, resulting in NaN.</p>
</li>
</ul>
The True implementation of LeNet5 using pytorch?

username: vaxin
cooked: <p>I am a beginner in pytorch, many examples on the internet for LeNet5 implementations via PyTorch. Most of them are like this:</p>
<blockquote>
<pre><code>   self.c1 = nn.Conv2d(3, 6, 5)
   self.s2 = nn.MaxPool2d(2)
   self.c3 = nn.Conv2d(6, 16, 5)
   self.s4 = nn.MaxPool2d(2)
   self.c5   = nn.Linear(16*5*5, 120)
   self.f6   = nn.Linear(120, 84)
   self.output   = nn.Linear(84, 10)
</code></pre>
</blockquote>
<p>Obviously, it is not the original design by Lecun. This is just connect all feature maps in s2 with all feature maps in c3.</p>
<p>But in Lecun’s paper, the connection relationship between s2 and c3 c is very specified as follow:<br>
6 feature maps connect 3 feature maps in S2, another  6 feature map connect to 4 feature maps in S2, and 3 feature maps  connect to another 4 feature maps in S2, the final 1 feature map connects all feature maps in S2.<br>
As we know, pytorch is very flexiable, how to implement this in pytorch?</p>

username: colesbury
cooked: <p>Two options:</p>
<ol>
<li>
<p>Use 16 nn.Conv2d modules and manually index the output of s2 to pass it to the correct c3 layers. Use torch.cat at to combine the outputs and then add the biases.</p>
</li>
<li>
<p>Zero out the c3.weight tensor at the places which aren’t connected. You’ll have to do this after every SGD step too.</p>
</li>
</ol>

username: vaxin
cooked: <p>Thank you, colesbury!<br>
I think the second way is not friendly for developers. The first way looks nice.</p>

username: maors
cooked: <p>You can find an example implementation in this repo: <a href="https://github.com/maorshutman/lenet5-pytorch" rel="nofollow noopener">https://github.com/maorshutman/lenet5-pytorch</a>.</p>
What is the version of pytorch and libtorch used in pytorch/extension_script ？

username: dalalaa
cooked: <p>I compiled the code following the instruction in docker. but got the following error while running <code>python script.py</code>:</p>
<pre><code class="lang-auto">OSError: /home/huang/vsopencv/extension-script/example_app/build/warp_perspective/libwarp_perspective.so: undefined symbol: _ZN3c1017RegisterOperators25checkSchemaAndRegisterOp_ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEONS0_7OptionsE
</code></pre>
<p>I opened an issue in the repository <a href="https://github.com/pytorch/extension-script/issues/6" rel="nofollow noopener">issue#6</a>, but got no answer.</p>

username: ptrblck
cooked: <p>Are you importing <code>torch</code> before your extension?<br>
This error is often seen, if you try to import the extension directly.</p>

username: dalalaa
cooked: <p>Yes，I imported torch before loading extension，my code：</p>
<pre><code class="lang-auto">import torch

torch.ops.load_library("/home/example_app/build/warp_perspective/libwarp_perspective.so")


@torch.jit.script
def compute(x, y):
    if bool(x[0][0] == 42):
        z = 5
    else:
        z = 10
    x = torch.ops.my_ops.warp_perspective(x, torch.eye(3))
    return x.matmul(y) + z


print(compute.graph)
print(compute(torch.randn(4, 8), torch.randn(8, 5)))

compute.save("example.pt")
</code></pre>
<p>I got this error when running <code>python script.py</code>：</p>
<pre><code class="lang-auto">(base) root@1b05a3dbfb17:/home# python script.py
Traceback (most recent call last):
  File "script.py", line 3, in &lt;module&gt;
    torch.ops.load_library("/home/example_app/build/warp_perspective/libwarp_perspective.so")
  File "/root/local/miniconda/lib/python3.7/site-packages/torch/_ops.py", line 106, in load_library
    ctypes.CDLL(path)
  File "/root/local/miniconda/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/example_app/build/warp_perspective/libwarp_perspective.so: undefined symbol: _ZN3c104impl24ExcludeTensorTypeIdGuardD1Ev
</code></pre>