akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to use the "ImageFolder" function to load single channel image

Pei: <p>Hello! I am new to Pytorch. and l wanna know how to use to load grey-scale image. When i used it to load image data to train Lenet-5 model , it shows that the dataset has three channels and do not fit the model’s input . I tried to translate the image to grey-scale image. but it did not work. would you please tell me the reason and help me to solve this problem? thanks!<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/17e5f9cf23f151bc0ecabbb8a046e2ffea5a24cb.JPG" data-download-href="https://discuss.pytorch.org/uploads/default/17e5f9cf23f151bc0ecabbb8a046e2ffea5a24cb" title="IMG_9344.JPG"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/17e5f9cf23f151bc0ecabbb8a046e2ffea5a24cb_2_666x500.JPG" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/17e5f9cf23f151bc0ecabbb8a046e2ffea5a24cb_2_666x500.JPG, https://discuss.pytorch.org/uploads/default/optimized/1X/17e5f9cf23f151bc0ecabbb8a046e2ffea5a24cb_2_999x750.JPG 1.5x, https://discuss.pytorch.org/uploads/default/optimized/1X/17e5f9cf23f151bc0ecabbb8a046e2ffea5a24cb_2_1332x1000.JPG 2x" data-dominant-color="AAAEB1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_9344.JPG</span><span class="informations">2851×2138 2.53 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

chsasank: <p>This line <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/folder.py#L57" rel="nofollow noopener">here</a> in torchvision is reason for 3 channel loading. You can write your own function to load image like at this line and pass to <code>loader</code> parameter of <code>ImageFolder</code>.</p>
<p>Just a headsup: If you need to load MNIST, there’s a torch vision dataset for that.</p>
<p>Hope this helps.</p>

Prabhat_Sharma: <p>You could also do a transform to Grayscale</p>
<pre><code class="lang-auto">dataset = torchvision.datasets.ImageFolder(
        root = data_path,
        #    By default the imageFolder loads images with 3 channels and we expect the image to be grayscale.
        #    So let's transform the image to grayscale
        transform =  transforms.Compose([transforms.Grayscale(), transforms.ToTensor()])
    )
</code></pre>
Gradient penalty loss with modified weights

tetratrio: <p>Hey, I am currently experiencing an issue with my regularization loss (which is calculated and back propagated through without first being combined with a normal loss) giving me the error ‘trying to backward through the graph a second…’.</p>
<p>I modify the weights using the outputs of another layer and that does not allow me to specify <code>retain_graph=False</code> in <code>torch.autograd.grad()</code>.</p>
<p>Here is a simple code example which fails:</p>
<pre><code class="lang-python">w = nn.Parameter(torch.zeros(10, 10))
dense = nn.Linear(10, 10)

x = torch.rand(1, 10).requires_grad_(True)
y = x.matmul(w.t() * dense(x).mean(0)).mean()

grad = torch.autograd.grad(
    outputs=y,
    inputs=x,
    retain_graph=False,
    create_graph=True,
    only_inputs=True
)[0]

grad.mean().backward()
</code></pre>
<p>And here is a version that works but without modifying the weight.</p>
<pre><code class="lang-python">w = nn.Parameter(torch.zeros(10, 10))

x = torch.rand(1, 10).requires_grad_(True)
y = x.matmul(w.t()).mean()

grad = torch.autograd.grad(
    outputs=y,
    inputs=x,
    retain_graph=False,
    create_graph=True,
    only_inputs=True
)[0]

grad.mean().backward()
</code></pre>
<p>Is there any way for me to run this without using <code>retain_graph=True</code>? Or is there no difference in memory/performance from retaining the graph for my specific example?</p>
Efficient way to calculate the code

JinyangGuo: <p>Hi there,</p>
<p>Now I have a tensor farthest_idxs of size (Batch, Feature) = (24, 32). I also have a tensor nearest_idxs of a size (points, Batch, feature) = (1024, 24, 1023). For one point p and one sample s (i.e. nearest_idxs[p,s,:] of size 1x1023), I want to find the first element in this vector that in the farthest_idxs[s,:] (size 1x32), and return a matrix to record the result (size 24x1024). Is there any efficient way to implement it?</p>
<p>Here is my code, which is a unefficient way to implement it.</p>
<pre><code class="lang-auto">def nearest_indices(self, relation, farthest_idxs):
        '''Generate the nearest indices
            return:
                [B, N] matrix
        '''
        device = relation.device
        nearest_value, nearest_idxs = torch.topk(relation, k=1023, dim=2, largest=False, sorted=True)
        print('nearest idxs', nearest_idxs)
        nearest_idxs = nearest_idxs.transpose(0,1) # 1024x24x1023
        print('transposed nearest_idxs', nearest_idxs.shape)
        N, B, P = nearest_idxs.shape
        upsample_idxs = torch.zeros((B, N), dtype=torch.long).to(device)
        for n in range(N):
            for b in range(B):
                for p in range(P):
                    if nearest_idxs[n, b, p] in farthest_idxs[b,:]:
                        upsample_idxs[b, n] = nearest_idxs[n, b, p]
                        break
        print(upsample_idxs.shape)
</code></pre>
<p>Any help?</p>
Batch inference processing not quicker in compiled model

lbcv: <p>I was able to speed up our inference process on CPU significantly by sending a batch of input to inference at once, rather than inference on one by one.   However, once the pytorch code is embedded in C++ all the time improvements are lost, and even slightly worse than inference on one data point at a time.</p>
<p>Multiprocessing also didn’t work once embedded in C++.</p>
<p>Any ideas of what is holding up the inference in the compiled code?</p>

lbcv: <p>Solved:  the discrepancy was due to our development environment running on torch, and our delivery environment using pytorch.  While pytorch is faster than torch, the inference batch is no faster using pytorch than one by one inference, whereas in torch there is a significant speed up.</p>
How to use SyncBatchNorm in nn.parallel.DistributedDataParallel with v1.1.0?

cb_zhang: <p>Could you please post a short code to introduce the instructions of it?<br>
I have a machine with two GPUs, which means I want to use single process multi gpus.<br>
I tried to use SyncBatchNorm, but failed, sadly like this …</p>
<p>It raise a “ValueError: SyncBatchNorm is only supported for DDP with single GPU per process”…!<br>
But in <a href="https://pytorch.org/docs/stable/nn.html?highlight=distributeddataparallel#torch.nn.parallel.DistributedDataParallel" rel="nofollow noopener">docs of DDP</a>, it says single-process multi-gpus.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.convBlock = nn.Sequential(
            nn.Conv2d(3, 128, 3, 1, 1),
            nn.SyncBatchNorm(128),
            nn.ReLU(),
            nn.Conv2d(128, 512, 3, 1, 1),
            nn.SyncBatchNorm(512),
            nn.ReLU(),
            nn.Conv2d(512, 1, 3, 1, 1),
            nn.SyncBatchNorm(1),
            nn.ReLU()
        )
    def forward(self, x):
        x = self.convBlock(x)
        return x
torch.distributed.init_process_group(backend='nccl', init_method='tcp://127.0.0.1:12345', world_size=1, rank=0)

model = net().cuda()
model = nn.parallel.DistributedDataParallel(model, device_ids=[0, 1], output_device=0)
model = model
 optimizer = torch.optim.Adam(model.parameters())
mseloss = torch.nn.L1Loss()
for i in range(1000):
    x = torch.rand(10, 3, 224, 224)
    y = torch.rand(10, 1, 224, 224)
    x = x.cuda()
    y = y.cuda()
    out = model(x)
    optimizer.zero_grad()
    loss = mseloss(out, y)
    print(i, loss)
    loss.backward()
    optimizer.step()
</code></pre>

pietern: <p>This is expected.</p>
<p>While DDP supports using multiple GPUs from a single process, <code>nn.SyncBatchNorm</code> does not and requires you to use a single GPU per process. Also see the docs for <a href="https://pytorch.org/docs/master/nn.html#torch.nn.SyncBatchNorm" rel="noopener nofollow ugc"><code>torch.nn.SyncBatchNorm</code></a>:</p>
<blockquote>
<p>Currently SyncBatchNorm only supports DistributedDataParallel with single GPU per process. Use torch.nn.SyncBatchNorm.convert_sync_batchnorm() to convert BatchNorm layer to SyncBatchNorm before wrapping Network with DDP.</p>
</blockquote>

tstandley: <p>I think this is worth fixing. Distributed data parallel uses a lot of CPU threads. This is okay for expensive servers used by industry, but a lot of us have a limited number of CPU cores at our disposal.</p>
Why eval takes more memory?

JamryZzz: <p>Memory is enough when i’m training the net, and i save the model.when i load the trained model and test, the code raise ‘out of memory’<br>
train:batch_size=8. test:batch_size=1<br>
i used eval() and ‘with torch.no_grad():’</p>
<p>In addition, I used 2 GPUs for training and testing in the code, but it only called one GPU’s memory when testing</p>

ptrblck: <p>Does your validation loop yield the OOM directly during the first iteration or after a few iterations?<br>
Could you post a (small) code snippet, which shows how you are using the training and validation loop?</p>

JamryZzz: <p>i have solved it,anyway,tks!!</p>
How to get the first dimension size of tensor with onnx?

dalalaa: <p>I want to check if the size of tensor <code>x</code>.<br>
But when I used <code>len(x)</code>, I got error:</p>
<pre><code class="lang-auto">ONNX export failed on ATen operator len because torch.onnx.symbolic_opset11.len does not exist
</code></pre>
<p>and</p>
<pre><code class="lang-auto">KeyError: 'len'
</code></pre>
<p>When I used <code>x.size(0)</code>, I got error:</p>
<pre><code class="lang-auto">RuntimeError: ONNX export failed on ne, which is not implemented for opset 11. Try exporting with other opset versions
</code></pre>
<p>When I used <code>x.shape[0]</code>, I got error:</p>
<pre><code class="lang-auto">UserWarning: ONNX export failed on ATen operator __getitem_ because torch.onnx.symbolic_opset11.__getitem_ does not exist
</code></pre>
<p>and</p>
<pre><code class="lang-auto">KeyError: '__getitem_'
</code></pre>
<p>What can I do to solve this problem? Looking forward to your reply.</p>
TypeError: where() missing 2 required positional argument: "input", "other"

dalalaa: <p>torch.where(condition) works well in torch 1.3.1, but it throws TypeError in torch 1.2.0, but torch.where(condition) can be found in documentation of torch1.2.0:<br>
<a href="https://pytorch.org/docs/1.2.0/torch.html?highlight=torch%20where#torch.where" rel="noopener nofollow ugc">doc of torch 1.2.0</a></p>
<p>How can I use torch.where(condition) in torch 1.2.0?</p>
<blockquote>
<p><code>torch.nonzero(condition, as_tuple=True)</code>  doesn’t work in torch 1.2.0 either.</p>
</blockquote>

dalalaa: <p>It doesn’t work even in torch1.3.1, while working with onnx.</p>
Model is giving RuntimeError for CUDA but I mounted the data

seankala: <p>Hello. The make the question title a bit more eloquent, basically I’m running a model with PyTorch tensors and for some reason every time I try to get the ouput from the model I get</p>
<pre><code class="lang-auto">*** RuntimeError: Expected object of device type cuda but got device cpu for argument #1 'self' in call to _th_index_select
</code></pre>
<p>Here’s the model I have:</p>
<pre><code class="lang-auto">class MatEmbedding(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.embedding_dim = self.config.embedding_dim
        self.num_embeddings = self.config.num_embeddings

        self.embed = nn.Embedding(num_embeddings=self.num_embeddings, embedding_dim=self.embedding_dim)

    def forward(self, inputs):
        indices = inputs[:, 0].to(torch.long)
        embed_result = self.embed(indices)

        return torch.cat([embed_result, inputs[:, 1:]], dim=-1)


### Simple MLP model ###
class MLP(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.in_features = self.config.in_features
        self.out_features = self.config.out_features
        self.hidden_dim = self.config.hidden_dim
        self.embedding_dim = self.config.embedding_dim
        self.alpha = self.config.alpha

        self.embed = MatEmbedding(self.config)
        self.linear1 = nn.Linear(in_features=(self.in_features + self.embedding_dim - 1), out_features=self.hidden_dim)
        self.activ = nn.ELU(alpha=self.alpha)
        self.linear2 = nn.Linear(in_features=self.hidden_dim, out_features=self.out_features)

    def forward(self, inputs):
        out = self.embed(inputs)
        out = self.linear1(out)
        out = self.activ(out)
        out = self.linear2(out)

        return out
</code></pre>
<p>Regarding the data, when I run <code>data.is_cuda</code> I get <code>True</code>. Does anybody have any idea where I might be going wrong? Thanks in advance.</p>

ptrblck: <p>Could you run the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py</code> and post the stack trace here? This would hopefully point to the line of code, which throws this error.<br>
Also, could you add <code>print(indices.device)</code> before the <code>self.embed(indices)</code> call in <code>MatEmbedding</code>?</p>
KeyError: 'prim_Uninitialized'

dalalaa: <p>I got this error when exporting model to ONNX:</p>
<pre><code class="lang-auto">/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/utils.py:651: UserWarning: ONNX export failed on primitive operator Uninitialized; please report a bug
  warnings.warn("ONNX export failed on primitive operator {}; please report a bug".format(op_name))
Traceback (most recent call last):
  File "onnx_demo.py", line 280, in &lt;module&gt;
    do_constant_folding=True,input_names=["input"],output_names=["boxes"]
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/__init__.py", line 143, in export
    strip_doc_string, dynamic_axes, keep_initializers_as_inputs)
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/utils.py", line 66, in export
    dynamic_axes=dynamic_axes, keep_initializers_as_inputs=keep_initializers_as_inputs)
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/utils.py", line 382, in _export
    fixed_batch_size=fixed_batch_size)
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/utils.py", line 262, in _model_to_graph
    fixed_batch_size=fixed_batch_size)
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/utils.py", line 132, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/__init__.py", line 174, in _run_symbolic_function
    return utils._run_symbolic_function(*args, **kwargs)
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/utils.py", line 652, in _run_symbolic_function
    symbolic_fn = sym_registry.get_registered_op(symbolic_name, '', opset_version)
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/symbolic_registry.py", line 91, in get_registered_op
    return _registry[(domain, version)][opname]
KeyError: 'prim_Uninitialized'
</code></pre>
<p>This error could come from these two lines:</p>
<pre><code class="lang-auto">if xy_text.size(0) == 0:
        return torch.tensor(0)
</code></pre>
<p>What’s the function of prim_Uninitialized? How can I define it? Looking forward to your replay.</p>

eellison: <p>prim::Uninitialized is used to represent values which the compiler can prove will never be used. As you correctly guessed, it’s introduced by the early return statement there. You can work around this by removing early returns, breaks, and continues from your scripted function.</p>
<p>Generally, you should expect ONNX to work with tracing and with anything scripted you will likely run into some issues.</p>
Grid Search Using Data Loaders

Achakma: <p>Hello All:</p>
<p>I was looking for a boilerplate code for hyperparameter searching in deep nets. All the available examples fit the model on a randomly generated input whereas practically we use a data loader function to process the input data and fit the model.</p>
<ol>
<li>Can you please provide any code snippet that considers data loaders while executing the GridSearchCV method from the Sklearn library.</li>
</ol>
<p>Thank you very much.</p>
How does cosinembeddingloss work?

msqiqi10: <p><a href="https://pytorch.org/docs/master/nn.functional.html?highlight=cosineembeddingloss" rel="noopener nofollow ugc">Here</a> and the <a href="https://pytorch.org/docs/master/nn.functional.html?highlight=cosineembeddingloss" rel="noopener nofollow ugc">reference</a> it gives shows the parameters to be passed to cosinembeddingloss, but it didnt give a clear definition of target.<br>
So what value am I supposed to assign to target and what purpose does it serve?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d2de71d0008880f75004a5964920fb7d5b44942.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d2de71d0008880f75004a5964920fb7d5b44942" title="%E6%8D%95%E8%8E%B7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d2de71d0008880f75004a5964920fb7d5b44942_2_345x85.png" alt="%E6%8D%95%E8%8E%B7" data-base62-sha1="b0L6Uzk2pI0Vm1xhQIOmj9a4lbA" width="345" height="85" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d2de71d0008880f75004a5964920fb7d5b44942_2_345x85.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d2de71d0008880f75004a5964920fb7d5b44942_2_517x127.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d2de71d0008880f75004a5964920fb7d5b44942_2_690x170.png 2x" data-dominant-color="F2F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%95%E8%8E%B7</span><span class="informations">1091×271 20.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f72cad1b7a1dfa0fb9d5833461c59de6cb1f570.png" data-download-href="https://discuss.pytorch.org/uploads/default/4f72cad1b7a1dfa0fb9d5833461c59de6cb1f570" title="%E6%8D%95%E8%8E%B71"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f72cad1b7a1dfa0fb9d5833461c59de6cb1f570_2_517x500.png" alt="%E6%8D%95%E8%8E%B71" data-base62-sha1="bkPF0CVQrXEnArLXvQN3Qu2pJHW" width="517" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f72cad1b7a1dfa0fb9d5833461c59de6cb1f570_2_517x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4f72cad1b7a1dfa0fb9d5833461c59de6cb1f570_2_775x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f72cad1b7a1dfa0fb9d5833461c59de6cb1f570.png 2x" data-dominant-color="F4F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E6%8D%95%E8%8E%B71</span><span class="informations">864×834 80.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vainaijr: <p>maybe 1 or -1; if inputs are similar, then 1, if inputs are not similar then -1.</p>
Flip and transpose with gpu?

oasjd7: <p>Hi, all<br>
My codes for flip and transpose need a lot of time.<br>
Is there any flip method with gpu? or something bad in my code?</p>
<pre><code class="lang-auto">def tensor_rot_90(x):
    return x.flip(2).transpose(1, 2)

def apply(func, M):
    tList = [func(m) for m in torch.unbind(M, dim=0)]
    res = torch.stack(tList, dim=0)
    return res

def create_rot_batch(images, rot=['90', '180', '270']):
    rot_batch = images
    rot_labels = torch.ones(len(images)) * 0
    counter = 1

    if '90' in rot:
        images_rot90 = apply(tensor_rot_90_digit, images)
        rot_batch = torch.cat((rot_batch, images_rot90), 0)
        rot_labels = torch.cat((rot_labels, torch.ones(len(images)) * counter), 0)
        counter += 1
         ...
</code></pre>
CUDA not working in conda installation

Alankar_Mahajan: <p>I installed Pytorch 1.3 with gpu support in my conda environment using the command given on the website. But, <code>torch.cuda.is_available()</code> is returning False. I already have a system wide install of CUDA 10.0(which I know is not for pytorch 1.3 but the sand-boxed 10.1 is already installed in my pytorch environment) and CuDNN 7.5.1 for TensorFlow 2.0.</p>
<p>On typing <code>torch.version.cuda</code> , I get 10.1.243 which is the sandboxed cuda toolkit installed with pytorch.</p>
<p>How do I resolve this version conflict? Please Help.</p>
<p>Thanks in advance.</p>
<p>Edit: Fixed the issue by updating the device driver.</p>
A question on data balancing and splitting for segmentation

sebbecht: <p>Dear Everyone,</p>
<p>I have made and labeled a small segmentation dataset with 5 classes for first proof-of-concepting and understanding the training workflow for later upscaling. With the small set I realize there are some balance issues and splitting may be tricky. Each class contains 100 100 100 100 37, labeled images respectively, as a planning error meant i didnt have time to take more images for class 5.</p>
<p>Each image only contain a single class. The total number of objects and average objects per image are as follows:<br>
Total: 62119 18958 66478 5238 17851<br>
Avg: 627 187 651 50 525</p>
<p>As for data balancing: I could easily create more images for class 5 which I did not get to create a 100 images. Would it be reasonable to subsample class 1 and 3 to reduce the total number of objects closer to the others? Will I have enough objects then?</p>
<p>As for data splitting: would random .2 split do or should I randomly subsample each class separately to ensure even distribution? will the uneven number of objects per image have negative implications?</p>
<p>I realize the this may not be 100% relevant to pytorch forums but it seemed like a really good place to start with a lot of helpful people (and of course im using pytorch).</p>
<p>Thank you and merry christmas!</p>
How calling an instance of class 'Linear' works if there is no method/function '__call__()' defined in class 'Linear'?

Deepak: <p>Code:</p>
<p>from torch.nn import Linear<br>
model  = Linear(in_features = 1, out_features = 1)<br>
y = model(x)           # calling an instance of class Linear</p>
<p>Although, ‘Linear’ class inherits ‘Module’ class, which has ‘__ call__()’ defined in it. I could not catch it from there. Kindly help.</p>
<p>As far as I know; In python, to call an instance (here model()) of a class, the class must have ‘__ call__()’ defined in it. But how does this work in the case of class- ‘Linear’?</p>

albanD: <p>If a function is not redefined in the class itself, the one from its parent is used. So in this case, the <code>__call__()</code> method from the <code>Module</code> class will be used.</p>

Deepak: <p>Ok. __call ___() from it’s super class (Module) includes following lines:</p>
<blockquote>
<pre><code>    if torch._C._get_tracing_state():
        result = self._slow_forward(*input, **kwargs)
    else:
        result = self.forward(*input, **kwargs)
</code></pre>
</blockquote>
<p>Can you please tell me, from where to get information about this <strong>torch._C._get_tracing_state()</strong> ?<br>
I am attaching the whole __call __() function for your reference:</p>
<blockquote>
<pre><code>def __call__(self, *input, **kwargs):
    for hook in self._forward_pre_hooks.values():
        result = hook(self, input)
        if result is not None:
            if not isinstance(result, tuple):
                result = (result,)
            input = result
    if torch._C._get_tracing_state():
        result = self._slow_forward(*input, **kwargs)
    else:
        result = self.forward(*input, **kwargs)
    for hook in self._forward_hooks.values():
        hook_result = hook(self, input, result)
        if hook_result is not None:
            result = hook_result
    if len(self._backward_hooks) &gt; 0:
        var = result
        while not isinstance(var, torch.Tensor):
            if isinstance(var, dict):
                var = next((v for v in var.values() if isinstance(v, torch.Tensor)))
            else:
                var = var[0]
        grad_fn = var.grad_fn
        if grad_fn is not None:
            for hook in self._backward_hooks.values():
                wrapper = functools.partial(hook, self)
                functools.update_wrapper(wrapper, hook)
                grad_fn.register_hook(wrapper)
    return result
</code></pre>
</blockquote>

albanD: <p>That would be a cpp function. Most likely defined by the jit given the name.<br>
If you look for it, you can find it defined <a href="https://github.com/pytorch/pytorch/blob/1111a6b8100386af82f5612ef551004188cf396c/torch/csrc/jit/python_tracer.cpp#L164">here</a> in cpp.<br>
But that will only be used during jit tracing. So you can safely assume that it is None outside of this use case.</p>

Deepak: <p>Thank you <a class="mention" href="/u/alband">@albanD</a> for your support.</p>
Indexing along some of the dimensions with LongTensor in LibTorch

mnicnc404: <p>Say I have a 3-dim tensor (A) and 2 long tensor (B and C, with only 1 dimension) to be used for indexing. I want to index all of the first dimension and the last 2 dimension with B and C.</p>
<p>In PyTorch, I can index tensor A with <code>A[:, B, C]</code>:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; A
tensor([[[ 0,  1,  2,  3],
         [ 4,  5,  6,  7],
         [ 8,  9, 10, 11]],

        [[12, 13, 14, 15],
         [16, 17, 18, 19],
         [20, 21, 22, 23]]])
&gt;&gt;&gt; B
tensor([0, 2])
&gt;&gt;&gt; C
tensor([3, 1])
&gt;&gt;&gt; A[:, B, C]
tensor([[ 3,  9],
        [15, 21]])
</code></pre>
<p>How can I do the equivalent in libtorch? I know I can use <code>A.index({B, C, D});</code> if I want to index all dimensions with B, C, D, but how can I filter only the last two dimension with B and C?</p>

mnicnc404: <p>After investigating the comments in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Indexing.cpp#L20-L22" rel="nofollow noopener">ATen/native/indexing.cpp</a>, I found the solution which seems to be equivalent to <code>A[:, B, C]</code>:</p>
<pre><code class="lang-auto">A.permute({1, 2, 0}).index({B, C}).permute({2, 0, 1});
</code></pre>
Register onnx op failed

dalalaa: <p>I’m registering custom ops from TorchScript to ONNX refer to <a href="https://pytorch.org/docs/stable/onnx.html#custom-operators" rel="nofollow noopener">https://pytorch.org/docs/stable/onnx.html#custom-operators</a>, But I met the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "onnx_demo.py", line 23, in &lt;module&gt;
    print(torch.ops.custom_ops.intersection)
  File "/home/dai/py36env/lib/python3.6/site-packages/torch/_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator custom_ops::intersection
</code></pre>
<p>Here is my code:</p>
<pre><code class="lang-auto">torch.ops.load_library("build/libexample.so")

def intersection(g,p):
    return torch.ops.my_ops.iou(g[:8],p[:8])

print(torch.ops.my_ops.iou)
register_custom_op_symbolic('custom_ops::intersection', intersection, 11)
# register_custom_op_symbolic('custom_ops::intersection', torch.ops.my_ops.iou, 11)
print(torch.ops.custom_ops.intersection)
</code></pre>
<p>libexample.so is the custom op file of TorchScript defined in C++. How can I convert it to ONNX op?</p>
Can Dataloader use the same child processes for every epoch?

HeavyWeapon: <p>I use PyTorch in Python2 to do some deep learning tasks.</p>
<p>In my Dataset class, I use cuda.jit of Numba to preprocess some 3D point cloud data. When I set num_workers=0 in Dataloader, everything is OK, but the speed is slow.</p>
<p>When I set num_workers &gt; 0, there will be CudaSupportError. Because when forking process, if CUDA has been used in main process, the CUDA will be initialized in sub process again and this will trigger the error.</p>
<p>According to <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-cuda-note" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html#multiprocessing-cuda-note</a>, the best solution is to set start method to “spawn”. However, my whole project is written in Python2, which does not support mulitprocessing.set_start_method(‘spawn’).</p>
<p>Then I tried to initialize cuda-related Tensors after the dataloader creates the subprocesses, i.e., in the first batch. It worked for the first epoch, but the same error occurs in the second epoch. Then I found that the dataloader creates new subprocesses for every epoch, and destroys them when the loop exits.</p>
<p>I wonder that if I can use the same subprocesses in the dataloader for every epoch. Then I can initialize cuda when those subprocesses are created in the beginning, and avoid the CUDA re-initialization errors.</p>

albanD: <p>Hi,</p>
<p>There is already an issue open aiming at doing this here: <a href="https://github.com/pytorch/pytorch/issues/15849">https://github.com/pytorch/pytorch/issues/15849</a></p>

HeavyWeapon: <p>Thanks! I have implemented this function using the method in the link you provide, as below:</p>
<pre><code class="lang-auto">import torch
import torch.utils.data


class _RepeatSampler(object):

    def __init__(self, sampler):
        self.sampler = sampler

    def __iter__(self):
        while True:
            for item in iter(self.sampler):
                yield item 


class DataLoader(torch.utils.data.dataloader.DataLoader):

    def __init__(self, *args, **kwargs):
        super(DataLoader, self).__init__(*args, **kwargs)
        self.batch_sampler = _RepeatSampler(self.batch_sampler)
        self.iterator = super(DataLoader, self).__iter__()

    def __len__(self):
        return len(self.batch_sampler.sampler)

    def __iter__(self):
        for i in range(len(self)):
            yield next(self.iterator)
</code></pre>
Finding indices with zeros from 4D Tensors

Hasan_Sait_ARSLAN1: <p>Hi all,</p>
<p>I have an interesting question for you. I have a batch of images with shape (3, 224, 224). So, if my batch size is 64, let’s say, the final tensor I have has the shape of (64, 3, 224, 224).</p>
<p>Now, here is the question. Suppose that some of the images in this batch is filled with only zeros. What is the fastest way to find out which batch indices are with only zeros?</p>
<p>I don’t want to create for loop for that, since it is slow.</p>
<p>Thanks for your answer.</p>
Model Loading issue Android

mohit7: <p>I was loading and using my model in android app till yesterday but today my model is not loading.<br>
I am using torch-1.4.0-Snapshot version.<br>
Is there anything changed from last day that I should aware?</p>

IvanKobzarev: <p>Hello <a class="mention" href="/u/mohit7">@mohit7</a>,<br>
snapshot version is based on the latest master branch, it is possible that serialization etc. was changed in libtorch.</p>
<p>Did you try it with the retraced/rescripted model using the latest pytorch nightly?</p>
<p>Do you see any messages in logcat?</p>
<p>Also we have one recently resolved issue with our nightlies<br>
(Crash, that happened if <code>Tensor.fromBlob</code> was called before Module loading):<br>
Issue: <a href="https://github.com/pytorch/pytorch/issues/31419" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/31419</a><br>
Fix in master: <a href="https://github.com/pytorch/pytorch/commit/3a19980b78b6638235edef367784ecc3dc37e364" rel="nofollow noopener">https://github.com/pytorch/pytorch/commit/3a19980b78b6638235edef367784ecc3dc37e364</a><br>
Nightlies are already republished.</p>
<p>That might be the reason of your Module loading issue.</p>

vgsprasad: <p>Hi Ivan,</p>
<p>I am also facing same issue while loading model in android with pytorch nightly version. To reproduce this issue, I simply modified the build.gradle file in android ‘HelloWorldApp’ as follows</p>
<p>repositories {<br>
//jcenter()<br>
maven {<br>
url “<a href="https://oss.sonatype.org/content/repositories/snapshots" rel="nofollow noopener">https://oss.sonatype.org/content/repositories/snapshots</a>”<br>
}<br>
}</p>
<p>dependencies {<br>
implementation ‘androidx.appcompat:appcompat:1.1.0’</p>
<pre><code>//implementation 'org.pytorch:pytorch_android:1.3.0'
//implementation 'org.pytorch:pytorch_android_torchvision:1.3.0'

implementation 'org.pytorch:pytorch_android:1.4.0-SNAPSHOT'
implementation 'org.pytorch:pytorch_android_torchvision:1.4.0-SNAPSHOT'
</code></pre>
<p>}</p>
<p>With this change, the app is not working. It is trowing an error</p>
<p>A*<code>/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 28782 (orch.helloworld), pid 28782 (orch.helloworld)</code>*</p>
<p>Thanks<br>
VGS</p>

David_Reiss: <p>This is a bug.  Fix is <a href="https://github.com/pytorch/pytorch/pull/31582">https://github.com/pytorch/pytorch/pull/31582</a> .  Should be available in the next nightly build.  Sorry for the trouble.</p>
Strange behavior in return list tensor

l4zyf9x: <p>Hi guys,<br>
I wanna convert tensor with size (4,3,2) to list of tensor (3,2) and run it on mobile.</p>
<p>Here is code creating torchscript:</p>
<pre><code class="lang-auto">class BatchAsList(torch.nn.Module):
    def __init__(self):
        super().__init__()
    
    def forward(self):
        x = torch.arange(4*3*2).view(4,3,2)
        a = []
        for i in range(x.size(0)):
            a.append(x[i])
        return a
    
module = BatchAsList()
script = torch.jit.script(module)        
script.save('batch_as_list.pth')
</code></pre>
<p>And code run on android:</p>
<pre><code class="lang-auto">    // mTestModule is loaded from file batch_as_list.pth
    Tensor [] output = mTestModule.forward().toTensorList();
    for (int i = 0 ; i&lt; output.length; i++){
      Log.i(TAG, "Output " + i + " shape : " + output[i]);
    }

    long [] output0 = output[0].getDataAsLongArray();
    long [] output1 = output[1].getDataAsLongArray();
    long [] output2 = output[2].getDataAsLongArray();
    long [] output3 = output[3].getDataAsLongArray();

    for (int j = 0 ; j&lt; output0.length; j++){
      Log.i(TAG, "Output : " + output0[j] +" ,"+ output1[j] +" ,"+ output2[j] + ", "+ output3[j] );
    }
</code></pre>
<p>And result:</p>
<pre><code class="lang-auto">2019-12-12 10:35:10.352 18859-18859/org.pytorch.demo D/TEST: /data/user/0/org.pytorch.demo/files/batch_as_list.pth
2019-12-12 10:35:10.381 18859-18859/org.pytorch.demo I/TEST: Output 0 shape : Tensor([3, 2], dtype=torch.int64)
2019-12-12 10:35:10.381 18859-18859/org.pytorch.demo I/TEST: Output 1 shape : Tensor([3, 2], dtype=torch.int64)
2019-12-12 10:35:10.382 18859-18859/org.pytorch.demo I/TEST: Output 2 shape : Tensor([3, 2], dtype=torch.int64)
2019-12-12 10:35:10.382 18859-18859/org.pytorch.demo I/TEST: Output 3 shape : Tensor([3, 2], dtype=torch.int64)
2019-12-12 10:35:10.382 18859-18859/org.pytorch.demo I/TEST: Output : 0 ,0 ,0, 0
2019-12-12 10:35:10.382 18859-18859/org.pytorch.demo I/TEST: Output : 1 ,1 ,1, 1
2019-12-12 10:35:10.382 18859-18859/org.pytorch.demo I/TEST: Output : 2 ,2 ,2, 2
2019-12-12 10:35:10.382 18859-18859/org.pytorch.demo I/TEST: Output : 3 ,3 ,3, 3
2019-12-12 10:35:10.382 18859-18859/org.pytorch.demo I/TEST: Output : 4 ,4 ,4, 4
2019-12-12 10:35:10.382 18859-18859/org.pytorch.demo I/TEST: Output : 5 ,5 ,5, 5
</code></pre>
<p>Shape of output is correct, but something wrong with the value of output. Output that i expected:</p>
<pre><code class="lang-auto">0, 6,  12, 18
1, 7,  13, 19
2, 8,  14, 20
3, 9,  15, 21
4, 10, 16, 22
5, 11, 17, 23
</code></pre>

David_Reiss: <p>This looks like a bug with the handling of tensor strides.  We should be able to get it fixed in PyTorch 1.5.  In the mean time, if you change your line to <code>a.append(x[i].clone())</code>, it works around the bug.</p>

David_Reiss: <p>This is the PR that fixes the issue: <a href="https://github.com/pytorch/pytorch/pull/31584">https://github.com/pytorch/pytorch/pull/31584</a></p>
Pytorch - ONNX - mobile vs Pytorch#jit - mobile

Chame_call: <p>I saw at least two ways to export pytorch models to android device namely via transfering a model from <code>Pytorch</code> to <code>Caffe2</code> and exporting a model with <code>jit</code> module.</p>
<p>What’s the difference of the ways?<br>
Which of them in which situation is preferable?</p>

David_Reiss: <p>Using <code>torch.jit.script</code> or <code>torch.jit.trace</code> is preferable to using Caffe2.  The Caffe2 mobile engine is not being actively developed.</p>
Output larger than specified

Stanley_C: <p>When I double the number of layers in the densely connected neural network, my network’s output is increased despite the output neuron number being the same. Could someone help me debug my neural network? Thank you</p>

ptrblck: <p>By “increased” do you mean the value range or the number of outputs, i.e. tensor shape?</p>

jchaykow: <p>Can we see the class?</p>

Stanley_C: <p>My Apologies.  I merely forgot to change the network model in the inference script.</p>
Having difficulties printing the W in linear regression

Yolkandwhite: <p>I’m an beginner in deep learning</p>
<p>what I’m trying to do is print the epochs, W, b and cost on each steps.</p>
<p>I could print the epochs, b and cost but when I tried to print W, the value error occurs.</p>
<p>I have no idea why it happens.</p>
<p>this is the error what I got.</p>
<p>please someone explain me why this error occurs.<br>
It works perfect without the W part.</p>
<hr>
<p>ValueError                                Traceback (most recent call last)<br>
 in <br>
25     optimizer.step()<br>
26<br>
—&gt; 27     print(“epoch : {:4d}/{}, cost : {:.4f}, b : {:.3f}, W : {:.4f}” .format(epoch, nb_epoch, cost.item(), b.item(), W.item()))</p>
<p>ValueError: only one element tensors can be converted to Python scalars</p>

Yolkandwhite: <p>what I assumed why the error occur is</p>
<p>I defined the W matrix like this</p>
<p>W = torch.zeros((3, 1), requires_grad = True)</p>
<p>so the python couldn’t understand it.</p>
<p>how can I print the W when its type is like that</p>

ptrblck: <p>You could directly print it without the <code>item()</code> call.<br>
<code>item()</code> will return a Python variable. Since <code>W</code> is a multi-dimensional tensor, it cannot be converted to a Python value.</p>
Should I use a model class for a fined tuned model?

maria: <p>I created a fine-tuned model based on resnet18:</p>
<pre><code class="lang-auto">resnet18 = torch.hub.load('pytorch/vision:v0.4.2', 'resnet18', pretrained=True)
reset18_base = torch.nn.Sequential(*(list(resnet18.children())[:-2]))
head_layers = [some layers.. ]
resnet_head = nn.Sequential(*head_layers)
model = nn.Sequential(reset18_base, resnet_head)
</code></pre>
<p>I trained it and it works well, now I need to save and load it elsewhere.<br>
Should I use this code for the model object (to load a state dict into) instead of a Class containing an init and a forward?<br>
I’m wondering about best practices in cases such as mine.</p>

ptrblck: <p>I would say it depends on your future use cases.<br>
If you think this code snippet with <code>nn.Sequential</code> blocks will work in your scripts, just go for it.<br>
On the other hand, if you want to change something in the <code>forward</code> pass, e.g. add some additional residual connections, change or skip some layers, etc., I would rather implement an <code>nn.Module</code> class to get the full flexibility and hackability. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

maria: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
If I choose to implement a module, how can I use the loaded weights from the pretrained ResNet? Is there a way to use the weights without the names matching?</p>

ptrblck: <p>I would create the custom module, load the resnet inside and write the <code>forward</code> method as you wish.<br>
Once this module works as you want it to, save the <code>state_dict</code> and reload this one.<br>
This approach will make it easier to use your custom module without having to map the original resnet parameters to your custom layers.</p>
How to set the correct number of threads for parallelism?

Kushaj: <p>My CPU has 4 physical cores and with hyperthreading, I get 8 threads to work with. When I run <code>torch.get_num_threads()</code> the output is 4.</p>
<p>So my question is, Is this the optimal value or should I set this value to 8?</p>
<p>Also, should I do the same for <code>torch.get_num_interop_threads()</code>?</p>
torch.cuda.FloatTensor at version X

Nagaraj_S_Murthy: <p>What does “torch.cuda.FloatTensor at version X” actually mean?? How to change the version to a different value?</p>

albanD: <p>These versions track how many times a Tensor has been modified inplace. The version increase every time you modify it.<br>
You cannot change the version manually. If you see error with wrong version, you need to remove inplace operations to avoid the version to be bumped.</p>
Memory_mask in nn.Transformer

111146: <p>I’m implementing training codes of transformer model using nn.Transformer.</p>
<p>In the documents, there is a memory_mask optional argument. I read the document but I don’t understand the purpose of this argument.</p>
<p>Could you explain what memory_mask is?</p>
<p>Additionally, is there any code that uses nn.Transformer module?</p>

zhangguanheng66: <p>It’s an attention mask working on the second input of transformer decoder layer. Within the encoder-decoder architecture, it works on the output of transformer encoder, which we call it “memory”.</p>

111146: <p>Then, Why is the shape of memory_mask (T, S)?</p>

zhangguanheng66: <p>because the memory_mask works on the multihead_attn layer in decoder.</p>

xdwang0726: <p>I am wondering how to generate the memory mask? The generate_square_subsequent_mask function can only generate square masks, but memory_mask requires the dimension (T, S). I am wondering is there a built in function in transformer?? Thank you!</p>

vainaijr: <p>maybe change it to something like this,</p>
<pre><code class="lang-auto">def _generate_subsequent_mask(tgt_sz, src_sz):
    mask = (torch.triu(torch.ones(src_sz, tgt_sz)) == 1).transpose(0, 1)
    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
    return mask
</code></pre>
Loss.backward(retain_graph=True) giving Runtime error in Distributed data parallel

Nagaraj_S_Murthy: <p>I’m trying to do DDP in PyTorch. I have multiple loss functions and using retain_graph= True is giving a run time error as shown below:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256]] is at version 57; expected version 56 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!<br>
PyTorch version: 1.0.0<br>
I also updated PyTorch to the latest version(1.4.0), but the same error keeps coming back.</p>
Tf.batch_gather in PyTorch

Xuyang_Bai: <p>I was reimplementing a tensorflow code in pytorch but found that there is no corresponding functions for <code>tf.batch_gather</code>, like the following code,</p>
<pre><code class="lang-python">new_neighbors_indices = tf.batch_gather(neighbors_indices, inds)
</code></pre>
<p>The dimension of <code>neighbors_indices</code> is [a, b] and the dimension of <code>inds</code> is [a, c]. Is there any ways to implement such operations in pytorch?</p>

albanD: <p>Why not use the <a href="https://pytorch.org/docs/stable/torch.html#torch.gather"><code>gather</code></a>  function?<br>
If it does not match what you want, please give a full example of what you want as input and output !</p>

Xuyang_Bai: <p>Hi, Thanks for your reply! I want to change the following two lines into pytorch</p>
<pre><code class="lang-python">new_neighbors_indices = tf.batch_gather(neighbors_indices, new_neigh_inds)
new_sq_distances = tf.batch_gather(sq_distances, new_neighb_inds)
</code></pre>
<p>Here the <code>neighbors_indices</code> is dimension of [a, b] and <code>new_neigh_inds</code> is dimension of [a, c], I find that for the first line I can use <code>torch.gather</code> as,</p>
<pre><code class="lang-python">new_neighbors_indices = torch.gather(neighbors_indices, dim=1, index=new_neighb_inds)
</code></pre>
<p>But the second line <code>sq_distances</code> is dimension of [a, b, k] and  <code>new_neigh_inds</code> is dimension of [a, c],  the expected results is [a, c, k] ,so how could I use <code>torch.gather</code>? I tried <code>torch.gather(sq_distances, dim=1, index=new_neighb_inds)</code> but get the error</p>
<pre><code class="lang-auto">*** RuntimeError: invalid argument 4: Index tensor must have same dimensions as input tensor at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/generic/THCTensorScatterGather.cu:16
</code></pre>

albanD: <p>You want to make sure the indices are the same size as the values: <code>new_neigh_inds.unsqueeze(-1).expand_as(sq_distances)</code>.</p>

Xuyang_Bai: <p>I have tried and it works ! Thanks a lot.</p>
Np.column_stack equivalent

jacek: <p>Hey guys,</p>
<p>is there <code>np.column_stack</code> equivalent in PyTorch? Or is a easy way to replicate its functionality by using <code>stack</code>, <code>cat</code> etc. Or should I just stick to np.column_stack (I don’t need to preserve gradient at the moment)</p>
<p>Thanks!</p>

albanD: <p>It is the same as <code>stack(your_tensors, dim=0)</code>.</p>
How to pass arguments to worker_init_fn in multiprocessing dataloader?

hacky: <p>I want to pass arguments to worker_init_fn in multiprocessing dataloader.</p>
<p>e.g.</p>
<pre><code class="lang-auto">def worker_init_fn(worker_id, **kwargs):
    pass
</code></pre>
<p>BUT in pytorch 1.3 only support one argument that is worker_id</p>
<p>I tried to use lambda or local function which don’t work with multiprocessing.</p>
<p>Any one can help me?  Thanks</p>
Building model from given architecture

Aviv_Shamsian: <p>Hi all,<br>
I trained a FRCNN model using Detectron2 on my custom data and I wanted to use the trained model without holding the Detectron2 repo. Because of my false tries, I have posted this issue over Detectron’s Github and they replied that this option is currently not available.<br>
I thought to build the model myself and use the saved weights in order to be able and use the model. Is there any way (automatic one) to build the model given the architecture (YMAL file)? if so, how do I save the model’s architecture?</p>
<p>Thank you in advance for helping</p>