akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the difference between mul and mul_ about multiplication in pytorch

username: shaoyang_wang
cooked: <p>What is the difference between  mul and mul_  about multiplication in pytorch</p>

username: ptrblck
cooked: <p>The operations with an underscore are inplace operations, i.e. they are working directly on the tensor.<br>
Have a look at the following example:</p>
<pre><code class="lang-auto">a = torch.ones(10)
a.mul_(2)
print(a)
b = a.mul(2)
print(a)
print(b)
</code></pre>

username: JaeDukSeo
cooked: <p>Ah ha! so they just replace the values</p>

username: Weihai_Shen
cooked: <p>excellent example ~, add the result~</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.ones(10)
&gt;&gt;&gt; a.mul_(2)
tensor([2., 2., 2., 2., 2., 2., 2., 2., 2., 2.])
&gt;&gt;&gt; print(a)
tensor([2., 2., 2., 2., 2., 2., 2., 2., 2., 2.])
&gt;&gt;&gt; b = a.mul(2)
&gt;&gt;&gt; print(a)
tensor([2., 2., 2., 2., 2., 2., 2., 2., 2., 2.])
&gt;&gt;&gt; print(b)
tensor([4., 4., 4., 4., 4., 4., 4., 4., 4., 4.])

</code></pre>
Do I have to use the same GPU for saved state dict?

username: WeiQin_Chuah
cooked: <p>Hi,</p>
<p>So I did the training of my neural network, say on GPU <span class="hashtag">#3</span> and the state dict is saved. Do I have to use GPU <span class="hashtag">#3</span> every time when I want to load my state dict?</p>
<p>Also, will it work if I don’t put my model to cuda before loading the state dict into the model?</p>
<p>Thank you.</p>

username: ptrblck
cooked: <p>You can use the <code>map_location</code> argument in <code>torch.load</code> to map your GPU tensors to the CPU as explained in the <a href="https://pytorch.org/docs/stable/torch.html#torch.load" rel="nofollow noopener">note section</a>.<br>
Alternatively you could of course push the model to the CPU before saving its <code>state_dict</code>.</p>

username: WeiQin_Chuah
cooked: <p>Say if I want to save the <code>state_dict</code> for every epoch, do I have to push the model back to GPU after saving the <code>state_dict</code> on CPU?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Yes, the model would have to be copied to the GPU again.<br>
However, depending on the length of your epoch and how often you save the model (e.g. only the best model using the validation accuracy), the latency might be negligible.</p>
ReduceLROnPlateau

username: Megh_Bhalerao
cooked: <p>How is the “loss has not improved” quantified? Is there any threshold e.g. if the loss has not changed by +/- epsilon, the learning rate will be reduced?</p>

username: ptrblck
cooked: <p>Yes, the <code>threshold</code> is used to determine, if a significant change in the metric was registered.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau" rel="nofollow noopener">docs</a> and the arguments for this method.</p>
Diagonal repetition of a matrix?

username: zfzhang
cooked: <p>Hi,<br>
Given <code>m*n</code> input matrix (tensor),</p>
<pre><code class="lang-auto">A B
C D
</code></pre>
<p>I want to repeat it diagonally, into <code>(m*p)*(n+p-1)</code> matrix</p>
<pre><code class="lang-auto">A B 0 0
0 A B 0
0 0 A B
C D 0 0
0 C D 0
0 0 C D
</code></pre>
<p>Where <code>(m,n,p)</code> are provided during init().<br>
How to do it in Pytorch with autograd support? Thank you!</p>

username: akernel
cooked: <p>Ok, so I couldn’t find a super easy or efficient way to do this, and if people have better solutions I’d love to see them, but this is what I came up with:</p>
<pre><code class="lang-auto">def block_diag(x, n):
    """Repeats a tensor diagonally n times as specified by @zfzhang """
    outputs = []
    for row in x:
        out = torch.zeros(n, n + x.shape[1] - 1)
        for ii, elem in enumerate(row):
            d = torch.diag(elem.expand(n))
            padded = torch.nn.functional.pad(d, pad=(ii, x.shape[1] - ii - 1))
            out = out + padded
        outputs.append(out)
    return torch.cat(outputs)
</code></pre>
<p>This method works, but is… not fast:</p>
<pre><code class="lang-auto">In [1]: x = torch.rand(10, 10)

In [2]: %timeit block_diag(x, 2)
1.78 ms ± 53.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [3]: %timeit block_diag(x, 10)
1.76 ms ± 60.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [4]: x = torch.rand(50, 50)

In [5]: %timeit block_diag(x, 2)
43.2 ms ± 2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [6]: %timeit block_diag(x, 10)
44.9 ms ± 1.65 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre>
<p>The method should work with <code>autograd</code>, but may be infeasibly slow, depending on the size of your matrix! Hope that helps a bit.  Again, if someone else has a better solution, please share, I’m very curious!</p>
How to concatenate embedding layer

username: muon
cooked: <p>I am trying to concatenate embedding layer with other features. It doesn’t give me any error, but doesn’t do any training either. Is anything wrong with this model definition, how to debug this?</p>
<p>Note: The last column (feature) in my X  is  feature with word2ix (single word).<br>
Note: The net works fine without the embedding feature/layer</p>
<pre><code class="lang-python">

    
class Net(torch.nn.Module):
    def __init__(self, n_features, h_sizes, num_words, embed_dim, out_size, dropout=None):
        super().__init__()
   
        
        self.num_layers = len(h_sizes)  # hidden + input
        
        
        self.embedding = torch.nn.Embedding(num_words, embed_dim)
        self.hidden = torch.nn.ModuleList()
        self.bnorm = torch.nn.ModuleList()
        if dropout is not None:
            self.dropout = torch.nn.ModuleList()
        else:
            self.dropout = None
        for k in range(len(h_sizes)):
            if k == 0:
                self.hidden.append(torch.nn.Linear(n_features, h_sizes[0]))
                self.bnorm.append(torch.nn.BatchNorm1d(h_sizes[0]))
                if self.dropout is not None:
                    self.dropout.append(torch.nn.Dropout(p=dropout))
                    
            else:
                if k == 1:
                    input_dim = h_sizes[0] + embed_dim
                else:
                    input_dim = h_sizes[k-1]
                
                self.hidden.append(torch.nn.Linear(input_dim, h_sizes[k]))
                self.bnorm.append(torch.nn.BatchNorm1d(h_sizes[k]))
                if self.dropout is not None:
                    self.dropout.append(torch.nn.Dropout(p=dropout))

        # Output layer
        self.out = torch.nn.Linear(h_sizes[-1], out_size)
        
    def forward(self, inputs):

        # Feedforward
        
        for l in range(self.num_layers):
            if l == 0:
                x = self.hidden[l](inputs[:, :-1])
                x = self.bnorm[l](x)
                if self.dropout is not None:
                    x= self.dropout[l](x)
                    
                embeds = self.embedding(inputs[:,-1])#.view((1, -1)
                x = torch.cat((embeds, x),dim=1)
                
            else:
                x = self.hidden[l](x)
                x = self.bnorm[l](x)
                if self.dropout is not None:
                    x = self.dropout[l](x)
            x = F.relu(x)
        output= self.out(x)

        return output

</code></pre>
<p>also posted on stackoverflow <a href="https://stackoverflow.com/questions/57029817/how-to-concatenate-embedding-layer-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/57029817/how-to-concatenate-embedding-layer-in-pytorch</a></p>
Glow example run error

username: glossyfungus
cooked: <p>cloned the repo. All unit tests pass. Tried running the mnist and run in to this issue, any ideas?</p>
<p>$:/hdd1/glow/build_Debug$ ./bin/mnist<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I0712 13:22:43.377008 18779 mnist.cpp:46] Loading the mnist database.<br>
I0712 13:22:53.249351 18779 mnist.cpp:81] Loaded 50000 images.<br>
mnist: …/lib/Graph/NodeValue.cpp:25: glow::NodeValue::NodeValue(glow::Node*): Assertion `(!N || (N-&gt;getNumResults() == 1)) &amp;&amp; “Constructing a value for a multi-res node”’ failed.<br>
Aborted</p>

username: jfix
cooked: <p>Thanks for reporting this! It was broken by a PR from last week. I’ve put up a fix <a href="https://github.com/pytorch/glow/pull/3232" rel="nofollow noopener">here</a>.</p>

username: glossyfungus
cooked: <p>Works with the fix. Thanks for providing a solution rightaway.</p>
Convulational neural network to predict a number

username: Aww
cooked: <p>Hello, I have a database of images where every image is represented by a number, eg. color entropy calculated for that image. I am trying to train a CNN by giving to the network an image and the corresponding number to it and I’m expecting after the train is done to have a CNN that will see an image and predict me a number(eg. color entropy). Can anyone give me ideas? ( I am new to pytorch and I can’t fiind anything similar to this in tutorials). How to train the network so that it can estimate me a number after the CNN gets an image as an input?</p>

username: KFrank
cooked: <p>Hi Public!</p>
<aside class="quote no-group" data-username="Aww" data-post="1" data-topic="50578" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aww/48/14075_2.png" class="avatar"> Aww:</div>
<blockquote>
<p>Hello, I have a database of images where every image is represented by a number, eg. color entropy calculated for that image. I am trying to train a CNN by giving to the network an image and the corresponding number to it and I’m expecting after the train is done to have a CNN that will see an image and predict me a number(eg. color entropy). Can anyone give me ideas? ( I am new to pytorch and I can’t fiind anything similar to this in tutorials). How to train the network so that it can estimate me a number after the CNN gets an image as an input?</p>
</blockquote>
</aside>
<p>Let’s say your images are 256x256 pixels, and each pixel is<br>
given by an rgb value made up of three 8-bit bytes.  Therefore<br>
each image is given by 196608 numerical (8-bit) values.  Convert<br>
these to floats.</p>
<p>Because your example of color entropy doesn’t care about the<br>
<em>structure</em> of the image, that is, whether two given pixels are<br>
close to one another or far apart, convolutional layers won’t add<br>
any value.  (However, the fact that the three color components<br>
of a given pixel are naturally grouped together is something you<br>
might want to build into the structure of your network, rather than<br>
having your network learn.)</p>
<p>So you might try a simple, single-hidden-layer network – something<br>
like this:</p>
<p>Have 512 hidden neurons.  So you start with a fully-connected<br>
layer (<code>nn.Linear (166608, 512)</code>), followed by nonlinear<br>
activations (e.g., <code>nn.ReLU()</code> or <code>nn.Tanh()</code>), followed by a<br>
second fully-connected layer with a single output<br>
(<code>nn.Linear (512, 1)</code>).  You would most likely use the square<br>
of the difference between your predicted and known color entropy<br>
as your loss function (<code>nn.MSELoss()</code>).</p>
<p>My gut tells me that color entropy might be hard for a network to<br>
learn.  As a practical matter, if what you need is the color entropy<br>
of an image, it will be much more satisfactory – faster, more<br>
accurate – simply to calculate it directly, rather than predict it<br>
with a network.</p>
<p>Good luck.</p>
<p>K. Frank</p>
Error while loading tensors RuntimeError: [enforce fail at CPUAllocator.cpp:56] posix_memalign(&data, gAlignment, nbytes) == 0. 12 vs 0

username: Hao_Wang1
cooked: <p>Hi, I am new to pytorch and I tried to load tensors to my server. However, I encountered an error.  Here is my code.</p>
<pre><code class="lang-auto"># -*- coding: utf-8 -*-
import torch
import torchnet
torch.load("imgs0.pt")
</code></pre>
<p>I got an error like</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-7-f6f1f025b648&gt; in &lt;module&gt;
      2 import torch
      3 import torchnet
----&gt; 4 torch.load("imgs0.pt")
      5 model.to(device)

/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    385         f = f.open('rb')
    386     try:
--&gt; 387         return _load(f, map_location, pickle_module, **pickle_load_args)
    388     finally:
    389         if new_fd:

/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    572     unpickler = pickle_module.Unpickler(f, **pickle_load_args)
    573     unpickler.persistent_load = persistent_load
--&gt; 574     result = unpickler.load()
    575 
    576     deserialized_storage_keys = pickle_module.load(f, **pickle_load_args)

/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py in persistent_load(saved_id)
    533             location = maybe_decode_ascii(location)
    534             if root_key not in deserialized_objects:
--&gt; 535                 obj = data_type(size)
    536                 obj._torch_load_uninitialized = True
    537                 deserialized_objects[root_key] = restore_location(obj, location)

RuntimeError: [enforce fail at CPUAllocator.cpp:56] posix_memalign(&amp;data, gAlignment, nbytes) == 0. 12 vs 0
</code></pre>
<p>At first, I thought maybe my cpu’s memory is not big enough for loading this tensor, and I tried to load it with my gpu. However, I got the same error, so I wondered whether this error is caused because of unpickling in python. Thank you for your help.</p>

username: ptrblck
cooked: <p>This error might point to not enough RAM on your machine.<br>
Do you know the size of the data you are trying to load and if so, would it theoretically fit in your RAM?</p>

username: Hao_Wang1
cooked: <p>Thank you! My problem is solved. My data is roughly 8gb, but my cpu’s memory is only 3.75 gb. It’s solved after I upgraded my cpu.</p>
Executing PyTorch Models in WASM

username: zeroexcuses
cooked: <p>Currently, what is state of the art for running PyTorch models on WASM for inference (not training)?</p>
<p>Is there something similar to <a href="https://www.tensorflow.org/js/guide/layers_for_keras_users" rel="nofollow noopener">https://www.tensorflow.org/js/guide/layers_for_keras_users</a> , but for PyTorch instead?</p>
Softargmax in pytorch (in 2D)

username: Naman-ntc
cooked: <p>Softargmax is used quite many place. <a href="http://arxiv.org/abs/1509.06113" rel="nofollow noopener">Deep Spatial Autoencoders for Visuomotor Learning</a> probably introduced it.</p>
<p>Is there any function/layer in pytorch that performs it or any custom implementation.</p>
<p>I implemented it on following idea :</p>
<pre><code class="lang-auto"># consider input [N,C,H,W] (batch, channels, height, width)
temp = input.view(N,C,-1)
m = torch.softmax(dim=-1)
weights = m(temp)
semi_indices = (weights*(torch.arange(H*W).unsqueeze(0).unsqueeze(0).expand(weights.size()))).sum(dim=-1)
indices_x = semi_indices / H
indices_y = semi_indices % W
# maths could be off a bit but basically somewhat like this
</code></pre>
<p>But I really didnt like this implementation, please suggest something (maybe using Softmax2d but it similar i guess). Would there be any other way to do this please suggest!</p>

username: salihkaragoz
cooked: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/MWPainter/cvpr2019/blob/master/stitched/soft_argmax.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/MWPainter/cvpr2019/blob/master/stitched/soft_argmax.py" target="_blank" rel="nofollow noopener">MWPainter/cvpr2019/blob/master/stitched/soft_argmax.py</a></h4>
<pre><code class="lang-py">from __future__ import division

import torch

def _make_radial_window(width, height, cx, cy, fn, window_width=10.0):
    """
    Returns a grid, where grid[i,j] = fn((i**2 + j**2)**0.5)

    :param width: Width of the grid to return
    :param height: Height of the grid to return
    :param cx: x center
    :param cy: y center
    :param fn: The function to apply
    :return:
    """
    # The length of cx and cy is the number of channels we need
    channels = cx.size(0)

    # Explicitly tile cx and cy, ready for computing the distance matrix below, because pytorch doesn't broadcast very well
    # Make the shape [channels, height, width]
</code></pre>

  This file has been truncated. <a href="https://github.com/MWPainter/cvpr2019/blob/master/stitched/soft_argmax.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How could I define a GroupNorm in PyTorch0.3?

username: isaac1644
cooked: <p>when I see GroupNorm’s definition in pytorch1.1.0. I cannot find the decorator called “weak_module” and “weak_script_method”. I really want to know what they have done due to having to use the GroupNorm in pytorch0.3.0 but it was not supported.<br>
Could anyone give me some help?</p>

username: Diego
cooked: <p>Hello, this are flags introduced for the <code>torch.jit</code> module, indicating it to create a static computation graph. So if you are just trying to implement GroupNorm, you shouldn’t worry about it.</p>

username: isaac1644
cooked: <p>it does not have a torch.jit file in version 0.3</p>
Mutli-features prediction with RNNs

username: JohnCwok
cooked: <p>Hi,</p>
<p>I am trying to develop a sequential model (using <code>LSTM</code>s) that has to predict several features at each time-step. I originally wanted to use <code>NLLLoss</code> as a loss function but I am so sure about how to handle in such a multi-feature case.<br>
The input of the model corresponds to the different one-hot encoded features concatenated together.<br>
In the doc, it is said that <code>NLLLoss</code> can accept an input of the form  <code>(N,C,d1,d2,...,dK)</code>, where I assume the different d’s are dimensions you have to create for each of the feature to predict ? I tried it but it didn’t work.</p>
<p>Am I using the wrong loss function or is there a proper way to use NLLLoss for thie specific case ?<br>
I googled a lot, but am still lost …</p>
Tensor size mismatch calling torch.onnx

username: waumbat
cooked: <p>I’m trying to convert the model in the run_classifier.py example from the pytorch-pretrained-BERT repo to ONNX format but run into a problem with a tensor size mismatch. The details are, hopefully, covered in <a href="https://stackoverflow.com/questions/54220042/how-do-you-generate-an-onnx-representation-of-a-pytorch-bert-pretrained-neural-n" rel="nofollow noopener">this case on Stack Overflow</a>.</p>
<p>I notice that libtorch with the PyTorch JIT is another option… perhaps they are the path forward? I would like to get this into a c++ server-side component. I’ve tried asking in the jit category but they appear to be solely focused on existing JIT issues.</p>
<p>Thanks!</p>

username: jbm
cooked: <p>Did you manage to get your model converted? I’ve been trying to convert the bert-base-uncased model to ONNX, but I’m hitting what appears to be a memory-related error: <code>builtins.RuntimeError: [enforce fail at CPUAllocator.cpp:56] posix_memalign(&amp;data, gAlignment, nbytes) == 0. 12 vs 0</code>. Looking into it with <code>free -m</code> I noticed that it happens with as little as 6GB of RAM usage (running on cpu), which seems very odd, since the machine has 24GB installed. Any help appreciated.</p>
<p>For reference, my conversion is super straightforward:</p>
<pre><code class="lang-auto">
output_dir = '/home/james/src/pytorch-pretrained-BERT/outputs'
max_seq_length = 128
num_labels = 128

# Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased')
model_state_dict = os.path.join(output_dir, 'pytorch_bert.bin')
model.load_state_dict(torch.load(model_state_dict))
print('Model loaded: ', model_state_dict)

# Save ONNX
msl = max_seq_length
dummy_input = torch.randn(1, msl, msl, msl, num_labels).long()
output_onnx_file = os.path.join(output_dir, "bert_test.onnx")
torch.onnx.export(model, dummy_input, output_onnx_file)
</code></pre>
DistributedDataParallel Freezes on Model Wrapping

username: askliar
cooked: <p>This question has already been asked before (<a href="https://discuss.pytorch.org/t/distributeddataparallel-deadlock/14809" class="inline-onebox">DistributedDataParallel deadlock</a>) but it does not seem active anymore.</p>
<p>I am running DistributedDataParallel and both with nccl and gloo backends on Pytorch 1.1 my code freezes on the following line:</p>
<pre><code class="lang-auto">model = torch.nn.parallel.DistributedDataParallel(model)
</code></pre>
<p>I am not using built-in dataloader so problem with workers not being set to 0 should not be a problem.<br>
Is there any other place where this problem might be coming from?</p>

username: mateja
cooked: <p>Same problem here, with Pytorch v1.0.1 and NCCL backend. I am running one of the distributed examples from Ignite: <a href="https://github.com/pytorch/ignite/blob/master/examples/mnist/mnist_dist.py" rel="nofollow noopener">https://github.com/pytorch/ignite/blob/master/examples/mnist/mnist_dist.py</a></p>
<p>Mine freezes on the line:<br>
<code>model = DistributedDataParallel(model, [args.gpu])</code></p>
<p>Where <code>model</code> is an instance of LeNet5 and <code>args.gpu</code> is <code>0</code>.</p>

username: Shoaib_Ahmed_Siddiqu
cooked: <p>I have opened up a new issue on this to discuss since I am facing the same problem with DDP.<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/shoaibahmed" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/3884843?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">Issue: Script freezes with no output when using DistributedDataParallel</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/shoaibahmed" target="_blank" rel="nofollow noopener">shoaibahmed</a>
	on <a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">2019-07-13</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
I was trying to evaluate the performance of the system with static data but different models, batch sizes and AMP...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: distributed</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
[SOLVED] TypeError: 'tuple' object is not callable

username: isaac
cooked: <p>Does anyone know what the issue is please?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-130-4a1832287d27&gt; in &lt;module&gt;()
      1 model, criterion, optimizer = build_model()
----&gt; 2 train_model(config, model, criterion, optimizer)
      3 
      4 if config.config_type == 'Q1_4' or config.config_type == 'Q1_5':
      5     dropout_validates = []

&lt;ipython-input-127-85332b119cd8&gt; in train_model(config, model, criterion, optimizer)
     24             optimizer.zero_grad()
     25             # compute loss
---&gt; 26             loss = criterion(model(inputs), targets)
     27             loss.backward()
     28             optimizer.step()

/anaconda/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    355             result = self._slow_forward(*input, **kwargs)
    356         else:
--&gt; 357             result = self.forward(*input, **kwargs)
    358         for hook in self._forward_hooks.values():
    359             hook_result = hook(self, input, result)

&lt;ipython-input-123-94533970db9c&gt; in forward(self, x)
      9 
     10     def forward(self, x):
---&gt; 11         x = nn.ReLU(self.fc1(x))
     12         x = nn.ReLU(self.fc2(x))
     13         x = nn.ReLU(self.dropout(x))

TypeError: 'tuple' object is not callable
</code></pre>
<p>My model is defined like this:</p>
<pre><code class="lang-auto">class MLPb(nn.Module):
    def __init__(self):
        super(MLPb, self).__init__()
        self.config = config
        self.fc1 = nn.Linear(784, 600),
        self.fc2 = nn.Linear(600, 200),
        self.dropout = nn.Dropout(p=0.5), #last layer dropout
        self.fc3 = nn.Linear(200, 10)
        
    def forward(self, x):
        x = nn.ReLU(self.fc1(x))
        x = nn.ReLU(self.fc2(x))
        x = nn.ReLU(self.dropout(x))
        x = nn.Softmax(self.fc3(x))
        return x
    
</code></pre>
<p>EDIT: model defined with <code>,</code> between layers</p>

username: ptrblck
cooked: <p>You should initialize the <code>nn.ReLU</code> modules in your <code>__init__</code> function and just call them in <code>forward</code> or use the functional API: <code>x = F.relu(x)</code>. Now you are creating a <code>nn.ReLU</code> module with the arguments <code>self.fc1(x)</code>.</p>
<p>EDIT: Did the comma issue solve your problem? It still gives an error for me for the aforementioned reason.</p>

username: torayeff
cooked: <aside class="quote no-group" data-username="isaac" data-post="1" data-topic="14930">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/8e8cbc/48.png" class="avatar"> isaac:</div>
<blockquote>
<p>,</p>
</blockquote>
</aside>
<p>remove commas between lines</p>
Creating a pybind wrapper

username: Navneet_M_Kumar
cooked: <p>I am trying to create bindings and successfully so.</p>
<p>However, when importing I face the following error:</p>
<pre><code class="lang-auto">ImportError: dlopen(/anaconda3/envs/reinforcement/lib/python3.5/site-packages/dqn_cpp-0.0.0-py3.5-macosx-10.6-x86_64.egg/dqn.cpython-35m-darwin.so, 2): 
Symbol not found: __ZN3c1019UndefinedTensorImpl10_singletonE
</code></pre>
<p>I have used a library (ALE) in my class and was wondering how to include that for creating the bindings.</p>
<pre><code class="lang-auto">ImportError: dlopen(/anaconda3/envs/reinforcement/lib/python3.5/site-packages/dqn_cpp-0.0.0-py3.5-macosx-10.6-x86_64.egg/dqn.cpython-35m-darwin.so, 2): 
Symbol not found: __ZN12ALEInterface10reset_gameEv
  Referenced from: /anaconda3/envs/reinforcement/lib/python3.5/site-packages/dqn_cpp-0.0.0-py3.5-macosx-10.6-x86_64.egg/dqn.cpython-35m-darwin.so
  Expected in: flat namespace
 in anaconda3/envs/reinforcement/lib/python3.5/site-packages/dqn_cpp-0.0.0-py3.5-macosx-10.6-x86_64.egg/dqn.cpython-35m-darwin.so
</code></pre>
<p>The ALEInterface refers to the shared library. I am not sure how to make a binding in such a scenario.</p>

username: tom
cooked: <p>Did you load PyTorch before you did?<br>
You might have to manually link to torch and / or c10.<br>
On Linux the default seems to be to not explicitly link to torch but rely on “import torch” having made the symbols globally visible.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Navneet_M_Kumar
cooked: <p>I linked and now I am having errors with one of the libraries that I am using.</p>

username: Navneet_M_Kumar
cooked: <p>I have used the import statement and still does not work.</p>
<p>For what its worth, I am working on mac.</p>
Epoch is slow with an addition in loss function

username: ASMIftekhar
cooked: <p>I have a network which is getting loss from three streams. I am adding them as the final loss like this:</p>
<p>loss = nn.BCEWithLogitsLoss(reduction=‘sum’)<br>
lossf=loss(outputs1,labels1.float())/N+loss(outputs2,labels2.float())/N+loss(outputs3,labels3.float())/N<br>
lossf.backward()</p>
<p>Now if I use two of the loss terms then each epoch is taking close to 5 mints. Anytime I add the third one it is taking like 10 minutes. For using two terms I didn’t change anything I just commented out the third term.<br>
Any idea, why this is happening?</p>

username: akernel
cooked: <p>Is the size of <code>outputs3</code> significantly different from the size of <code>outputs1</code> and <code>outputs2</code>?  Also are you using CPU or GPU for your computations?  (And is there any reason you’re passing in <code>labels1.float()</code> instead of just <code>labels1</code>?)</p>
<p>You could also try running <a href="https://pytorch.org/docs/stable/bottleneck.html" rel="nofollow noopener"><code>torch.utils.bottleneck</code></a> on your code to see where exactly the slowdown is happening.</p>
RuntimeError: derivative for _cudnn_rnn_backward is not implemented

username: 111109
cooked: <p>I have no idea to solve this problem.<br>
I impletemented torch.autograd.grad to get the gradient penalty loss, but this error just show again and again , did there any one has same problem?</p>
<p>Thanks,<br>
Peter</p>

username: ngimel
cooked: <p>Unfortunately, double backward for cudnn rnn is not supported, there’s an upstream issue tracking this <a href="https://github.com/pytorch/pytorch/issues/5261" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/5261</a>. Recommended way of doing what you want to do is writing custom rnns with torch script <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/" rel="nofollow noopener">https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/</a></p>
Unable to locate state_dict, vgg16_bn.pth

username: Uri_Isserles
cooked: <p>I am doing transfer learning with vgg16.<br>
Trying to include the state_dict using<br>
vgg16.load_state_dict(torch.load("…/input/vgg16bn/vgg16_bn.pth"))<br>
results in FileNotFoundError: [Errno 2] No such file or directory: ‘…/input/vgg16bn/vgg16_bn.pth’<br>
Haven’t found the path to this file or a place to download it from the internet.<br>
Thanks,<br>
Uri</p>

username: ptrblck
cooked: <p>Are you sure the passed path is right, e.g. you are using three dots at the beginning, while you might wanted to use two?</p>

username: Uri_Isserles
cooked: <p>I am using only two dots.<br>
Thanks</p>

username: akernel
cooked: <aside class="quote no-group" data-username="Uri_Isserles" data-post="1" data-topic="50357">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/uri_isserles/48/14000_2.png" class="avatar"> Uri_Isserles:</div>
<blockquote>
<p>Haven’t found the path to this file or a place to download it from the internet.</p>
</blockquote>
</aside>
<p>Are you looking for a way to download the <code>vgg16_bn</code> pretrained weights?  Or just to load them?  To download them, you could use</p>
<pre><code class="lang-auto">from torchvision import models
vgg16 = models.vgg16_bn(pretrained=True)
</code></pre>
<p>which will create a <code>VGG16</code> network with batch-normalization and pretrained weights, saving the pretrained weights to <code>$TORCH_HOME/checkpoints</code> on your system.</p>
Feeding in a npy file

username: P_C
cooked: <p>Hi, I’m completely new to PyTorch, only having experiences with Tensorflow.</p>
<p>I have datasets with shape [?, 128, 2048], with ? being the data number (This number can change).<br>
The image size is [128, 2048], and I have saved them by shape [500, 128, 2048]<br>
I want to feed these data into the neural network using PyTorch.</p>
<p>I heard that there is a common way in PyTorch that is used by most, and I need some help with it…<br>
I think that putting thousands of [128, 2048] on a RAM at once will trigger a memory error, so I want a way to avoid this problem.<br>
Thanks in advance.</p>

username: akernel
cooked: <p>Hi P_C,</p>
<p>If I understand correctly, you have several <code>.npy</code> files saved, each with five hundred <code>128x2048</code> arrays, and would like to use all of these <code>.npy</code> files in a dataset?  Have you tried something akin to <a href="https://discuss.pytorch.org/t/loading-npy-files-using-torchvision/28481">this link</a>?</p>
<p>Best,<br>
Adam</p>
Cudnn.benchmark for the network

username: Niki
cooked: <p>I am new about using CUDA. I am using the following code for seeding:</p>
<pre><code class="lang-auto">use_cuda = torch.cuda.is_available()
    if use_cuda:
        device = torch.device("cuda:0")
        torch.cuda.manual_seed(SEED)
        cudnn.deterministic = True
        cudnn.benchmark = False
</code></pre>
<p>that is correct if I use this code for the network?</p>
<pre><code class="lang-auto">if use_cuda:
        net.cuda()
        net = torch.nn.DataParallel(net)
        cudnn.benchmark = True
</code></pre>
<p>I mean setting <code>cudnn.benchmark</code>, for seeding <code>False</code> and then for network <code>True</code>. They are related?</p>

username: ptrblck
cooked: <p><code>cudnn.benchmark</code> is a global option, so if you first disable and later enable it, it will be used again if you pass some input to your model, which might yield non-deterministic results.</p>

username: Niki
cooked: <p>Thank you for your response, <a class="mention" href="/u/ptrblck">@ptrblck</a>. I am feeding input to my model, this means <code>cudnn.benchmark</code> should be <code>True</code> for the network?</p>

username: ptrblck
cooked: <p>If you would like to use the benchmark mode, then yes.<br>
If activated, cudnn will perform some benchmarking internally using the current input shape and your model to determine the best performing algorithms to use for the operations.</p>
<p>This will most likely slow down the first iteration, but should generally yield better performance for the following iterations.</p>
<p>However, if you are dealing with varying input shapes, cudnn will benchmark each of them, thus slowing down your code.</p>

username: Niki
cooked: <p>Got it. Thank you very much for your help <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
Doubt in Element 0 of tensors does not require grad and does not have a grad_fn

username: ABHISHEK_SETHI
cooked: <p>Hi, I am unable to debug my code with the existing posts so creating a new. Following are the dataloader, model and training code.</p>
<p>Also, in what all ways can a variable be detached from the computation graph</p>
<p>DataLoader</p>
<pre><code class="lang-auto">class cifar10(Dataset):
    def __init__(self,image_path, labels_file, transform=None):
        self.image_path = image_path
        self.labels = pd.read_csv(labels_file)
        self.transform = transform
        
    def __len__(self):
        return self.labels.shape[0]
    
    def __getitem__(self, idx):
        image_name = self.labels['filenames'][idx]
        img = io.imread(os.path.join(self.image_path, image_name))
        img = torch.from_numpy(img).float().view(-1).unsqueeze(0).cuda() #image.view(-1).unsqueeze(0).cuda()
        label = self.labels['labels'][idx]
        label = torch.Tensor([label]).long().cuda()

        sample = {'image': img, 'label': label}

        if self.transform:
            sample = self.transform(sample)

        return sample
</code></pre>
<p>Model</p>
<pre><code class="lang-auto">class cifarnet(nn.Module):
    def __init__(self, num_input, num_hidden, num_output):
        super(cifarnet, self).__init__()
        self.linear1 = nn.Linear(num_input, num_hidden) 
        self.linear2 = nn.Linear(num_hidden, num_output)
        
    def forward(self, x):
        h_relu = F.relu(self.linear1(x))
        y_pred = self.linear2(h_relu)
        return y_pred
</code></pre>
<p>training code</p>
<pre><code class="lang-auto">for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_data):
        # get the inputs; data is a list of [inputs, labels]
        image, label = data['image'], data['label']

#         zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(image)
        loss = criterion(image, label)

        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 50 == 0:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
</code></pre>

username: ptrblck
cooked: <p>You should pass <code>outputs</code> to the criterion instead of <code>image</code> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Using non_blocking=True with multiprocessing for efficient GPU pre-fetch

username: veritas
cooked: <p>Hello. I am currently trying to implement asynchronous GPU pre-fetching with Pytorch’s DataLoader.</p>
<p>I am doing this because data I/O from disk is a major bottleneck in my task. I therefore want to have a dedicated process reading data from disk at all times, with no gaps.</p>
<p>The idea that I have now is to transfer the data to GPU in Pytorch Dataset’s <code>__getitem__</code> method. Inside <code>__getitem__</code>, I have code that looks like the following.</p>
<p><code>tensor = tensor.to('cuda:1', non_blocking=True)</code> (I have 2 GPUs)</p>
<p>Meanwhile, I am using Pytorch’s multiprocessing as follows.</p>
<p><code>multiprocessing.set_start_method(method='spawn').</code></p>
<p>This line comes before the DataLoaders are initialized.</p>
<p>Also, I have set <code>DataLoader(num_workers=1)</code>.</p>
<p>It is my hope that this will allow the DataLoader will create a dedicated process that will transfer data from host (CPU) to device (GPU) while it is reading data from disk. Meanwhile, memory transfer should be non-blocking so that the worker created by the DataLoader can read data from disk at all times without having to wait while data is being transferred from CPU to GPU through the PCIe bus, which is a high latency data transfer in its own right.</p>
<p>However, I am not at all sure whether this is how Pytorch is doing it. There are several things I do not know.</p>
<p>First, does non_blocking=True work <em>inside</em> DataLoader when multiprocessing is being used? I do not know how CUDA memcpyAsync works with multiprocessing. This is especially confusing since data is being transferred between processes too. I do not know when the Pytorch implementation of non_blocking=True is triggered to block.</p>
<p>Second, does Pytorch DataLoader wait until all processes are finished with their work at each iteration? If so, does it wait until data transfer is finished? I do not see why this should be the case, but I do not know if transferring tensors between processes causes such behavior.</p>
<p>I should mention that I am using a very simple custom <code>collate_fn</code> in Dataloader that just returns the input unpacked into its separate components, without transferring to shared memory, etc. I found that this was necessary for GPU pre-fetching on multiple GPUs.</p>
<p>Finally, if the method that I have proposed does not work, is there any way to implement having a dedicated disk reading process using Pytorch’s DataLoader? Hopefully it should not be interrupted in its data reading task.</p>
<p>Many thanks in advance for anyone who can help out. I know that this requires a lot of in-depth Pytorch and CUDA knowledge. But I think that many people would be interested since data I/O is often a problem for large datasets, especially for those of us who cannot afford specialized hardware at the necessary scale.</p>
Topk on 2D dimension

username: Jose_A_Alvarado_More
cooked: <p>I am trying to run a topk of a 2d matrix</p>
<pre><code class="lang-auto">x = torch.tensor([[1, 2,9,87],[6, 32,8,1],[4,6,7,2],[3,6,2,6]])
print("x",x)
values, indices = torch.topk(x,k=2,dim=0)
print("values",values)
print("indices",indices)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">x tensor([[ 1,  2,  9, 87],
        [ 6, 32,  8,  1],
        [ 4,  6,  7,  2],
        [ 3,  6,  2,  6]])
values tensor([[ 6, 32,  9, 87],
        [ 4,  6,  8,  6]])
indices tensor([[1, 1, 0, 0],
        [2, 2, 1, 3]])
</code></pre>
<p>but I am expecting output for values as:</p>
<pre><code class="lang-auto">values tensor([[ 6, 32,  8,  1],
        [ 4,  6,  7,  2]])
</code></pre>

username: tom
cooked: <p>The topk works column by column, and you get the top values there.<br>
From what you write, do you expect the rows with the largest leading values? I’m afraid there is an easy way to get those…<br>
Depending on what you know about your values, just running topk on the first column and using those indices on x might work.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Jose_A_Alvarado_More
cooked: <p>Thank you… I ended up doing the following:<br>
I needed the last 3 elements of each array so using ranges</p>
<pre><code class="lang-auto">x = torch.tensor([[1, 0,9,87,1],[6, 0,8,1,8],[4,0,7,2,2],[3,0,2,6,3]])
print("x",x)
values, indices = torch.topk(x,k=2,largest=True,dim=0)
print("indices",indices)
#2d data samples
newdataset = torch.zeros(2, 3)
print("newdataset",newdataset)
newdataset[0] = x[indices[0,0].item(),2:5]
newdataset[1] = x[indices[1,0].item(),2:5]
print("newdataset",newdataset)
</code></pre>

username: akernel
cooked: <p>For what it’s worth, a slightly more concise way to do this (that also avoids creating a new tensor, which I find can be cumbersome for autograd purposes) would be</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.tensor([
        [1, 0, 9, 87, 1],
        [6, 0, 8, 1, 8],
        [4, 0, 7, 2, 2],
        [3, 0, 2, 6, 3]
    ])
&gt;&gt;&gt; _, indices = torch.topk(x[:, 0], 2)
&gt;&gt;&gt; indices
tensor([1, 2])
&gt;&gt;&gt; new_dataset = x[indices, 2:5]
&gt;&gt;&gt; new_dataset
tensor([[8., 1., 8.],
        [7., 2., 2.]])
</code></pre>
Real time detection - would using 30 frames as a one batch faster and how?

username: sleung852
cooked: <p>So I am trying to build a real-time object tracking algorithm.</p>
<p>My current pipeline is for each frame:</p>
<ol>
<li>Use OpenCV to obtain webcam frame</li>
<li>frame -&gt; pytorch model -&gt; output</li>
<li>applying object track algorithm</li>
</ol>
<p>But I think doing this would be faster, for each frame:</p>
<ol>
<li>Use OpenCV to obtain webcam frame</li>
<li>stack 30 frames</li>
<li>frame -&gt; pytorch model -&gt; output</li>
<li>apply object track algorithm</li>
</ol>
<p>However I am not sure how to batch 30 frames together? Can anyone give me some guidance?</p>

username: Diego
cooked: <p>Hello, I suggest something like this,</p>
<pre><code class="lang-auto"># store the first frame
&gt;&gt;&gt; frame = torch.rand(3, 500, 500)
&gt;&gt;&gt; batch_frame = frame

# loop over over frame here
batch_frame = torch.stack((batch_frame, new_frame))
</code></pre>
<p>If you loop 29 times <code>batch_frame</code> will have a shape of <code>(30, 3, 500, 500)</code>.</p>
Which metric to use for early stopping?

username: Wafaa_Wardah
cooked: <p>I have a binary classification problem with imbalanced data (1:17 ratio). I want to implement early stopping but not sure which metric value to use as my decider. Actually, if you look at the graph attached, you’ll see accuracy and specificity are similar and sensitivity is the opposite. MCC is pretty low and I’m wondering if I should just use MCC as the decider for early stopping. Or loss? Maybe all hope is not lost and I could train for many more epochs?And how many epochs to check in early stopping and roll back on? Please help, I’m new to early stopping. The plots below represent validation scores of each epoch, trained with unbalanced data and validated on unbalanced data.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/d/dace6ed082402408ce0e29117418bd7aa396c405.png" alt="60_epochs" data-base62-sha1="vdEmScLzS7Zy0RgDiI8U4QrmD3L" width="640" height="480"><br>
When I train with balanced data (undersampled) and validate with unbalanced data, I get these scores: <img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c8e33a048e9dd2e2362f23747bcc55aa758cc92e.png" alt="60_bal" data-base62-sha1="sF8k8VElKjc9aJSKFcFVGj1sybI" width="640" height="480"></p>
<p>Please help with some advice. Thank you.</p>

username: LeviViana
cooked: <p>Maybe these two questions can make things clearer:</p>
<ul>
<li>In your application, what is the most important metric ?</li>
<li>In production, what is the distribution you expect to encounter (balanced or imbalanced) ?</li>
</ul>
<p>My guess is that you want to apply early stopping for the metric the most matters under the distribution you expect to find in production.</p>
Check if tensor_a is view of tensor_b

username: jhultman
cooked: <p>Was wondering if there is a way to check <code>tensor_a.is_view_of(tensor_b)</code>.</p>
<pre><code class="lang-auto">parent = torch.arange(4 * 5).view(4, 5)
child1 = parent[[0, 1]]
child2 = parent[:2]
child3 = parent[2:]
</code></pre>
<p><code>child2</code> and <code>child3</code> are views of <code>parent</code> because of the slice-style indexing, while <code>child1</code> is detached. I tried <code>child2.data_ptr() == parent.data_ptr()</code> and this correctly determines that <code>child2</code> shares memory with <code>parent</code>, but will not work for <code>child3</code> because of the memory offset (maybe with some pointer arithmetic).</p>

username: LeviViana
cooked: <p>Maybe <a href="https://discuss.pytorch.org/t/any-way-to-check-if-two-tensors-have-the-same-base/44310/2">This thread</a> can help you.</p>
<p>BTW, this snippet works for your case:</p>
<pre><code class="lang-python">import torch
def same_storage(x, y):
	x_ptrs = set(e.data_ptr() for e in x.view(-1))
	y_ptrs = set(e.data_ptr() for e in y.view(-1))
	return (x_ptrs &lt;= y_ptrs) or (y_ptrs &lt;= x_ptrs)

same_storage(parent, child1) # False
same_storage(parent, child2) # True
same_storage(parent, child3) # True
</code></pre>