akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Module 'torch' has no 'device' member

Kaneda: <pre><code class="lang-auto">import torch
import torch.nn as nn

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
</code></pre>
<p>This gives me an error <code>Module 'torch' has no 'device' member</code> i’m using version 1.6</p>
<pre><code class="lang-auto">&gt;&gt;&gt; print(torch.__version__)
1.6.0
</code></pre>
<p>It was working in Google Colab, also conda is available</p>
<pre><code class="lang-auto">if torch.cuda.is_available(): print(True)
True
</code></pre>
<p>If i do like this <code>net = net.to("cuda:0")</code> i get no output at all and computer starts freezing</p>
<p>EDIT<br>
I encounter this problem only in visual studio code…<br>
<a href="https://github.com/kanedaaaa/mnist-pytorch/blob/master/models/cnn.py" rel="nofollow noopener">https://github.com/kanedaaaa/mnist-pytorch/blob/master/models/cnn.py</a> here is the code, very simple cnn model</p>

albanD: <p>Hi,</p>
<p>Opening a brand new colab notebook and adding</p>
<pre><code class="lang-auto">import torch
print(torch.__version__)
print(torch.device)
</code></pre>
<p>prints just fine.<br>
Did you override some things in the torch library by any chance?</p>

Kaneda: <p>Uh, no, I’m working in vsc, vsc gives me that error.</p>
<p>It is working in jupyter. I mentioned colab because thought maybe torch.device was changed to something else in new update.</p>
<p>EDIT<br>
I dont think so that i messed up something, since jupyter is not giving me any errors and + its fresh new install</p>

albanD: <p>I don’t think any change happened no.<br>
Can you give more info about your platform, python and how you installed pytorch?</p>

Kaneda: <p>Found a solution</p>
<p><code>device = 'cuda' if torch.cuda.is_available() else 'cpu'</code></p>
<p>My pc froze a bit but finally it gave my desired output.</p>
<p>idk what happened and why torch.device isnt working tho…</p>
Is there a general relationship between the number of parameters and training time?

PhysicsHipster: <p>I’ve recently created a custom pytorch nn model that uses a combination of linear and non-linear connections between layers. I am trying to train a vector of weights on this model. Using about 700 weights, 20 layers, and a batch size of 100, the program takes about 11 seconds to evaluate a single batch while training. For 10 weights, 20 layers, and a batch size of 100, the time for a single batch is about 6 seconds. I find it weird that for 10 weights, the program doesn’t run faster. Is this a sign that I am doing something wrong, or does this seem feasible?</p>
<p>For reference, I am using the weights to create a symetric toeplitz matrix. In both cases, the Toeplitz matrices are about 700 by 700, but in the case where I use 10 weights, only the first 10 diagonals are filled in this matrix.</p>

googlebot: <aside class="quote no-group" data-username="PhysicsHipster" data-post="1" data-topic="92766">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/physicshipster/48/17785_2.png" class="avatar"> PhysicsHipster:</div>
<blockquote>
<p>I find it weird that for 10 weights, the program doesn’t run faster.</p>
</blockquote>
</aside>
<p>Look at it the other way, with 700 weights it doesn’t run 70 times slower, because various parallelization mechanisms amortize the time cost (until weights get too big). Plus, there is <a href="https://en.wikipedia.org/wiki/Amdal%27s_law" rel="noopener nofollow ugc">Amdahl’s law</a>, that explains why time/size dependency is not linear.</p>
Is there .max(axis=1) equivalent in tensorflow?

Sangwon_Jake: <pre><code class="lang-auto">B.overlap-&gt; tensor([[0.0000, 0.0000, 0.0000,  ..., 0.6466, 0.7945, 0.5389]],
       device='cuda:0')

            overlap_for_each_prior, object_for_each_prior = overlap.max(dim=0)  # (8732)
</code></pre>
<p>this .max(dim=0) return the two returns.<br>
is there equivalent method in tensorflow 2.0?</p>

albanD: <p>Hi,</p>
<p>You might want to ask on the Tensorflow question board. More people there are familiar with it and will be able to help you I think.</p>
<p>But checking the tf doc, reduce_max looks like what you’re looking for no? <a href="https://www.tensorflow.org/api_docs/python/tf/math/reduce_max">https://www.tensorflow.org/api_docs/python/tf/math/reduce_max</a></p>
Parallelizing for loop in inference compatible with autograd

droberts42: <p>The network I am using has a batch size of 1, but runs inference via a for loop over 32 samples before calling <code>.backward()</code>.</p>
<p>I would like to parallelize over this for loop. I realize one way of doing so is via <code>DataParallel</code> and <code>DistributedDataParallel</code> but I have few GPUs and my model is very small.</p>
<p>I am wondering if is possible to using <code>torch.multiprocessing</code> or <code>torch.distributed</code> to parallelize over several CPU threads on one single GPU? When trying <code>torch.multiprocessing</code> I get the error</p>
<pre><code class="lang-auto">Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).
</code></pre>
<p>Any suggestions would be much appreciated!</p>

albanD: <p>Hi,</p>
<p>You will have to use the distributed autograd engine to be able to run autograd across multiple processes. But it is experimental right now.</p>
<p>But for your problem, if your tasks are very small, you might want to make sure that the mutltiprocess overhead won’t be greater than the gain.</p>

droberts42: <p>Thank you for your reply <a class="mention" href="/u/alband">@albanD</a> !</p>
<p>I noticed that in PyTorch 1.6 there is now thread parallelism for autograd on CPU. Does this mean I can now achieve what I wanted?</p>

albanD: <p>This new change doesn’t change anything for processes.<br>
But if you have a multithreaded workload, it might help yes.</p>
Std::vector stores inherited torch::nn::Module objects

lampadephoria: <p>Hello,</p>
<p>I have created several modules derived from torch::nn::Module. How can I use std::vector to store them? Below is the sample code. The definition std::vector&lt; torch::nn::Module &gt; is not valid since torch::nn::Module is virtual. Thank you for your help.</p>
<p>class MyModule1Impl : torch::nn::Module {…};<br>
TORCH_MODULE(MyModule1);</p>
<p>class MyModule2Impl : torch::nn::Module {…};<br>
TORCH_MODULE(MyModule2);</p>
<p>class MyModule3Impl : torch::nn::Module {…};<br>
TORCH_MODULE(MyModule3);</p>
<p>auto net1=MyModule1();<br>
auto net2=MyModule2();<br>
auto net3=MyModule3();</p>
<p>std::vector&lt; torch::nn::Module &gt; models;<br>
models.push_back(net1);<br>
models.push_back(net2);<br>
models.push_back(net3);</p>

Vinod: <p>I am not sure if you have any specific requirement to use vector of modules. In C++ API, we have <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/nn/modules/container/modulelist.h" rel="nofollow noopener">ModuleList</a> which could be used for this purpose.</p>
<p>Hopefully this can meet your requirement</p>

lampadephoria: <p>Thanks a lot, Vinod.<br>
That works perfectly.</p>

lampadephoria: <p><a class="mention" href="/u/vinod">@Vinod</a>, any reason that clear() and reset() are not implemented? Maybe I don’t need to worry about it since all the elements are shared_ptr and ModuleList itself is also a shared pointer?</p>

Vinod: <p><a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/nn/modules/container/modulelist.h#L80" rel="nofollow noopener">reset</a> does nothing since it does not have any parameters of its own.</p>

lampadephoria: <p>Got it. Thank you very much.</p>

lampadephoria: <p>The problem with ModuleList is that its size can only grow. I would like to be able to reset the container through reset() or clear(). However _modules is private.<br>
Please let me know if I misunderstood the purpose of ModuleList. Any idea of creating a container of torch::nn::Module that can be reset is highly appreciated.</p>

Vinod: <p>Currently there are 2 containers which can hold the list of modules, <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/nn/modules/container/modulelist.h" rel="nofollow noopener">ModuleList</a> and <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/api/include/torch/nn/modules/container/sequential.h" rel="nofollow noopener">Sequential</a>. Both of them have different use cases. Neither of them supports deleting the contained list of modules. If the requirement is to delete the list of contained modules, a crude solution is to free the current ModuleList and create a new one.<br>
If there is any specific requirement / usecase we could discuss it in the community.</p>

lampadephoria: <p>I (for now) only need a simple std::vector&lt;std::shared_ptr<a>torch::nn::Module</a>&gt; style container so that I can insert and clear the container. I do not need the register_module() functionality yet. Currently I copied ModuleList.h locally and implemented reset()/clear() function.</p>
How to assing average of two models' weights into a new model?

Stefano_Setti: <p>Hi, so i have this network:</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()

        self.conv2d = nn.Sequential(
                nn.Conv2d(1, 64, (3,6), (1, 1)),
                nn.Hardsigmoid()
            )

    def forward(self, input):
        output = self.conv2d(input)
        
        return output
</code></pre>
<p>The network is trained on two different dataset so to obtain two different models:</p>
<pre><code class="lang-auto">model1
model2
</code></pre>
<p>my objective is to average the weights and bias of them and put the result on a third model:<br>
<code>model3</code></p>
<p>How can i achieve that?</p>

mathematics: <p>Cool question, I’ve tried,</p>
<p>I think, here’s you can solve this,<br>
We can get weights of any model by  <code>model.parameters()</code> which can be append into list as below</p>
<pre><code class="lang-auto">params1 = []
for param in model1.parameters():
  params1.append(param.data)
</code></pre>
<p>similary do this to trained model2 and save in list <code>params2</code></p>
<p>Now you initialize weights of model3 as</p>
<pre><code class="lang-auto">model3 = Network()
params3 = iter(params1 + params2)
for param in model3.parameters():
  param.data = next(params3)

</code></pre>

Stefano_Setti: <p>but in this way there is no chance to average the weight from model1 and model2 right?</p>

mathematics: <p>oh, You told to average weights and put on model3, I’ve misunderstood it,</p>
<p>I think your question is really to ensemble two or more pretrained model, here you can do that<br>
<a href="https://discuss.pytorch.org/t/combining-trained-models-in-pytorch/28383">https://discuss.pytorch.org/t/combining-trained-models-in-pytorch/28383</a><br>
Hope it solves your problem</p>

Stefano_Setti: <p>no in the post that you sended me he combine two models, my goal is to get two identical model, get weights and bias and average them into a third model that has the same structure of the previous two</p>

mathematics: <p>model parameters are actually weights and biases. As  earlier, you can also get name of those parameters as</p>
<pre><code class="lang-auto">for name, params in model.named_parameters():
  print(name)
</code></pre>
<p>gives your weights and biases names as</p>
<pre><code class="lang-auto">conv2d.0.weight
conv2d.0.bias
dense.0.weight
dense.0.bias
</code></pre>
Resampling for IterableDatasets

MaxHalford: <p>Hello everyone. I noticed that the <code>sampler</code> parameter of <code>DataLoader</code> doesn’t work with an <code>IterableDataset</code>, as explained in <a href="https://github.com/pytorch/pytorch/issues/28743" rel="nofollow noopener">this issue</a>. I have an interest in online learning and have thus developed some methods for balancing a streaming dataset through resampling. I’ve recently implemented these as part of a <a href="https://github.com/MaxHalford/pytorch-resample" rel="nofollow noopener">small package</a> dedicated to PyTorch.</p>
<p>I’m not very accustomed to PyTorch and its ecosystem, and would thus appreciate some feedback on the package. Do you find it useful? Is it idiomatic? Would you like to see any other features?</p>
<p>I’m sorry in advance if this post is off-topic. I’m happy to remove my post if this isn’t suitable.</p>
How to right shift each row of a matrix?

Bebo: <p>I have a matrix whose shape is (<code>TxK</code>, and <code>K &lt;&lt; T</code>). I want to extend it into shape <code>TxT</code>, and right shift the <code>i</code>-th row with <code>i</code> steps.</p>
<p>For an example:</p>
<pre><code class="lang-auto">inputs: T= 5, and K = 3
1 2 3
1 2 3
1 2 3
1 2 3
1 2 3

expected outputs:
1 2 3 0 0 0
0 1 2 3 0 0
0 0 1 2 3 0
0 0 0 1 2 3
0 0 0 0 1 2
</code></pre>
<p>My solutions:</p>
<pre><code class="lang-python">right_pad = T - K + 1
output = F.pad(input, (0, right_pad), 'constant', value=0)
output = output.view(-1)[:-T].view(T, T)
</code></pre>
<p>My solution will cause the error – <code>gradient computation has been modified by an inplace operation</code>. Is there an efficient and feasible way to achieve my purpose?</p>
Torch==1.6.0 on Google cloud with CUDA

JeffO: <p>Sorry if this is a newb question.  I don’t have a lot of experience with GPUs…</p>
<p>I created a GCE instance like this:</p>
<pre><code>gcloud compute instances create my-instance --image-family=pytorch-latest-gpu ...
</code></pre>
<p>The instance that spins up has torch 1.4 instead of the latest one.  With this version, <code>torch.cuda.is_available()</code> returns True.</p>
<p>After doing <code>pip install -U torch</code> to get the latest version, <code>torch.cuda.is_available()</code> returns False.</p>
<p>What is the best way to get a working torch==1.6.0 instance on GCE?</p>

ptrblck: <p>I would assume this command should install the latest stable PyTorch version with CUDA10.2.<br>
Which GPUs are you using in the node? Maybe the compute capability is not shipped anymore in the binaries?</p>

JeffO: <p>I’m using a V100 GPU.</p>
<p>I’m guessing that torch 1.4 and 1.6 use different CUDA versions so that the previously installed drivers don’t work with torch 1.6.</p>
<p>It is a bummer that Google isn’t updating the pytorch-latest-gpu image.  <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

ptrblck: <p>Could you post the install log from the notebook when you are trying to install the latest stable PyTorch version?</p>
How to make nn.Linear be y=Wx+b?

Hanxiong_Chen: <p>When I read the document the description of nn.Linear() is implemented as y = xA^T + b. However, I need y=Wx+b. Is there anyway to get y=Wx+b with nn.Linear()?</p>

InnovArul: <p><code>nn.Linear</code> does exactly <code>y=Wx+b</code>.<br>
In the documentation, <code>A</code> is a matrix and <code>A^T</code> denotes A-transpose.<br>
This is due to how the data is passed through the network.</p>
<p><code>nn.Linear</code> accepts inputs of size <code>(B, D_in)</code> and transforms it into output of size <code>(B, D_out)</code> by a linear transformation using the weight matrix of size <code>(D_out, D_in)</code>.</p>

Hanxiong_Chen: <p>Thanks for the answer. Actually I already solve this problem. Previously I had a misunderstanding of this implementation. I only treat my input vector as a column vector, so I need a left multiply weight matrix. However, I can use a row vector instead so right multiply a matrix then I can still get a new row vector.</p>

lijingxin: <pre><code class="lang-auto">m = nn.Linear(20, 30)
input = torch.randn(128, 20)
output = m(input)
</code></pre>
<ol>
<li>W: (30,20), x should transpose to(20,128)  so Wx = (30,20)*(20,128)=(30,128)</li>
<li>xW^T = (128,20)*(20,30)=(128,30)<br>
Wx and xW^T are the same result, but Wx is by column  and xW^T is by row? Thanks.</li>
</ol>
'numpy.ndarray' object has no attribute 'contiguous'

lifeblack: <p>Hi!<br>
I have encountered this error:</p>
<pre><code class="lang-auto">'numpy.ndarray' object has no attribute 'contiguous'
</code></pre>
<p>in this code:</p>
<pre><code class="lang-auto">a=numpy.array([[1,2,3,4],[5,6,7,8],[1,2,3,4],[5,6,7,8]])
dct.dct_2d(a)
</code></pre>
<p>that the dct func contains:</p>
<pre><code class="lang-auto">x_shape = x.shape
N = x_shape[-1]
x = x.contiguous().view(-1, N)
</code></pre>
<p>Could you mind helping me solve this?</p>

JuanFMontesinos: <p>contiguous is a torch method, it has to be a torch tensor not a numpy array</p>
TypeError: lstm() received an invalid combination of arguments

BroccoliFever: <p>I apologize in advance that this question appears a lot; however, I could not find an answer to my question in the forums, so I am asking this question anew. Rather than include blocks of code, I have uploaded the project to <a href="https://github.com/BroccoliFever/udacity-char-rnn-practice/blob/master/Character_Level_RNN_Exercise.ipynb" rel="nofollow noopener">GitHub</a>. Basically, I am <em>pretty</em> sure that I am creating–and calling the forward function on–an LSTM cell correctly, as I am following <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM" rel="nofollow noopener">the documentation</a>, but obviously I am doing something wrong. One thing to note is that I am doing this as part of an online course provided by Udacity, and the code works perfectly fine on their cloud machine, so it is something to do with the code on my machine.</p>
<p><strong>machine specs:</strong></p>
<ul>
<li>Ubuntu 20.04</li>
<li>Python 3.6</li>
<li>Built on conda using<br>
– conda install pytorch torchvision cudatoolkit=10.2 -c pytorch</li>
</ul>
<p>Please let me know if there is any more info that I need to provide to help with this. I’d love to figure why this code is not running on my local machine.</p>

mariosasko: <p>Please provide all the relevant code here. Try to print the input/hidden shapes that you are passing to the lstm when the error occurs so it’s easier for us to find a solution.</p>

BroccoliFever: <p>All the relevant code is included in the Jupyter Notebook, which I link to (it’s the GitHub link)</p>
Torch.jit.script(module) vs @torch.jit.script decorator

Vlad_A: <p>What is the purpose of <span class="mention">@torch.jit.script</span> decorator? Why is adding the decorator “<span class="mention">@torch.jit.script</span>” results in an error, while I can call torch.jit.script on that module, e.g. this fails:</p>
<pre><code class="lang-auto">import torch

@torch.jit.script
class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h

my_cell = MyCell()
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.script(my_cell, (x, h))
print(traced_cell)
traced_cell(x, h)

"C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\torch\jit\__init__.py", line 1262, in script
    raise RuntimeError("Type '{}' cannot be compiled since it inherits"
RuntimeError: Type '&lt;class '__main__.MyCell'&gt;' cannot be compiled since it inherits from nn.Module, pass an instance instead
</code></pre>
<p>While the following code works well:</p>
<pre><code class="lang-auto">class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h

my_cell = MyCell()
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.script(my_cell, (x, h))
print(traced_cell)
traced_cell(x, h)
</code></pre>

SplitInfinity: <p><code>@torch.jit.script</code> can be used as a decorator on functions to script them. So, these two code snippets are roughly equivalent:</p>
<p><strong>Decorator</strong></p>
<pre><code class="lang-auto">@torch.jit.script
def fn(a: int):
   return a + 1

fn(3) # fn here is a scripted function
</code></pre>
<p><strong>Function Call</strong></p>
<pre><code class="lang-auto">def fn(a: int):
   return a + 1

s_fn = torch.jit.script(fn)
s_fn(3) # s_fn here is a scripted function
</code></pre>
<p>This decorator can also be used on classes that extend <code>object</code> to script them (known as <a href="https://pytorch.org/docs/stable/jit.html#torchscript-classes" rel="nofollow noopener">Torchscript classes</a>).</p>
<p>Because only <em>instances</em> of <code>Modules</code> can be scripted, <code>@torch.jit.script</code> cannot be used as a decorator on a subclass of <code>Module</code>. You must create an instance of the <code>Module</code> and pass it to <code>torch.jit.script</code> in order to script it.</p>
Optimization issue with MSE loss in Pytorch

Yash-567: <p>I made a classic matrix factorization model for movie recommendation system using keras using batch size 128, Stochastic Gradient Descent and mse loss on the 20M movielens dataset. It gave me the minimum loss of 0.59 in 5 epochs. Then I recreated the exact same model ( the architecture was same and I also checked the total params in both to be same just for confirmation ) in Pytorch. The loss that I get for every batch in Pytorch is around 0.007 (in case of keras it came down from 1 to 0.5). When I multiply it with 128, the loss is in the range of 0.9 so I thought of comparing the pytorch loss after multiplying with 128. Now the issue is that the Pytorch model takes forever to train as I have had more than 40 epochs but the loss won’t go below 0.78 (after multiplying with 128) and the results are poor in comparison to keras model. Can anyone please explain me what I did wrong. My guess was that I did something wrong with the training loop. It would also help if someone shows how a training loop is supposed to be with Stochastic Grad descent in Pytorch with batch size 128 without the default pytorch DataLoader because I guess it does not support csv as of now. Thanks in advance</p>
<p>The Link for keras model code is: <a href="https://gist.github.com/Yash-567/344ad748be4c4d3df1344eb506e38d58" rel="nofollow noopener">https://gist.github.com/Yash-567/344ad748be4c4d3df1344eb506e38d58</a></p>
<p>The link for Pytorch implementation is: <a href="https://gist.github.com/Yash-567/3da2cc10ffc261f565d5cfa0b040f544" rel="nofollow noopener">https://gist.github.com/Yash-567/3da2cc10ffc261f565d5cfa0b040f544</a></p>
How to edit forward function of pretrained model from pretrainedmodels library

Tomash: <p>Hi guys,<br>
I am trying to edit forward function of some pretrained model (f.e. resnet50 or senet154) from pretrainedmodels library by adding additional input parameter.<br>
What do u think about solution like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.net = pretrainedmodels.__dict__["model_name"](pretrained=True)
        self.fc1 = nn.Linear(x_dim+y_dim, output_size_like_in_last_linear)
        
    def forward(self, x, y):
        x1 = self.net(x)
        x2 = y.
        x = torch.cat((x1, x2), dim=1)
        x = F.relu(self.fc1(x))
        return x
</code></pre>
<p>Do you think it will work proper? Do you have any suggestions?<br>
What I want to do is to consider parameter y in the last layer of the pretrained model.<br>
Anyone?</p>

jvhoffbauer: <p>Yes, this makes sense to me, if you want to consider both the output of the pretrained model and <code>y</code> in a dense layer at the end of your model.</p>
<p>You might want to use <code>net.eval()</code> and <code>net.train()</code> on the pretrained model and set <code>requires_grad</code> depending on whether you want to train the pretrained model or keep the weights fixed while training your own model.</p>
Calculating MSE and MAE

alameer: <p>Hi</p>
<p>I would like to calculate the MSE and MAE for a PyTorch model. How do I go about doing this? Are there functions already available to do this, please? Many thanks in advance</p>

mariosasko: <p>Yes, they are already implemented in Pytorch. Use <code>nn.MSELoss</code> and <code>nn.L1Loss</code> respectively.</p>

alameer: <p>Thanks a lot <a class="mention" href="/u/mariosasko">@mariosasko</a></p>
differing results when using model to infer on a batch vs individual with pytorch

Arun_Jose: <p>I have a neural network which takes input tensor of dimension (batch_size, 100, 1, 1) and produces an output tensor of dimension (batch_size, 3, 64, 64). I have differing results when using model to infer on a batch of two elements and on inferring on elements individually.</p>
<p>With the below code I initialize a pytorch tensor of dimension (2, 100, 1, 1). I pass this tensor through the model and I take the first element of the model output and store in variable result1. For result2 I just directly run the first element of my original input tensor through my model.</p>
<pre><code class="lang-auto">

inputbatch=torch.randn(2, Z_DIM, 1, 1, device=device)
inputElement=inputbatch[0].unsqueeze(0)

result1=model(inputbatch)[0]
result2=model(inputElement)

</code></pre>
<p>My expectation was result1 and result2 would be same. But result1 and result2 are entirely different. Could anyone explain why the two outputs differ.</p>

Arun_Jose: <p>The issue was not just using  <code>model = model.eval()</code> . I was able to solve this by calling model.eval()</p>
Help in Load a single image in a pretrained pytorch net: RuntimeError

alejo30: <p>I load my pretrained pytorch but i cant predict a single image</p>
<p>Part of Code:</p>
<p>from torchvision import transforms</p>
<p>transform = transforms.Compose([            #[1]</p>
<p>transforms.Resize(256),                    #[2]</p>
<p>transforms.ToTensor(),                     #[4]</p>
<p>transforms.Normalize(                      #[5]</p>
<p>mean=[0.485, 0.456, 0.406],                #[6]</p>
<p>std=[0.229, 0.224, 0.225]                  #[7]</p>
<p>)])</p>
<p>from PIL import Image</p>
<p>img = Image.open("/content/gdrive/My Drive/DCN(data)/Valid/Covid/ryct.2020200034.fig2.jpeg")</p>
<p>img_t = transform(img)</p>
<p>batch_t = torch.unsqueeze(img_t, 3)</p>
<p>batch_t.size()</p>
<p>model.eval()</p>
<p>out = model(batch_t)</p>
<p>print(out.shape)</p>
<p>out</p>
<p>ERROR:<br>
RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
1 model.eval()<br>
----&gt; 2 out = model(batch_t)<br>
3 print(out.shape)<br>
4 out</p>
<p>6 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in conv2d_forward(self, input, weight)<br>
340                             _pair(0), self.dilation, self.groups)<br>
341         return F.conv2d(input, weight, self.bias, self.stride,<br>
–&gt; 342                         self.padding, self.dilation, self.groups)<br>
343<br>
344     def forward(self, input):</p>
<p>RuntimeError: Given groups=1, weight of size 8 3 3 3, expected input[3, 256, 276, 1] to have 3 channels, but got 256 channels instead</p>

mathematics: <p>Hi there,<br>
please wrap your code with preformatted text ,</p>
<p>I think , your problem is given in your input i.e <code>[3,256,256,1]</code> ,</p>
<p>to clear, Pytorch uses <code>[batch, channel, height, width]</code> , so you need to change shape in that way&gt;</p>
<p>Here i found detail about this, <a href="https://discuss.pytorch.org/t/dimensions-of-an-input-image/19439/2">https://discuss.pytorch.org/t/dimensions-of-an-input-image/19439/2?u=ptrblk</a>.</p>
<p>Hope this works</p>
Understanding Backpropogation of the loss

Griffintaur1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Let say I have two versions of the same image and the output of the model be <code>I_s</code> and <code>I_w</code> and the loss is some loss function.<br>
Now I want to understand say if I use one <code>loss(I_w.detach()_, I_s)</code> and the other <code>loss(I_w, I_s)</code>.<br>
What’s the difference between the two way of computing loss.<br>
What’s the impact of first versus second?</p>
<p>and Which one should be used?</p>

albanD: <p>Hi,</p>
<p>The first one will prevent gradients flowing back towards <code>I_w</code>. So for the gradient computation, it will be as if this Tensor contained constant values that are independent of the weights.</p>
<blockquote>
<p>and Which one should be used?</p>
</blockquote>
<p>It depends on what you’re trying to do <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

Griffintaur1: <p><a class="mention" href="/u/alband">@albanD</a>  I want to use consistency between <code>I_w</code> and <code>I_s</code>. with shared weights model(i.e both <code>I_w</code> and <code>I_w</code> are passed to the same model).<br>
Loss is computed for both of these so I want my model to have consistent prediction across <code>I_w</code> and <code>I_s</code></p>

albanD: <p>In that case, I don’t think you want to detach as the gradients from both are important for what you want to do.</p>

Griffintaur1: <p><a class="mention" href="/u/alband">@albanD</a>  Now if I don’t use detach it will also make weights change in such a way that prediction of <code>I_w</code> is also close to <code>I_s</code>(movement towards <code>I_s</code>)  but what I want is to make predictions of <code>I_s</code> to as close as <code>I_w</code>(movement towards <code>I_w</code>). on the assumption that predictions of <code>I_w</code> are good.</p>

albanD: <p>Ok, So if you consider that <code>I_w</code> is a constant and has the right value, and you just want to make <code>I_s</code> get closer. Then you can detach <code>I_w</code> indeed.</p>
<p>But be aware that changes to <code>I_s</code> will actually impact <code>I_w</code> (even though you hide it to the autograd) and that can hinder the convergence.</p>
Dropout of whole tensor

jgdo: <p>I’m trying to port some tensorflow code to pytorch where a whole tensor is dropped out occasionally. This is done by multiplying the whole tensor with zero.</p>
<p>If I do the same in pytorch, I get errors from autograd about an invalid inplace operation:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 32, 32, 32]], which is output 0 of LeakyReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<p>Here is the original tensorfow code (“context” is the tensor):</p>
<pre><code class="lang-auto">maybe_dropout = tf.cast(tf.math.greater(tf.random.uniform([]), self._drop_out_rate),tf.float32)
context *= maybe_dropout
</code></pre>
<p>Here is my pytorch implementation:</p>
<pre><code class="lang-auto">maybe_dropout = (torch.rand([]) &gt; self._drop_out_rate).type(torch.get_default_dtype())
context *= maybe_dropout
</code></pre>
<p>I also tried to simply set the content to zero by</p>
<pre><code class="lang-auto">context[:] = 0
</code></pre>
<p>but this resulted in NaN gradients.<br>
The only thing that worked was detaching the tensor before setting it to zero:</p>
<pre><code class="lang-auto">context = context.detach()
context[:] = 0
</code></pre>
<p>but this kind of doesn’t seem right. What would be the right way to implement such a dropout behavior?</p>

albanD: <p>Hi,</p>
<p>The error mentions that you modify inplace a Tensor whole value is needed in the backward pass.<br>
In particular here, it seems that the content of <code>context</code> is needed. And so if you do <code>context *= maybe_dropout</code> (which modifies it inplace), then you can’t backward anymore.<br>
You need to make sure not to modify the original context:</p>
<pre><code class="lang-auto">context = context.clone() # Get new memory
context *= maybe_dropout # Now we can safely write inplace into it
</code></pre>
<p>Or</p>
<pre><code class="lang-auto">context = context * maybe_dropout # Use the out of place version 
</code></pre>

jgdo: <p>Thanks, that worked! Just one more question - why does</p>
<pre><code class="lang-auto">context = context * maybe_dropout
</code></pre>
<p>work but not</p>
<pre><code class="lang-auto">context = context * 0
</code></pre>
<p>?</p>

albanD: <p>How does the second “not work”? They should be doing very similar things.</p>
How to add a learnable weight for each sample when i compute loss?

Muyun99: <p>Hi everyone. I am developing a new method to learn the weight of each <strong>sample</strong>(not each batch or each class). But i get some error message, so i try to get some help from you, thanks for your reading!</p>
<p>At first, i wanna to use the loss function pytorch officially provided</p>
<pre><code class="lang-auto">nn.BCEWithLogitsLoss(weight=weight_score)
</code></pre>
<p>so i write a critetion, when i want to compute the loss wight sample_weight, i will call the forward function</p>
<pre><code class="lang-auto">class criterion(nn.Module):
    def init(self):
        super(criterion, self).init()

    def forward(self, true, pred, score):
        loss_fuc = nn.BCEWithLogitsLoss(weight=score)
        return loss_fuc(pred, true)
</code></pre>
<p>But there is a error,</p>
<pre><code class="lang-auto">The size of tensor a (321) must match the size of tensor b (16) at non-singleton dimension 3
</code></pre>
<p>i don`t konw how to solve it, i get the shape of [true, pre, score] as follows:</p>
<pre><code class="lang-auto">torch.Size([16, 1, 321, 321])
torch.Size([16, 1, 321, 321])
torch.Size([16])
# batch_size = 16
</code></pre>
<p>So the key problem is “How to add a <strong>learnable</strong> weight of each <strong>sample</strong> when i compute loss?”</p>
<p>Thanks for your reading, i really want to solve this problem and continue my experiment</p>
<p>I will appreciate if there is another better solution for this problem, thanks again</p>

bsridatta: <p>Are you sure you are printing the shapes just before nn.bce… the true seems to be of shape 16?</p>

Muyun99: <p>yes, the shape of true is torch.Size([16, 1, 321, 321]). Beacause i`m try to build a semantic segmentation application, the label is a gray image and the shape is 1x321x321.</p>
ONNX export of modification and access of model dict attribute

vadimkantorov: <p>Does export to ONNX support models like this? (that contain some modifiable state and later allow it to be retrieved. the end goal would be to export a transformer decoding beam search algorithm)</p>
<pre><code class="lang-auto">import torch
from typing import Any, Dict, List, Optional
from torch import Tensor

class Model(torch.nn.Module):
    state: Dict[str, Tensor]

    def __init__(self):
        super().__init__()
        self.state = {}

    def forward(self, x):
        self.state['abc'] = x
        return x

   def get_state(self):
      return self.state

dummy_input = torch.tensor([[3]], dtype=torch.float32)
model = torch.jit.trace_module(Model(), dict(forward=(dummy_input,), get_state=tuple() ))
print(model.code)
</code></pre>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "bug.py", line 20, in &lt;module&gt;
    model = torch.jit.trace_module(Model(), dict(forward=(dummy_input,), get_state=tuple() ))
  File "/miniconda/lib/python3.8/site-packages/torch/jit/__init__.py", line 1118, in trace_module
    _check_trace([inputs], func, check_trace_method,
  File "/miniconda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/miniconda/lib/python3.8/site-packages/torch/jit/__init__.py", line 598, in _check_trace
    check_mod = torch.jit.trace_module(
  File "/miniconda/lib/python3.8/site-packages/torch/jit/__init__.py", line 1109, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
RuntimeError: Encountering a dict at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module's inputs. Consider using a constant container instead (e.g. for `list`, use a `tuple` instead. for `dict`, use a `NamedTuple` instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.
</code></pre>
Torch.jit.trace_module: how to capture modification of passed dict?

vadimkantorov: <p>Fairseq Transformer models modify the passed incremental state dict without returning it (OTOH huggingface Transformers return it). ONNX export seems to drop these operations. Is it possible to still export these correctly?</p>
<pre><code class="lang-python">import torch
from typing import Any, Dict, List, Optional
from torch import Tensor
 
class InplaceModule(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.embeddings = torch.nn.Linear(1, 10, bias=False)
        
    def forward(self, x, state: Dict[str, Dict[str, Tensor]]):
        state['new_level_1'] = dict()
        state['new_level_1']['new_level_2'] = x
        return self.embeddings(x)
    
dummy_state = {'level_1': {'level_2': torch.tensor([[3]], dtype=torch.float32)}}
dummy_state = torch.jit.annotate(Dict[str, Dict[str, Optional[torch.Tensor]]], dummy_state)
dummy_input = torch.tensor([[3]], dtype=torch.float32)
model = torch.jit.trace_module(InplaceModule(), dict(forward=(dummy_input, dummy_state,)))
print(model.code)
</code></pre>
<pre><code class="lang-auto">def forward(self,
    input: Tensor,
    argument_2: Dict[str, Dict[str, Tensor]]) -&gt; Tensor:
  return (self.embeddings).forward(input, )
</code></pre>
Training on Graph edges in Pytorch Geometric

vbl: <p>Is it possible to train a neural net against the edges of a graph in pytorch geometric?</p>
<p>From the readthedocs page of PyG:</p>
<p><code>data.y</code> : Target to train against (may have an arbitrary shape),  <em>e.g.</em> , node-level targets of shape.</p>
<p>But can <code>data.y</code> be the edge attributes of the graph (so <code>data.edge_attr</code>)?</p>
Mimic torch.bucketsize in 1.5

Loreto_Parisi: <p>Pytorch 1.6 beta offers the new function <a href="https://pytorch.org/docs/stable/generated/torch.bucketize.html" rel="nofollow noopener">torch.bucketsize</a><br>
Is it possible to mimic this function in torch 1.5.1?</p>
<p>Thank you</p>

ptrblck: <p>You could either cherry-pick all PRs for this feature into your <code>1.5.1</code> branch, solve all merge conflicts, and rebuild PyTorch or alternatively try to copy-paste the code for this feature into custom classes and treat it as custom functions.</p>
<p>However, what’s the reason you cannot update PyTorch, which seems to be easier?</p>

Loreto_Parisi: <p>Thanks. Because Pytorch 1.6 is currently in beta.</p>

Loreto_Parisi: <p>Wops, 1.6 has just been released!</p>
Calculate mean accuracy over 100 epochs

shangguan91: <p>Dear all,</p>
<p>How to modify the code to save the test loss calculate for each epoch, then sum up loss/100 epoch.</p>
<pre><code class="lang-auto">def test_epoch(device):

    model.to(device)

    model.eval()

        

    iter_data_time = time.time()

    with Ctq(test_loader) as tq_test_loader:

        for i, data in enumerate(tq_test_loader):

            t_data = time.time() - iter_data_time

            iter_start_time = time.time()

            data.to(device)

            model.forward(data)           

            testtracker.track(model)

            tq_test_loader.set_postfix(

                **testtracker.get_metrics(),

                data_loading=float(t_data),

                iteration=float(time.time() - iter_start_time),

            )

            iter_data_time = time.time()
</code></pre>
Are torch.nn.Functional layers learnable?

marcman411: <p>I generally define my networks in the <code>__init__</code> function using <code>torch.nn</code> layers. However, if I want me convolution to be dependent on the size of the input, it might make more sense to use the <code>torch.nn.Functional</code> version. Are those parameters learnable too? If so, how does that effect the state graph, particularly when loading and saving models?</p>

jpeg729: <p>When using <code>torch.nn.Linear</code> for example, the <code>nn.Linear</code> class looks after initialising and using the parameters that it requires.<br>
When using the <code>torch.nn.Functional.linear</code> variant, it is up to you to provide the parameters on each forward pass.</p>
<p>Basically instead of</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = nn.Linear(in_features, out_features)

    def forward(self, input):
        return self.linear(input)
</code></pre>
<p>You would do this</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.weight = nn.Parameter(torch.randn(out_features, in_features))
        self.bias = nn.Parameter(torch.randn(out_features))

    def forward(self, input):
        return F.linear(input, weight, bias)
</code></pre>
<p>In the first case Model knows that it has an <code>nn.Linear</code> submodule, in the second case Model knows that it has two parameter tensors.</p>
<p>So in the first case <code>Model.parameters()</code> will list the weight and bias parameters of the <code>nn.Linear</code> submodule, in the second case <code>Model.parameters()</code> will list the weight and bias parameters defined in <strong>init</strong>.</p>
<p>Training, saving and loading can all be done in exactly the same way in both cases.</p>

luoan: <pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

    def forward(self, input):
        return F.linear(input, torch.randn(out_features, in_features))
</code></pre>
<p>In this condition, Model.parameter has no parameter, since</p>
<pre><code class="lang-auto">m  = Model ()
len(list(m.parameters())) # returns 0
</code></pre>
<p>So, in this condition, is the parameter tensor the size of [in_features, out_features] in linear function learnable?</p>
<p>In BackPropagation , can i use SGD to update parameters in linear funtion?</p>

roireshef: <p>No man. In the way you do it here a random tensor is indeed initialized, but it’s not part of the learnable set of parameters for your model because of the way you initialized it. See <a href="https://discuss.pytorch.org/t/are-torch-nn-functional-layers-learnable/13687/2" class="inline-onebox">Are torch.nn.Functional layers learnable?</a> for clarification</p>
Slow training using HDF5 dataset

samra-irshad: <p>Hi, I am training my model using HDF5 dataset (containing ~8000 images) (size:256x256). I switched to using HDF5 due to slow training speed, however, I did not notice any speed-gains. It takes 20 minutes with plain dataloader for one epoch <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> and it takes same amount of time using HDF5. I am not understanding where I could be going wrong. I have experience using tfrecords and I remember training models quite fastly on tensorflow.</p>
Procedural generation dataset: GPU vs CPU?

odats: <p>The dataset is generated by simple code (<code>for, if, random</code>). 2d matrices of 0 and 1 for input (20 by 10) and target (10 by 2):</p>
<pre><code class="lang-auto">[ [0,1,1,...],
  [1,1,1,...], ...]
</code></pre>
<p>I train LSTM (number of trainable parameters = 2778240) and it finds patterns and loss (cross entropy loss) decreases.</p>
<p>Can you please suggest how to speed up the process?</p>
<ol>
<li>What should I choose: GPU or CPU?</li>
<li>How to configure data loader: batch_size and num_workers? <code>batch_size=maximum_size</code> and <code>num_workers=4*os.cpu_count()</code>?</li>
<li>How to profile and find what is a bottleneck: data generation or training?</li>
</ol>
<p>NOTW this setup is similar to reinforcement learning when you spend time to collect data each epoch</p>

JuanFMontesinos: <p>Hi,<br>
I don’t know how complex your data are but take into account that most of the pytorch gens (torch.rand for example) can directly allocate the data into the gpu.</p>
<p>That way you save the time required to move data from cpu to gpu.</p>
<p>You may find this useful <a href="https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964/2" class="inline-onebox">How to measure time in PyTorch</a><br>
This way you can profile cpu vs gpu</p>

odats: <p>Do you mean: <code>input = torch.randint(0,2, (20,10), device="cuda")</code>?<br>
My dataset is generated by CPU code: 2d matrices of 0 and 1 for input (20 by 10) and target (10 by 2). There is some logic behind: for, if,… Should I move this to GPU? I read that for simple operations CPU if better.</p>

JuanFMontesinos: <p>I mean, I cannot really reply you, just told you how to profile you to see what’s better. But for example if you generate tons of data but you have to allocate it, it’s also time demanding.</p>
Softmax function

sandeep1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e97cb9dd7c3b6ebcad93b40c59b094c51abba6fa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e97cb9dd7c3b6ebcad93b40c59b094c51abba6fa" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e97cb9dd7c3b6ebcad93b40c59b094c51abba6fa_2_690x76.jpeg" alt="image" data-base62-sha1="xjwtrziVUgcp1VeLDmpUFuHGaoy" width="690" height="76" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e97cb9dd7c3b6ebcad93b40c59b094c51abba6fa_2_690x76.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e97cb9dd7c3b6ebcad93b40c59b094c51abba6fa_2_1035x114.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e97cb9dd7c3b6ebcad93b40c59b094c51abba6fa_2_1380x152.jpeg 2x" data-dominant-color="E3E3E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1884×210 61.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>EXPLAINATION:<br>
<code>softmax</code>  that performs the softmax calculation and returns probability distributions for each example in the batch. Note that you’ll need to pay attention to the shapes when doing this. If you have a tensor  <code>a</code>  with shape  <code>(64, 10)</code>  and a tensor  <code>b</code>  with shape  <code>(64,)</code> , doing  <code>a/b</code>  will give you an error because PyTorch will try to do the division across the columns (called broadcasting) but you’ll get a size mismatch. The way to think about this is for each of the 64 examples, you only want to divide by one value, the sum in the denominator. So you need  <code>b</code>  to have a shape of  <code>(64, 1)</code> . This way PyTorch will divide the 10 values in each row of  <code>a</code>  by the one value in each row of  <code>b</code> . Pay attention to how you take the sum as well. You’ll need to define the  <code>dim</code>  keyword in  <code>torch.sum</code> . Setting  <code>dim=0</code>  takes the sum across the rows while  <code>dim=1</code>  takes the sum across the columns.</p>
<p>DOUBT:<br>
64X10 is shape of a and 64 is dimension of b then why a/b is defined then why when 64x1 is dimension of b then it can be divided and what will be shape of output.</p>
<p>Anything resource supporting this concept to understand will also be apprictiated.</p>
<p>Thanks</p>

mariosasko: <p>You need to understand broadcasting semantics for that. It can be a bit tricky in the beginning to wrap your head around it. A good starting point are the <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" rel="nofollow noopener">docs</a> on this topic (especially the first section).</p>