akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch C++ API Documentations

username: leimao
cooked: <p>I found many of the functions or C++ methods were not documented in detail and I would have to guess what each of the methods is doing. For example, the <code>at::Tensor</code> class:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/cppdocs/api/classat_1_1_tensor.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/cppdocs/api/classat_1_1_tensor.html" target="_blank" rel="noopener nofollow ugc">Class Tensor — PyTorch master documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I wonder if there is any plan to add a brief message to each of the methods, or where I could find these brief messages. Thank you.

username: alanzhai219
cooked: <p>You can refer to python API and they are same. BTW, if you just try to deploy, just flow the user guide and  it’s not necessary to dig into source code since lower code is always changing.</p>

username: leimao
cooked: <p><a class="mention" href="/u/alanzhai219">@alanzhai219</a> Thank you. Then a reference link to the Python API will be beneficial. The function names from the Python API and C++ API might not always be the same I think.</p>
Microsofct C++ exception: torch::jit::ErrorReport at memory location 0x0000

username: omeryasar
cooked: <p>Hi there, I am trying to migrate faster rcnn implementation of torchvision to c++ to do that I simply run these lines of code to obtain ‘pt’ file.</p>
<p>model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)<br>
model.eval()<br>
script_model = torch.jit.script(model)</p>
<p>This part is work as intended but after when I try to upload this produced model in c++ I get the Microsofct C++ exception: torch::jit::ErrorReport at memory location 0x0000… error. Any ideas about what should i do?</p>

username: ptrblck
cooked: <p>Could you post the complete error message, please?</p>

username: omeryasar
cooked: <p>Hi, the whole message ;<br>
Unhandled exception at 0x00007FF9013A3C58 in TorchDeneme2.exe: Microsoft C++ exception: torch::jit::ErrorReport at memory location 0x000000A7B4199900.</p>

username: ptrblck
cooked: <p>Thanks for the update. I’m not familiar enough with Windows, unfortunately, and the error message doesn’t give a lot of information, so we might need to wait for a Windows expert.</p>
Softmax and CrossEntropyLoss

username: Kla
cooked: <p>Hi, when I use nn.CrossEntrypyLoss(), is it necessary to define the nn.Softmax as the last Layer by Forward or is it in CrossEntropyLoss included?</p>
<p>For example: Which should be the correct version:<br>
def forward(self, x):<br>
x = F.relu(self.conv1(x))<br>
x = F.relu(self.conv2(x))<br>
x = F.relu(self.conv3(x))<br>
x = self.avgpool(x)<br>
x = self.fc1(x)<br>
x = nn.Softmax(x, dim=1)<br>
return x</p>
<p>or:</p>
<p>def forward(self, x):<br>
x = F.relu(self.conv1(x))<br>
x = F.relu(self.conv2(x))<br>
x = F.relu(self.conv3(x))<br>
x = self.avgpool(x)<br>
x = self.fc1(x)<br>
return x</p>

username: Abhilash_Srivastava
cooked: <aside class="quote no-group" data-username="Kla" data-post="1" data-topic="100826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kla/48/23505_2.png" class="avatar"> Kla:</div>
<blockquote>
<p>when I use nn.CrossEntrypyLoss(), is it necessary to define the nn.Softmax as the last Layer by Forward or is it in CrossEntropyLoss included?</p>
</blockquote>
</aside>
<p><code>nn.Softmax</code> is not required in this case. Your second example should work fine.</p>
About the conversion of _iter_() option for make_data_loader to C++

username: Chen0729
cooked: <p>Hi, ALL</p>
<p>I have a question about the conversion of <em>iter</em>() option for make_data_loader in conversion to C++</p>
<p>the python code is:<br>
unlabel_loader2 = DataLoader(self.unlabeled, batch_size = self.args.batch_size, shuffle=True, drop_last=True, num_workers = 4).<strong>iter</strong>()</p>
<p>In C++ code,<br>
I write it as:<br>
auto unlabel_loader2 = torch::data::make_data_loader(<br>
unlabeled.map(torch::data::transforms::Normalize&lt;&gt;({0.5, 0.5, 0.5}, {0.5, 0.5, 0.5}))<br>
.map(torch::data::transforms::Stack&lt;&gt;()),<br>
torch::data::DataLoaderOptions().batch_size(args.batch_size).workers(4).drop_last(<br>
true))<br>
But I donot have the option of iter in C++<br>
Any suggestion is welcome.</p>
<p>Thanks.</p>
Using specific loss for an optimizer

username: ChrisLiu2
cooked: <p>When we have multiple optimizers and multiple losses, is it possible to specify the which loss to backward() to? Something like a wrapper:</p>
<pre><code class="lang-auto">with opt_1 as optimizer:
    loss.backward()
</code></pre>

username: JuanFMontesinos
cooked: <p>losses backward depending on the graph.<br>
Graph is done at a tensor level. It is totally different from optimizers which just update weights.<br>
Therefore it’s not really clear what are you trying to do.</p>

username: ChrisLiu2
cooked: <p>I see. I considered just freezing part of the model and using one loss for that part, and a second loss for another part. But my concern is that after I backward the first loss, since the parameter has already been changed, will that affect the second backward call?</p>
About the type of dataset in the function input in libtorch

username: Chen0729
cooked: <p>Hi, All</p>
<p>I have a question about the type of dataset defined in function input in libtorch</p>
<p>in pytorch it is<br>
def <strong>init</strong>(self, labeled, unlabeled):<br>
self.unlabeled = unlabeled</p>
<p>def MnistUnlabel():<br>
raw_dataset = datasets.MNIST(’…/data’, train=True, download=True,<br>
transform=transforms.Compose([<br>
transforms.ToTensor(),<br>
]))<br>
return raw_dataset</p>
<p>May I ask in C++ libtorch how shall I define the type of the dataset unlabel, I tried with torch:data:datasets&amp; unlabel but it is incorrect.</p>
<p>Many thanks.</p>
What do the numbers after the grad_fn mean?

username: Torch_with_kimchi
cooked: <p>when you print a tensor</p>
<pre><code class="lang-auto">b=a+2
print(b)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor([[3., 3.],
        [3., 3.]], grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>grad_fn is appended with a function that calculates the derivative value and a number.<br>
However, there are times when the number is 0 or 1, so I don’t know what the standard is.<br>
What do the numbers after the function mean?</p>

username: albanD
cooked: <p>Hi,</p>
<p>This number is used because we need different functions for different overloads of the forward function: add(Tensor, Tensor) or <code>add(Tensor, Scalar)</code> (not sure if you can actually call this one from python) for example where the first will get <code>AddBackward0</code> and the second <code>AddBackward1</code>.<br>
But in general, you can ignore it.</p>
Update the Tensor by element's indice, in forward function

username: Abitalo
cooked: <p>I tried to define a customized layer for transforming <strong>scalar</strong> inputs into <strong>2d-matrix</strong>, while the transformation rules are based on the <strong>elements’ indices</strong>.</p>
<p>Following is the code:</p>
<pre><code class="lang-python">class CustomLayer(nn.Module):
    def __init__(self)
        ...
    
    def forward(thresholds):
        # @threshold: torch.uint8
        #             &lt;batch_size, 2&gt;.
        #             Thresholds  of x and y directions.

        canvas = torch.zeros((batch_size, height, width), dtype=torch.uint8)

        for i in range(64):
            for j in range(64):
                a = torch.relu(i-thresholds[:,0])
                b = torch.relu(i-thresholds[:,1])
                canvas[:,i,j] = a+b

        return canvas
</code></pre>
<p>I’m wondering if there’s any way to get the indices of the element in the <code>canvas</code> Tensor, so that I could do things like:</p>
<pre><code class="lang-python"># indice() does not really exists.
canvas.apply(
       lambda e: 
           torch.relu(e.indice()[0]- thresholds[0]) + \
           torch.relu(e.indice()[1]- thresholds[1]))
</code></pre>
<p>Is there any way to do so?<br>
(btw., can the <strong>for loop</strong> in forward function be handled well in the back propagation phase?)</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>The for-loop can be handled, but if the input is an integral type, then you won’t be able to get gradients for it (as gradients are only for continuous inputs).</p>
Sync parameter server implementation

username: sakh251
cooked: <p>As I understood, the Tutorial for Parameter server based on the RPC framework is a special implementation based on different assumptions.<br>
1- The data should be sent to the parameter server (for a large dataset is it possible???)<br>
2- it Async<br>
3- distributed auto grad has been used<br>
am I correct?</p>
<p>I am wondering is there any way to implement a sync parameter server with the RPC framework without moving data to the parameter server?  I mean just use local grad and for example, model averaging on parameter server. any exist?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/sakh251">@sakh251</a></p>
<blockquote>
<p>1- The data should be sent to the parameter server (for a large dataset is it possible???)<br>
2- it Async<br>
3- distributed auto grad has been used<br>
am I correct?</p>
</blockquote>
<p>Yep, you are correct.</p>
<blockquote>
<p>I am wondering is there any way to implement a sync parameter server with the RPC framework without moving data to the parameter server? I mean just use local grad and for example, model averaging on parameter server. any exist?</p>
</blockquote>
<p>Yep, this is possible. Check out <a href="https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-updating-parameter-server" rel="noopener nofollow ugc">this tutorial</a>. This is not exactly what you are looking for, as the trainers send gradients to the parameter server and the parameter server batch updates the model. But it shouldn’t be too far from your requirements. Editing the batch-processing <code>batch_update_size</code> and then send model params instead of gradients might do the job.</p>

username: sakh251
cooked: <p>Thanks a lot, It is what I was looking.</p>
Gradient Aggregation in PyTorch

username: Ohm
cooked: <p>a simple MLP model was considered and the initial model() shared between workers.<br>
Conventionally, each worker train model based on its own dataset and shared the resulted model. By averaging models from different workers, the new model achieved for the next round of algorithms.<br>
How efficiently share <strong>gradients</strong> of different workers and <strong>averaged them</strong> and use the “<strong>optimizer .step()</strong>” to update the initial model(initial model shared among workers)?</p>

username: mrshenli
cooked: <p>hey <a class="mention" href="/u/ohm">@Ohm</a>, if you are using <code>DistributedDataParallel</code>, you can try the <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync" rel="noopener nofollow ugc"><code>no_sync</code></a> context manager. For example, you can wrap local training iterations with <code>no_sync</code>. When you need to do gradient averaging, just run one fw-bw out of the <code>no_sync</code> context, and DDP should be able to take care of the gradient synchronization.</p>
<p>Another option would be building your application using <a href="https://pytorch.org/docs/master/rpc.html" rel="noopener nofollow ugc"><code>torch.distributed.rpc</code></a> and then use a parameter server to sync models. See <a href="https://pytorch.org/tutorials/intermediate/rpc_async_execution.html#batch-updating-parameter-server" rel="noopener nofollow ugc">this tutorial</a>.</p>
<p>If all parameters are dense, the DDP solution should be more efficient. If there are sparse parameters, the parameter server solution might be faster.</p>
How to overlap coping data to/from the GPU with performing computations on the GPU?

username: lzypku
cooked: <p>Hi, I am new to pytorch.</p>
<p>I want to know how to overlap copying data to/from the GPU, with performing computations on the GPU ?</p>
<p>Thanks very much!</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use the <code>non_blocking</code> argument to either the <code>copy_</code> or <code>to()</code> or <code>cuda()</code> functions to be able to do that.</p>
Update network after differentiation with autograd.grad()

username: MSS
cooked: <p>Hi,</p>
<p>I want to explicitly learn the gradients of a neural networks output w.r.t. its input.<br>
I coded a graph neural network, but for simplicity, this ffnn is not updating either and I guess its due to the autograd.grad() function.</p>
<pre><code class="lang-auto">import torch 

class Feedforward(torch.nn.Module):
        def __init__(self, input_size, hidden_size):
            super(Feedforward, self).__init__()
            self.input_size = input_size
            self.hidden_size  = hidden_size
            self.fc1 = torch.nn.Linear(self.input_size, self.hidden_size)
            self.relu = torch.nn.ReLU()
            self.fc2 = torch.nn.Linear(self.hidden_size, 1)
     
        
        def forward(self, x):
            hidden = self.fc1(x)
            relu = self.relu(hidden)
            output = self.fc2(relu)
            output = output.sum()
            output =torch.autograd.grad(outputs=output, inputs=x, retain_graph=True, create_graph=True)
            return output[0]

test_input = torch.rand((10,3),requires_grad=True)
test_output = torch.rand((10,3))


model = Feedforward(3,10)
optim = torch.optim.Adam(model.parameters())
optim.zero_grad()
loss_fn = torch.nn.L1Loss()
model.train()
out = model(test_input)
loss = loss_fn(out, test_output)
loss.backward()
optim.step() # if you break here and investigate the gradients
             # of the FFNN, the gradients will be 0
</code></pre>
<p>How are the gradients of this neural network updated?</p>

username: MSS
cooked: <p>Update: The layers receive gradients. My problem has to do something with My GNN.</p>
<p>Maybe the integration in pytorch geometric is breaking</p>
Shared model instances in multiple flask api

username: Tuyen_Vo_Quang
cooked: <p>I have a sample Flask app like this:</p>
<pre><code class="lang-auto">from flask import Flask
app = Flask(__name__)
model = pytorch_model_instance


@app.route('/task1')
def do_task1():
    pred = model.predict()
    return pred

@app.route('/task2')
def do_task2():
    pred = model.predict()
    return pred
</code></pre>
<p>The above 2 endpoints could use the model simultaneously for prediction, my question is if i define a single model for these 2 endpoints, are there any problems if both of them access the model instance at the same time for prediction?</p>
Implement dropout layer nn.Sequential

username: Vish2020
cooked: <p>I am trying to implement a Dropout layer using pytorch as follows:</p>
<pre><code class="lang-auto">class DropoutLayer(nn.Module):
    def __init__(self, p):
        super().__init__()
        self.p = p

    def forward(self, input):
        if self.training:
            u1 = (np.random.rand(*input.shape)&lt;self.p) / self.p
            u1 *= u1
            return u1
        else:
            input *= self.p
</code></pre>
<p>And then calling a simple NN.sequential:</p>
<pre><code class="lang-auto">model = nn.Sequential(nn.Linear(input_size,num_classes), DropoutLayer(.7), nn.Flatten())

opt = torch.optim.Adam(model.parameters(), lr=0.005)
train(model, opt, 5) #train(model, optimizer, epochs #)
</code></pre>
<p>But I’m getting the following error:</p>
<pre><code class="lang-auto">TypeError: flatten() takes at most 1 argument (2 given)
</code></pre>
<p>Not sure what I’m doing wrong.  Still new to pytorch.  Thanks.</p>

username: ptrblck
cooked: <p>During the training your custom dropout layer would only return the scaled drop mask without the input, while during evaluation nothing would be returned.</p>
<p>Could you post the complete stack trace for the flatten error?</p>

username: Vish2020
cooked: <p>Hi the error has been resolved, I simply needed to call nn.Flatten() first on the multidimensional array in order to convert into a 2d array after which dropout is called.  Thx.</p>
Hyperparameter in Checkpoint

username: gnadaf
cooked: <p>Is it required or necessary to save hyperparameters in the checkpoint?</p>

username: ptrblck
cooked: <p>The recommended way of serializing a PyTorch model is to store its <code>state_dict</code>, which contains all parameters and buffers. While this should be enough to recreate the model, load the <code>state_dict</code>, and create new predictions, you might want to e.g. continue the training of this model.<br>
In this case, it might be useful to store some hyperparameters (which are not already included in e.g. the optimizer’s <code>state_dict</code>) to properly restore the training routine.</p>

username: gnadaf
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="100678">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>buffe</p>
</blockquote>
</aside>
<p>If we save the following dict in a checkpoint, I believe we save all parameters and hyperparameters.<br>
Is it correct?</p>
<p>model.state_dict()<br>
optimizer.state_dict()<br>
scheduler.state_dict()</p>
Convolutional computing failed on Centos 7

username: 111332
cooked: <p>Hello, all.</p>
<p>I meet a strange problem. The codes below work successfully on Win10 ( python 3.7 and PyTorch 1.6). However, when running the same code on Centos 7 (python 3.7 and PyTorch 1.5), I got an error.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; import torch.nn as nn
&gt;&gt;&gt; cnn = nn.Conv2d(1, 100, (3, 300))
&gt;&gt;&gt; input = torch.randn(64, 1, 16, 300)
&gt;&gt;&gt; output = cnn(input)
</code></pre>
<p>The detail error message as follow:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/run/anaconda3/envs/DL/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/run/anaconda3/envs/DL/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 349, in forward
    return self._conv_forward(input, self.weight)
  File "/run/anaconda3/envs/DL/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 346, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: NNPACK SpatialConvolution_updateOutput failed
</code></pre>
<p>Waiting for your kind help!</p>
Calculate covariance of torch tensor (2d feature map)

username: spadel
cooked: <p>I have a torch tensor with shape (batch_size, number_maps, x_val, y_val). The tensor is normalized with a sigmoid function, so within range <code>[0, 1]</code> . I want to find the covariance for each map, so I want to have a tensor with shape (batch_size, number_maps, 2, 2). As far as I know, there is no <code>torch.cov()</code>  function as in numpy. How can I efficiently calculate the covariance without converting it to numpy?</p>
<p>I tried the following, but I’m pretty sure it’s not correct:</p>
<pre><code class="lang-auto">def get_covariance(tensor):
    bn, nk, w, h = tensor.shape
    tensor_reshape = tensor.reshape(bn, nk, 2, -1)
    x = tensor_reshape[:, :, 0, :]
    y = tensor_reshape[:, :, 1, :]
    mean_x = torch.mean(x, dim=2).unsqueeze(-1)
    mean_y = torch.mean(y, dim=2).unsqueeze(-1)
    
    xx = torch.sum((x - mean_x) * (x - mean_x), dim=2).unsqueeze(-1) / (h*w - 1)
    xy = torch.sum((x - mean_x) * (y - mean_y), dim=2).unsqueeze(-1) / (h*w - 1)
    yx = xy
    yy = torch.sum((y - mean_y) * (y - mean_y), dim=2).unsqueeze(-1) / (h*w - 1)
    
    cov = torch.cat((xx, xy, yx, yy), dim=2)
    cov = cov.reshape(bn, nk, 2, 2)

    return cov
</code></pre>
Two different sequence padding methods / dynamic nn.Linear

username: lisyuan
cooked: <p>My model will input a batch of sequence to <code>nn.Transformer</code> and the output of transformer with be feeded into <code>nn.Linear</code>. The input sequence has different length before feeding it into <code>nn.Transformer</code>, so I pad the sequence to the same length in every batch adatively using <code>collate_fn</code> in <code>dataloader </code></p>
<p>For example:</p>
<ul>
<li>batch 1:  max length of sequence in this batch is 10, padding 0 to each sequence</li>
<li>batch 2:  max length of sequence in this batch is 12, padding 0 to each sequence</li>
<li>batch 3:  max length of sequence in this batch is 15, padding 0 to each sequence</li>
</ul>
<p>Now, I have a problem because the output shape of Transformer is <em>[seq_len, batchsize, dinput]</em>, so the input dimension of <code>nn.Linear</code> is dynamical because of the sequence length is different.<br>
I have seen some posts mentioned that it can use the <code>nn.AdaptiveAvgPool</code> to solve this problem. Now, I have two solutions and question:</p>
<ul>
<li>
<p>Transpose the output of transformer into <em>[batchsize, dinput, seq_len]</em> using <code>nn.AdaptiveAvgPool1d</code> to make shape into <em>[batchsize, dinput, fix_num]</em>, and reshape to <em>[batchsize, -1]</em> to feed into <code>nn.Linear</code>. <strong>Is this make sense? The sequence length are different but we fix it into a fix number?</strong></p>
</li>
<li>
<p>Set a fixing maximum length, ex: 30, some sequence length is up to 25-30, but most of the sequence length are in length 5-10. <strong>The input data will be most of 0 (padding value). It will contain too much useless information!</strong></p>
</li>
</ul>
<p>Does anyone compare of these two padding methods or any other good suggestion?</p>
Export to Onnx error: "_thnn_fused_lstm_cell"

username: Frank_Lu
cooked: <p>Hi guys:<br>
I have build a seq2seq model using nn.LSTMCELL in it. And I want to export a trained one to onnx form. However, when I do this, I have an error says:</p>
<pre><code class="lang-auto">UserWarning: ONNX export failed on ATen operator _thnn_fused_lstm_cell because torch.onnx.symbolic_opset9._thnn_fused_lstm_cell does not exist
  .format(op_name, opset_version, op_name))
</code></pre>
<p>I think this maybe due to the opset nort supporting nn.LSTMCELL.<br>
Any one could help and share some advice?<br>
Thank you！</p>

username: hengye_liu
cooked: <p>hello have you solved this error?I got same error.</p>
How to get the Cumulative Distribution Function for Gaussian Distribution

username: KaiHoo
cooked: <p>Hello, I would like to use the Cumulative Distribution Function  for Gaussian Distribution.</p>
<p>I know it can be get it as:</p>
<p><code>fn = lambda x : torch.nn.GELU()(x) / x</code></p>
<p>But I am afraid it is not numerical stable. Any other suggestions? Thanks!</p>

username: googlebot
cooked: <p>distributions/normal.py has it:</p>
<pre><code class="lang-auto">        return 0.5 * (1 + torch.erf((value - self.loc) * self.scale.reciprocal() / math.sqrt(2)))
</code></pre>
Trying to backward through the graph while only sharing an input tensor

username: malicd
cooked: <p>I am trying to organize CycleGAN code in my own fashion and wile running (disclaimer: this is just a snipped but I hope it is self-explanatory)</p>
<pre><code class="lang-auto">def optimize_parameters(self):
    real_k = Variable(torch.rand([1, 128, 128])).cuda()
    real_w = Variable(torch.rand([1, 128, 128])).cuda()

    fake_k = self.generator_K(real_w)
    fake_w = self.generator_W(real_k)
    
    # G
    set_requires_grad([self.discriminator_K, self.discriminator_W], False)
    self.optimizer_generator.zero_grad()

    gen_K_loss = self.gan_loss(self.discriminator_K(fake_k), True)
    gen_W_loss = self.gan_loss(self.discriminator_W(fake_w), True)
    gen_loss = gen_K_loss + gen_W_loss
    gen_loss.backward()

    self.optimizer_generator.step()
    set_requires_grad([self.discriminator_K, self.discriminator_W], True)

    # D
    self.optimizer_discriminator.zero_grad()

    dis_K_loss = self.gan_loss(self.discriminator_K(real_k), True)

    dis_K_loss.backward() # &lt;&lt;&lt; RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
    self.optimizer_discriminator.step()
</code></pre>
<p>I get <code>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</code> and I can not understand why is this happening. To my understanding, discriminator and generator share just the input <code>real_k</code> and no discriminator graph should be created whilst optimizing the generator. However, this somehow happens.</p>
<p>Could someone help me understand this? Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>This is surprising indeed given the code you shared.<br>
Are you sure that you don’t save any state that you re-use in the gan_loss or the discriminator?</p>

username: malicd
cooked: <p>Hi, thanks for the replay! As you said it, I’ve reused state (returned reference of <code>real_[w|k]</code>) within the generator which forced both generator and discriminator to be traverse during backprop. Anyhow, I just find out this stupid bug and for the future references: be sure to double-check are you returning copy of a tensor or a reference.</p>
How to find out version when certain `pytorch` feature first became available

username: ddrevicky
cooked: <p>Hi, how can I find when a particular pytorch feature was added?<br>
For example, I want to know in which version <code>torch.bucketize</code> first became available in the API so that if we use it in our code, what version restriction we need to apply. What is the easiest way to find this out?</p>

username: ptrblck
cooked: <p>You could use the <a href="https://github.com/pytorch/pytorch/releases">release notes</a> and search for this particular method (which can be directly found as it was apparently added in <code>1.6</code> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).<br>
Alternatively, you could also check out the <a href="https://pytorch.org/docs/stable/index.html">docs</a> and search for this method in different versions (top left corner).<br>
If that also doesn’t help, search through the source code (<code>grep -r</code> or in GitHub) and use <code>git blame</code> to find the PR, where it was added.</p>
Trace a model and export its nodes, edges and [hyper]parameters

username: rashedkoutayni
cooked: <p>Hi,<br>
I have a simple PyTorch model that has a few layers, and I want to trace its graph and export it as a set of nodes and edges (as a traditional graph). Furthermore, I want to export the hyper-parameters for each layers (such as such as a Conv2D layer’s number of input channels and output channels etc…) as well as the parameters (such as a Conv2D layer’s kernels and bias).</p>
<ul>
<li>Is it possible to do this using torch.jit.trace (or  torch.jit._get_trace_graph) ?</li>
<li>Is there any other way I can trace the graph other than torch.jit (for more complex models such as ResNet or DenseNet) ?</li>
</ul>
<p>Thank you !</p>
Finetuning Loss is Plateauing very fast

username: skhu17
cooked: <p>Hi, I am trying to finetune a github’s (<a href="https://github.com/ericsujw/InstColorization/blob/master/README_TRAIN.md" rel="noopener nofollow ugc">https://github.com/ericsujw/InstColorization/blob/master/README_TRAIN.md</a>) image colorization pretrained model. The problem is that the training loss quickly (during the first epoch) plateaus around 0.17-0.19 depending on the dataset. There is not much colorization that happens, even inputting the images, during the training, into the model outputs poor results (not much changes in the LAB color space).</p>
<p>I was able to finetune a model over 50 epochs, but it was only able to colorize sky (blue) and grass (green). I am not too sure why my model does not learn and any input is much appreciated. I have introduced a LeakyRelu and tried increasing and decreasing the learning rate.</p>
<p>Cheers!</p>
Create subset of FMNIST data set with equal number of items in classes

username: Shashank.Mishra
cooked: <p>I want to split FMNIST data set into different sub-datasets containing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<ol>
<li>Containing random 500 samples for each class (so in total 500*no of class).</li>
<li>Containing random 100 samples for each class.<br>
and so on…<br>
Is there a direct way of doing this. I tried with subset but it is resulting in imbalanced dataset.</li>
</ol>

username: ptrblck
cooked: <p>You could get the target tensor and create split indices using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"><code>sklearn.model_selection.train_test_split</code></a>, which accepts the <code>stratify</code> argument to create balanced splits. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Once you have the indices, you could use <code>Subset</code>s to create the datasets.</p>
Padded values contradict in documentation and code

username: Gdhzyz
cooked: <p>Pytorch online documentation says “<a href="https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html#maxpool2d" rel="noopener nofollow ugc">if <code>padding</code>  is non-zero, then the input is implicitly zero-padded on both sides for <code>padding</code>  number of points</a>”.<br>
But it is not true in my experiment.<br>
The code is here:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

pool = nn.MaxPool2d(3, stride=1, padding=1)
x = torch.ones((1, 1, 3, 3))
x *= -1
print('=========== x ===========')
print(x)

y = pool(x)
print('=========== y ===========')
print(y)
</code></pre>
<p>The outputs are:<br>
=========== x ===========<br>
tensor([[[[-1., -1., -1.],<br>
[-1., -1., -1.],<br>
[-1., -1., -1.]]]])<br>
=========== y ===========<br>
tensor([[[[-1., -1., -1.],<br>
[-1., -1., -1.],<br>
[-1., -1., -1.]]]])</p>
<p>If it is zero-padded, then the outputs should contain zeros since 0 &gt; -1.</p>
<p>So which is correct, the documentation or the code?</p>
Load settings from config file and use setattr for torch modules?

username: BarCodeReader
cooked: <p>so i just want to create a config file, say, <strong>optimizer.config</strong>, like:</p>
<pre><code class="lang-auto">name: SGD
lr: 0.1
momentum:0.9
</code></pre>
<p>and i initialize my optimizer as below:</p>
<pre><code class="lang-auto">optimizer = getattr(torch.optim, config['name'])(filter(lambda p: p.requires_grad, model.parameters()),
                                                 lr=config.getfloat('lr')
        )
</code></pre>
<p>i don’t want to directly set the flag</p>
<pre><code class="lang-auto">momentum=config.getfloat('momentum')
</code></pre>
<p>because i may have more parameters and i want to put them in a loop to load.<br>
but when i try to assign value to “momentum”, I found out seems i cannot use setattr:</p>
<pre><code class="lang-auto">setattr(optimizer, 'momentum', 0.9)
</code></pre>
<p>it does not work…is there any way that i can load settings from a config file for torch modules?</p>
<p>really thanks</p>

username: AnimeshSinha1309
cooked: <p>Can you please mention if there is an error, or what is the behavior.<br>
Also, please try:</p>
<pre><code class="lang-auto">momentum = getattr(optimizer, 'momentum')
momentum = 0.9
</code></pre>
<p>I am new to PyTorch, but I presume if the <code>__eq__</code> operator is custom defined by PyTorch to do more than just assigning, if it also edits some computation graph elements, this should work.</p>

username: BarCodeReader
cooked: <p>no error, just cannot assign the value.<br>
your solution is also not working.</p>
Is torch.requires_grad required?

username: Debasmit_Das
cooked: <p>I have been working on fine-tuning and freezing. Systematically, the method is to set torch.requires_grad = False for those parameters which are to be frozen. But is it required other than for computational reasons ? As a toy example if I want to freeze l1 and train l2 the following code should also work, right ? I don’t need to set torch.requires_grad =  False, right ?</p>
<pre><code class="lang-auto">import torch

X = torch.rand(10,3)



l1 = torch.nn.Linear(3,2)
rel = torch.nn.ReLU()
l2 = torch.nn.Linear(2,1)



optim1 = torch.optim.Adam(l1.parameters(), lr = 0.1)
optim2 = torch.optim.Adam(l2.parameters(), lr = 0.1)

#print("Initial parameter: ", l2.weight)

for i in range(10):

  optim1.zero_grad()
  optim2.zero_grad()

  y=l2(rel(l1(X)))

  loss = torch.sum(y*y)

  loss.backward()

  optim2.step()
  

</code></pre>

username: ptrblck
cooked: <p>Yes, <code>optim2</code> will only update the parameters of <code>l2</code>.<br>
The gradients in the parameters of <code>l1</code> will be computed, assigned, and replaced with zeros again.</p>

username: Debasmit_Das
cooked: <p>So, no need to set torch.requires_grad = False for the parameters of l1, right ?</p>

username: ptrblck
cooked: <p>You don’t need to set this attribute if you are using the posted code snippet and <code>l1</code> won’t be updated.<br>
However, I would recommend to explicitly set it in case your code tries to use the gradients of the models in any way, which might be hard to debug later on.</p>
Interpreting the output of pytorch profiler

username: hexpheus
cooked: <p>Hi. Could someone please help understand these outputs by torch.utils.bottleneck?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9c4f81d3c1f0fc3386182bda8445cf1270edadb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f9c4f81d3c1f0fc3386182bda8445cf1270edadb" title="photo_2020-10-26_21-26-01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9c4f81d3c1f0fc3386182bda8445cf1270edadb_2_690x385.jpeg" alt="photo_2020-10-26_21-26-01" data-base62-sha1="zDyTtdCLJximW9Q5hSsOMKTr9sn" width="690" height="385" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9c4f81d3c1f0fc3386182bda8445cf1270edadb_2_690x385.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9c4f81d3c1f0fc3386182bda8445cf1270edadb_2_1035x577.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9c4f81d3c1f0fc3386182bda8445cf1270edadb.jpeg 2x" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">photo_2020-10-26_21-26-01</span><span class="informations">1280×716 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Bottleneck DataParallel

username: MehdiZouitine
cooked: <p>Hey !</p>
<p>I have 4 gpu in my machine. I trained a model with 1 gpu then with 4 gpu.<br>
But the training in 4 gpu is too long.</p>
<p>I guess its because I have to load my model on each gpu (that is the things that make this so slow).</p>
<p>Someone have a solution because loading my models in all the gpu seems to slow.</p>
<p>Thanks</p>

username: seungjun
cooked: <p>Hi <a class="mention" href="/u/mehdizouitine">@MehdiZouitine</a>,</p>
<p>DataParallel has a speed issue due to python GIL and its parallelization. As you mentioned, DataParallel has to recast the parameters from gpu 0 to gpu 1~3 at every forward computation.</p>
<p>You can try DistributedDataParallel which is faster. It is the officially recommended multi-gpu scheme.<br>
Here’s the tutorial for DDP.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/dist_overview.html" target="_blank" rel="noopener nofollow ugc">PyTorch Distributed Overview — PyTorch Tutorials 1.6.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Also, you can refer to a complete training/evaluation <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch" rel="noopener nofollow ugc">example</a>.</p>
WeightNorm differentiation

username: vadimkantorov
cooked: <p>It seems that in OpenAI implementation, the L2-normalization of the weight computation is being differentiated through: <a href="https://github.com/openai/weightnorm/blob/master/tensorflow/nn.py#L200" rel="noopener nofollow ugc">https://github.com/openai/weightnorm/blob/master/tensorflow/nn.py#L200</a></p>
<p>Is it being differentiated through in PyTorch? <a href="https://github.com/pytorch/pytorch/blob/852f8526c52190125446adc9a6ecbcc28fb66182/aten/src/ATen/native/WeightNorm.cpp#L58" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/852f8526c52190125446adc9a6ecbcc28fb66182/aten/src/ATen/native/WeightNorm.cpp#L58</a></p>
<p>It seems so, but just to be sure.</p>
How to add num_of_worker and data shuffle option in data loader in C++

username: Chen0729
cooked: <p>The python code is<br>
Dataloader（dataset，bathsize，shuffle=true, drop_last=true, num_of_worker=3)</p>
<p>May I ask about how to convert it into C++ please ?<br>
I checked make data loader does not have num of worker and datashuffle options ?</p>
<p>Thanks.</p>
Gradient with respect to weight function

username: msr
cooked: <p>I want to find the gradient with respect fo a function w, where w is an MLP that selects part of my input data to use. Then I use another neural network for normal training. The psuedocode would be something like:</p>
<pre><code class="lang-auto">x_train, y_train =w(training_data)
output = model(x_train)
update(model)
update(w)
</code></pre>
<p>How do I make sure that w is included in propagation?</p>
Caculate loss with selected values in the output only cause zeroing of unselected region

username: peepeepoopoo
cooked: <p>I’m observing something That I’m not sure what to make of: So the output is a HxW matrix, and the loss is calculated by a selected number of values within this matrix. I found that as the training went on the unselected region would just go to zero. My network contains mostly convolutional layers,  so I’d assume the loss of the selected pixel would propagate to enough locations to be covered.</p>
<p>Right now I’m assuming I might not be propagating the loss through the corresponding pixel locations that are not used during the calculation of loss. Is my assumption correct? Is there a way in PyTorch to set the backprop for all values in the matrix and not just the selected ones?</p>

username: YazanShannak
cooked: <p>Here’s my intuition<br>
Based on the fundamental theory of neural networks and back propagation, your network is trying to meet the objective defined by the loss function, which is to generate an output for the selected region as close to as the target, therefore the weights are updated to meet that and that only with no intention to pay attention to the surrounding region.</p>

username: peepeepoopoo
cooked: <p>right, but wouldn’t the values would be kept randomized? and isn’t’ the loss value being propagated uniformly which means that the unselected region would at least get random values instead of zero?</p>

username: YazanShannak
cooked: <p>Values being zero is actually what is confusing indeed, my wild guess is that you’re doing some kind of batch normalization or L2loss (weight decay) that might keep the values of the other outputs closer to zero, this is just a guess, I am not completely sure what’s happening, what you could do is try and debug a sample from one layer to the other and see what’s actually going on</p>

username: peepeepoopoo
cooked: <p>that’s true, probably the best place to start with</p>
Loss class implementation for KLDivLoss

username: lijunzh
cooked: <p>I am testing the KLDivLoss implementation here: <a href="https://github.com/liuzechun/ReActNet/blob/465f9ba458b3937915e5e5613a85b74123d9ff00/utils/KD_loss.py#L8" rel="noopener nofollow ugc">https://github.com/liuzechun/ReActNet/blob/465f9ba458b3937915e5e5613a85b74123d9ff00/utils/KD_loss.py#L8</a></p>
<p>It can be simplified as</p>
<pre><code class="lang-auto">class DistributionLoss(_Loss):
    def forward(self, model_output, real_output):

        model_output_log_prob = F.log_softmax(model_output, dim=1)
        real_output_soft = F.softmax(real_output, dim=1)
        del model_output, real_output

        real_output_soft = real_output_soft.unsqueeze(1)
        model_output_log_prob = model_output_log_prob.unsqueeze(2)

        cross_entropy_loss = -torch.bmm(real_output_soft, model_output_log_prob)
        cross_entropy_loss = cross_entropy_loss.mean()

        return cross_entropy_loss

</code></pre>
<p>If we define KLDivLoss as <code>KL_loss = - \sum P(x) * log (Q(x) / P(x))</code> for distribution P and Q. Then, I couldn’t find anywhere in the code above has the division between Q and P (or substraction between log(Q) and log( P)).</p>
<p>The following iPython history confirms what I suspected. However, when putting into the inherent class of _Loss, it just magically worked out. I couldn’t understand why. Please help.</p>
<p>Line [60] is running the code outside the class =&gt; wrong answer.<br>
Line [61] is running the code inside the class =&gt; correct answer.<br>
Line [62] is running the “correct” code outside the class =&gt; correct answer.</p>
<pre><code class="lang-auto">In [60]: -torch.bmm(F.softmax(outputs_teacher, dim=1).unsqueeze(1), F.log_softmax(outputs, dim=1).unsqueeze(2)).mean()
Out[60]: tensor(11.1128, grad_fn=&lt;NegBackward&gt;)

In [61]: DistributionLoss()(outputs, outputs_teacher)
Out[61]: tensor(5.4601, grad_fn=&lt;MeanBackward0&gt;)

In [62]: -torch.bmm(F.softmax(outputs_teacher, dim=1).unsqueeze(1), (F.log_softmax(outputs, dim=1) - F.log_softmax(outputs_teacher, dim=1)).unsqueeze(2)).mean()
Out[62]: tensor(5.4601, grad_fn=&lt;NegBackward&gt;)
</code></pre>
How to change a batch RGB images To YCbCr images during training?

username: 1119
cooked: <p>what I want do is:</p>
<pre><code class="lang-auto">RGB_images = netG(input)   #netG is a pretrained model and not change during training,RGB_images  is a batch of RGB images
YCbCr_images = f(RGB_images)   # YCbCr_images  is a batch of YCbCr mode  images
# do things with YCbCr_images 
</code></pre>
<p>Is there any function <strong>f</strong> in pytorch can achieve what i want?</p>

username: smth
cooked: <p>there isn’t an in-built way to do this.<br>
However, you can simply write this as an autograd function</p>
<pre><code class="lang-python">def rgb_to_ycbcr(input):
  # input is mini-batch N x 3 x H x W of an RGB image
  output = Variable(input.data.new(*input.size()))
  output[:, 0, :, :] = input[:, 0, :, :] * 65.481 + input[:, 1, :, :] * 128.553 + input[:, 2, :, :] * 24.966 + 16
  # similarly write output[:, 1, :, :] and output[:, 2, :, :] using formulas from https://en.wikipedia.org/wiki/YCbCr
  return output
</code></pre>

username: gordinmitya
cooked: <aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://kornia.readthedocs.io/en/latest/_modules/kornia/color/ycbcr.html/../../../_static/kornia_logo_mini.png" class="site-icon" width="" height="">
      <a href="https://kornia.readthedocs.io/en/latest/_modules/kornia/color/ycbcr.html#rgb_to_ycbcr" target="_blank" rel="noopener nofollow ugc">kornia.readthedocs.io</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://kornia.readthedocs.io/en/latest/_modules/kornia/color/ycbcr.html#rgb_to_ycbcr" target="_blank" rel="noopener nofollow ugc">kornia.color.ycbcr — Kornia  documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">def rgb_to_ycbcr(image: torch.Tensor) -&gt; torch.Tensor:
    r"""Convert an RGB image to YCbCr.

    Args:
        image (torch.Tensor): RGB Image to be converted to YCbCr.

    Returns:
        torch.Tensor: YCbCr version of the image.
    """

    if not torch.is_tensor(image):
        raise TypeError("Input type is not a torch.Tensor. Got {}".format(
            type(image)))

    if len(image.shape) &lt; 3 or image.shape[-3] != 3:
        raise ValueError("Input size must have a shape of (*, 3, H, W). Got {}"
                         .format(image.shape))

    r: torch.Tensor = image[..., 0, :, :]
    g: torch.Tensor = image[..., 1, :, :]
    b: torch.Tensor = image[..., 2, :, :]

    delta = .5
    y: torch.Tensor = .299 * r + .587 * g + .114 * b
    cb: torch.Tensor = (b - y) * .564 + delta
    cr: torch.Tensor = (r - y) * .713 + delta
    return torch.stack((y, cb, cr), -3)
</code></pre>
The params in model.named_parameters don't update

username: ayushm-agrawal
cooked: <p>Hello,<br>
I save my model’s initial parameters with names using the following code in a dict.</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
  if param.requires_grad:
    pre_params[name] = param
</code></pre>
<p>Then I save the parameters again after a single epoch into a different dict. When I try to compare if these parameters were changed using the code below, I don’t see them being changed. Although, if I look at just <strong>model.parameters()</strong>, they are changed. I need to store the weights using the name for a specific use case.</p>
<pre><code class="lang-auto">for name in pre_params:
  print(pre_params[name] == curr_params[name])
</code></pre>
<p>Any insights on this behavior? Am I missing something here?</p>

username: ptrblck
cooked: <p>You are storing references so you might want to clone the parameters:</p>
<pre><code class="lang-python">pre_params[name] = param.clone()
</code></pre>

username: ayushm-agrawal
cooked: <p>That worked. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
Using optimiser inside another optimiser

username: sobha
cooked: <p>I need to use <code>solve</code> on sparse equations but since it’s not supported, i’m thinking of using the conjugate gradient(or any other optimizer really) to solve the sparse equation. However to implement that i’m not sure how i can run the conjugate gradient and not break the computational graph. i.e i can’t run it in no_grad since i need the gradients to calculate the conjugate gradient. what can i do?</p>