akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Debugging Glow in terms of Glow IR

leejaymin: <p>I have watched on 2018 LLVM Developers’ meeting: Glow LLVM_based machine learning compiler.<br>
In this talk, they mentioned that Glow-IR-level debugging is available.</p>
<p>The following screenshot represents the debugging example on Glow IR.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/94d266ffc60491838d5bb6ad434263ba5827c61f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/94d266ffc60491838d5bb6ad434263ba5827c61f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/94d266ffc60491838d5bb6ad434263ba5827c61f_2_532x499.jpeg" alt="image" data-base62-sha1="lexrVXqJmPcbrScML6ahlGRNh2n" width="532" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/94d266ffc60491838d5bb6ad434263ba5827c61f_2_532x499.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/9/94d266ffc60491838d5bb6ad434263ba5827c61f.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/9/94d266ffc60491838d5bb6ad434263ba5827c61f.jpeg 2x" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">752×706 168 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Unlike C and Assembly modes, I wonder how I could enter the debugging mode for Glow-IR.</p>
Torch.save not UTF-8 encoded

Xiaoyu_Song: <p>I’m using  <code>torch.save(model.state_dict(),'model')</code> to save my model.<br>
My jupyter notebook shows the error:</p>
<pre><code class="lang-auto">model is not UTF-8 encodes, saving disabled.
</code></pre>
<p>Someone knows how to solve this?</p>

Oli: <p>Can you link your model code?</p>

Xiaoyu_Song: <p>Hi, thank you for your reply, I think it’s not about model, I just defined a file named “model”</p>

Oli: <p>What happens if you do</p>
<pre><code class="lang-auto">torch.save(model.state_dict(),'model.pth')
</code></pre>
<p>If that still gives and error, can you report what the <code>model.state_dict()</code> contains, and we might be able to spot something weird in there</p>

farshidbalan: <p>That happens due to the format of the saved file with is Zip and cannot be handled by Jupyter. You can unzip it and open in another app (VSCode) and that would work.</p>
Can't pickle local object 'generate_dataset.<locals>.<lambda>'

kkjh0723: <p>I got the following after I change my code to use multiprocessing and distributed dataparallel module.<br>
Additionally I also started to use <a href="https://github.com/NVIDIA/apex" rel="nofollow noopener">apex</a> package for mixed precision computation.</p>
<pre><code class="lang-auto">-- Process 0 terminated with the following error:

Traceback (most recent call last):

File "/usr/local/lib/python3.5/dist-packages/torch/multiprocessing/spawn.py", line 19, in _wrap

fn(i, *args)

File "/app/train_action_model_apex.py", line 385, in main_worker

evaluate_model(args, root_dir)

File "/app/train_action_model_apex.py", line 343, in evaluate_model

dict_results = evaluator.inference()

File "/app/evaluators/action_model_evaluator.py", line 224, in inference

for data in self.dataloader:

File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 193, in __iter__

return _DataLoaderIter(self)

File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 469, in __init__

w.start()

File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)

File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen

return _default_context.get_context().Process._Popen(process_obj)

File "/usr/lib/python3.5/multiprocessing/context.py", line 274, in _Popen

return Popen(process_obj)

File "/usr/lib/python3.5/multiprocessing/popen_spawn_posix.py", line 33, in __init__

super().__init__(process_obj)

File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__

self._launch(process_obj)

File "/usr/lib/python3.5/multiprocessing/popen_spawn_posix.py", line 48, in _launch

reduction.dump(process_obj, fp)

File "/usr/lib/python3.5/multiprocessing/reduction.py", line 59, in dump

ForkingPickler(file, protocol).dump(obj)

AttributeError: Can't pickle local object 'generate_dataset.&lt;locals&gt;.&lt;lambda&gt;'
</code></pre>
<p>It seems that the error is caused by <code>lambda</code> used for <code>ThreeCrop</code> transformation which mimics <code>FiveCrop</code> from pytorch. I follow the example code in the <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.FiveCrop" rel="nofollow noopener">link</a>.<br>
I found that without <code>ThreeCrop</code> the error was not happened.<br>
And before I used multiprocessing dataparallel, the error was not occured.</p>
<p>The code was tested on linux environment.</p>

ptrblck: <p>Could you try to write a transform class and replace the lambda method with it?<br>
As far as I know there are some limitation in Python regarding pickling lambdas, which is apparently the case here.</p>

kkjh0723: <p>Thanks for your help. Maybe I should write a new transform class as you suggested.</p>
What are the best practices for Multi-Sequence RNNs?

Akamight: <p>So I have the input sequence in the shape of (17,240,512) where the dimensions represented sequence_index, sequence_length and num_features respectively. I dont want to treat these 17 sequences as one whole sequence and squeeze it into an LSTM by merging the first and the third dimensions, so I have to write some very repulsive code:</p>
<pre><code class="lang-auto">self.LSTMs = nn.ModuleList([nn.LSTM(512,512)])
out_tensor = torch.zeros(x.shape)
for i in range(x.shape[1]):
    out_tensor[:,i,:,:] = self.LSTMs[i](x[:,i,:,:])
</code></pre>
<p>However, since LSTMs are running sequentially, I personally think that different LSTMs in this case can be ran in parallel for a quite significant speedup, so is there some kind of layer (Bagging layer or something similar) or custom code that can make me achieve this?</p>
Is the chatbot tutorial not unpacking sequences correctly?

pinocchio: <p>I was going through the <a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" rel="nofollow noopener">chatbot tutorial</a> and noticed the comment and name of the functions didn’t quite match:</p>
<pre><code class="lang-auto">        # Unpack padding
        outputs, _ = nn.utils.rnn.pad_packed_sequence(outputs)
        # Sum bidirectional GRU outputs
</code></pre>
<p>is that a typo/bug?</p>
<hr>
<p>code:</p>
<pre><code class="lang-auto">class EncoderRNN(nn.Module):
    def __init__(self, hidden_size, embedding, n_layers=1, dropout=0):
        super(EncoderRNN, self).__init__()
        self.n_layers = n_layers
        self.hidden_size = hidden_size
        self.embedding = embedding

        # Initialize GRU; the input_size and hidden_size params are both set to 'hidden_size'
        #   because our input size is a word embedding with number of features == hidden_size
        self.gru = nn.GRU(hidden_size, hidden_size, n_layers,
                          dropout=(0 if n_layers == 1 else dropout), bidirectional=True)

    def forward(self, input_seq, input_lengths, hidden=None):
        # Convert word indexes to embeddings
        embedded = self.embedding(input_seq)
        # Pack padded batch of sequences for RNN module
        packed = nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)
        # Forward pass through GRU
        outputs, hidden = self.gru(packed, hidden)
        # Unpack padding
        outputs, _ = nn.utils.rnn.pad_packed_sequence(outputs)
        # Sum bidirectional GRU outputs
        outputs = outputs[:, :, :self.hidden_size] + outputs[:, : ,self.hidden_size:]
        # Return output and final hidden state
        return outputs, hidden
</code></pre>
Unfold function？ MNIST use a 5-step LSTM model to preprocess the input data

Jun_Ren: <p>I split one image of MNIST into 5 segments, for example, 28*28  --&gt; 28x5x5 (delete the left three columns). In this step, I used “fold” function to do this.</p>
<pre><code class="lang-auto">
import torch
from torch import nn
from torch.autograd import Variable
import torchvision.datasets as dsets
import torch.utils.data as Data
import torchvision

torch.manual_seed(1)

EPOCH = 1
BATCH_SIZE = 64
TIME_STEP = 28
INPUT_SIZE = 28
LR = 0.01
DOWNLOAD_MNIST = True

train_data = dsets.MNIST(root='./mnist',
                         train=True,
                         transform=torchvision.transforms.ToTensor(),
                         download=DOWNLOAD_MNIST)

test_data = torchvision.datasets.MNIST(root='./mnist', train=False)

train_loader = Data.DataLoader(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True)

test_x = Variable(torch.unsqueeze(test_data.test_data, dim=1), volatile=True).type(torch.FloatTensor) / 255
test_y = test_data.test_labels


class RNN(nn.Module):
    def __init__(self):
        super(RNN, self).__init__()

        self.rnn = nn.LSTM(
            input_size=100,
            hidden_size=64,
            num_layers=1,
            batch_first=True,
        )
        self.lstm_skb = LSTMTCN(100, 1, 10, 10, 100, 3, 100)
        self.out = nn.Linear(64, 10)

    def forward(self, x):
        x = self.lstm_skb(x)
        r_out, (h_n, h_c) = self.rnn(x, None)

        out = self.out(r_out[:, -1, :])
        return out


rnn = RNN()
print(rnn)

optimizer = torch.optim.Adam(rnn.parameters(), lr=LR)
loss_func = nn.CrossEntropyLoss()

for epoch in range(EPOCH):
    for step, (x, y) in enumerate(train_loader):
        b_x = Variable(x.view(-1, 28, 28))
        b_y = Variable(y)

        output = rnn(b_x)
        loss = loss_func(output, b_y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print('Epoch: ', epoch, '| train loss:%.4f' % loss.item())
</code></pre>
Max and argmax memory leak?

maris_ancans: <p>Using a simple loop, I noticed that the memory slowly increased. This over time will lead to crash. I tried adding detach(), tried adding no grad guards, nothing seems to help.</p>
<pre><code class="lang-auto">         while(true){
             auto Q_val = dqn_module.forward(torch::zeros({10, 4}));
             auto Q_val_max = Q_val.max_values(1, true); 
         }


dqn_module is simple:
    seq = Sequential(
      Linear(g.num_observations, 64),
      Functional(relu),
      Linear(64, 32),
      Functional(relu),
      Linear(32, g.num_actions)
    ); 
</code></pre>
Dot product of 4-d tensor with a 3-d tensor

ari9dam: <p>I have a matrix A with dimension [batch_size,N,M,D] and another matrix B with dimension [batch_size,P,D]. I want to get a tensor C as output with dimension <strong>[batch_size,N,M,P]</strong> in the following way:<br>
C[i,j] = matrix_dot_product(A[i,j], B[i]) 0&lt;=i&lt;batch_size, 0&lt;=j&lt;N</p>
<p>What is the most memory-efficient way to do this?<br>
Thanks!</p>

LeviViana: <p>You can use Einstein summation.</p>
<pre><code class="lang-python">import torch
batch_size, N, M, D, P = 4, 5, 6, 7, 8
A = torch.rand(batch_size, N, M, D)
B = torch.rand(batch_size, P, D)
print(torch.einsum('bnmd,bpd -&gt; bnmp', [x, w]).size()) # batch_size, N, M, P
</code></pre>

ari9dam: <p>It does not seem that it is giving the correct result.</p>
<pre><code class="lang-auto">&gt; C = torch.zeros(batch_size, N,M,P)
&gt; for i in range(batch_size):
&gt;     for j in range(N):
&gt;         C[i,j] = torch.mm(A[i,j],B[i].transpose(0,1))
&gt; is_same = C==torch.einsum('bnmd,bpd -&gt; bnmp', [A, B])
</code></pre>
<p>is_same is not all 1’s.</p>

LeviViana: <p>It works just fine for me:</p>
<pre><code class="lang-python">import torch
batch_size, N, M, D, P = 2, 2, 2, 2, 3
A = torch.arange(16).view(batch_size, N, M, D)
B = torch.arange(16, 28).view(batch_size, P, D)
C = torch.zeros(batch_size, N, M, P)
for i in range(batch_size):
    for j in range(N):
        C[i,j] = torch.mm(A[i,j],B[i].transpose(0,1))

F = torch.einsum('bnmd,bpd -&gt; bnmp', [A, B])

print(F.equal(C.long())) # Prints True
</code></pre>

ari9dam: <aside class="quote no-group quote-modified" data-username="LeviViana" data-post="4" data-topic="51751">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>F = torch.einsum(‘bnmd,bpd → bnmp’, [A, B]) print(F.equal(C.long()))</p>
</blockquote>
</aside>
<p>My bad. Made a silly mistake before. Thank you.</p>
How to correctly list PyTorch as pip package dependency?

ProGamerGov: <p>In my setup.py, I have both <code>torch</code> and <code>torchvision</code> specified for <code>install_requires</code>:</p>
<pre><code>install_requires=['torch', 'torchvision'],
</code></pre>
<p>I setup a test package on <a href="http://test.pypi.org" rel="nofollow noopener">test.pypi.org</a>, and when I try to download my project I get this error:</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch (from project-name) (from versions: none)
ERROR: No matching distribution found for torch (from project-name)
</code></pre>
<p>If I install torch ahead of time using <code>sudo pip3 install torch</code> or <code>sudo pip install torch</code>, I don’t get the error message and <code>torchvision</code> is installed automatically.</p>
Custom autograd

Yifan_Xu: <p>I am reading this snippet of code and I don’t quite get the backward static function. Specifically, what is grad_ouput and why do we copy grad_output. I can guess that grad_input is the value that’s store in .grad field of variables that are set <code>requires_grad=True</code> but how is it related to grad_output?</p>
<pre><code class="lang-auto">class MyReLU(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input &lt; 0] = 0
        return grad_input
</code></pre>
RNN prediction results do not match training results

saschalippert: <p>Hi,</p>
<p>my sine approximation during training works fine, while the prediction (which is feed by previous predictions, with a head start of 20 original values) does not approximate.</p>
<p>I guess there is a bug/misunderstanding in my prediction code <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> A hint would be great.</p>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/saschalippert/36b72313afff86e00f3e10254fb4ff25" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/saschalippert/36b72313afff86e00f3e10254fb4ff25" target="_blank" rel="nofollow noopener">https://gist.github.com/saschalippert/36b72313afff86e00f3e10254fb4ff25</a></h4>
<h5>simplesine.py</h5>
<pre><code class="Python">from collections import deque
import torch
from torch import nn
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import TensorDataset, DataLoader


class RNN(nn.Module):
    def __init__(self, input_size, output_size, hidden_size, n_layers):</code></pre>
This file has been truncated. <a href="https://gist.github.com/saschalippert/36b72313afff86e00f3e10254fb4ff25" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>            </p><div class="onebox imgur-album">
              <a href="https://imgur.com/a/GCQ6QgG" target="_blank" rel="nofollow noopener">
                <span class="outer-box" style="width:600px">
                  <span class="inner-box">
                    <span class="album-title">[Album] Imgur</span>
                  </span>
                </span>
                <img src="https://i.imgur.com/g9k8R89.jpg?fb" title="Imgur" height="315" width="600">
              </a>
            </div>

<p>Thanks</p>

saschalippert: <p>Further testing showed that feeding the network with the predicted values (which have a slight deviation from the target values) cause to spiral the future values out of control. Using a more complex model and training for more episodes seems to solve the issue.</p>
(Suggestion/Question) Dynamic weight networks

Tyler: <p>Currently, there seems to be no (easy) way to set the weights of a layer to a non-Parameter Tensor (eg one produced by another layer). And my question is why?</p>
<p>I would image it would be fine to replace the parameters with any-given tensor (at which point the tensor would no longer appear in <code>self.parameters()</code> and I guess the side-effect of old optimizer taking steps in parameters that are not used any more is also fine).</p>
<p>Is this right or am I missing some important detail that makes this impossible?</p>

SimonW: <p>For uses like this, use the functional API under <code>torch.nn.functional.*</code></p>
If model already on cpu does model.to('cpu') return a copy or a reference?

ONTDave: <p>Sorry for the basic question:<br>
If model already on cpu does model.to(‘cpu’) return a copy of the model or a reference to the existing model?<br>
Same question if model already on gpu and model.to(‘gpu’)?</p>

SimonW: <p>It copies the parameters and buffer only if the requested device/dtype is different. It never copies the module object.</p>
Semantic Segmentation for beginner

DrunkAlex: <p>Hi world!!<br>
I am doing an image semantic segmentation task and I want to use pretrained models such as <code>PSPNet</code> or <code>SegNet</code>. I generally understand the idea of segmentation and its structure but do not know clearly how to implement it in code. I also have some pictures which I should use for training and validation. My question is, what are the steps of this process, from downloading models to training on a particular dataset?<br>
Maybe you know some nice examples which can lead me through all of it.<br>
Thank you!</p>

Nishanth_Sasankan: <p>There are only few Steps,</p>
<p>First Make a Dataloader for semantic segmentation</p>
<p>Second, choose to write your network or perform transfer learning using pretrained network from torchvision models.</p>
<p>Follow any simple github example of training using the same hyperparameters</p>
List of Arbitrary types not supported

lawrencekiba: <p>Hi</p>
<p>I was adding some code to allow accepting list inputs &amp; got this error when running the script</p>
<blockquote>
<p>RuntimeError: Tracing a list of arbitrary type is currently not supported!</p>
</blockquote>
<p>To which it leads me to <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/tracer.h" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/tracer.h</a></p>
<p>Apparently there’s also a similar error for dicts of arbitrary type.</p>
<p>Why is this error coming up in the first place, if I may ask?</p>

Michael_Suo: <p>Tracing can only record what happens to tensors, so a trace that involves lists of arbitrary type is high likely to produce incorrect results. Therefore we throw rather than silently change your output.</p>
F.conv2d functionality

zahra: <p>Hi,</p>
<p>The following code is from PyTorch master documentation but I can not understand it; because I expect when we have an input with dimension [batch size, 1 , 3, 3], a filter tensor with dimension of [1, 1, 2, 2] should exist. As I think, F.conv2d would generate a tensor with dimension [batch_size, 1, 1, 1]. May I ask you to explain about the functionality of the following code and the way that I could see the real convolution like what I described with this code?</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

filters = torch.randn(8,2,2,2)
inputs = torch.randn(1,2,3,3)
a = F.conv2d(inputs, filters)
print(a.shape)
</code></pre>
<p>Thanks</p>

Nikronic: <p>Hi,</p>
<p>The only way to have a output such <code>[batch, 1, 1, 1]</code> is to have a filter with <em>same_size</em> as input when <code>padding=0</code> and <code>stride=1</code>.</p>
<p>The output have to be <code>[batch, 1, 2, 2]</code> for your example input and filters. And the reason is you can have four 2x2 squares in a 3x3 matrix.</p>
<pre><code class="lang-python">step one:
[[x, x, 0],
 [x, x, 0],
 [0 ,0, 0]]

step two:
[[0, y, y],
 [0, y, y],
 [0 ,0, 0]]

step three:
[[0, 0, 0],
 [z, z, 0],
 [z ,z, 0]]

step four:
[[0, 0, 0],
 [0, s, s],
 [0 ,s, s]]

output:
[[x, y],
 [z, s]]
</code></pre>
<p>So as you can see, we can extract 4 numbers which will be in a 2x2 matrix.</p>
<p>PS: number of filters determine number of channels in the output. Each filter a channel.</p>
<p>Good luck<br>
Nik</p>

zahra: <p>Thanks, it could help me.</p>
How to limit the number of threads in libtorch

cyanM: <p>With pytorch, we can use  torch.set_num_threads() to control CPU parallelization, but I don’t know what torch.set_num_threads() corresponds to in libtorch or how to control cpu parallelization in libtorch？Does anyone know？</p>
<p>I have tried to use omp_set_num_threads(), but there’s no effect.（In the test of inference with resnet on cpu, using set_num_threads in python to set differenct number of threads can get differenct cpu usage）</p>
Possible to classify image based on its proximity to similar ones?

Oli: <p>Hello everyone,</p>
<p>I’m looking for help in finding some reading material on the topic of classifying an image based on its proximity to a reference image. I’m working on a project just now where I try to separate cat &amp; dog images based on their cosine similarities. I’m now getting an accuracy of about 98.5% on a small validation set by using Resnet18.</p>
<p>The way that it works (briefly) is that I compute embeddings for an unknown image and checks it cosine similarity against a bunch of dog &amp; cat reference images. If the unknown image matches better with the dog reference images, I’ll classify the unknown image as a dog.</p>
<p>The closest method I can think of is e.g. identifying people via triplet loss. I also know there are fields such as image matching or image retrieval that tangent the subject.</p>
<p>I’m looking for resources or people willing to discuss the subject of classifying a data point in accordance to its proximity with other similar data points.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/sunny.png?v=9" title=":sunny:" class="emoji" alt=":sunny:"></p>
Need help converting pytorch model to onnx

lugookeke: <p>hey guys I’ve been trying to get my pytorch segmentation model to coreML, but looks like I have to convert it to onnx first but I can’t seem to get it to work with everything I tried, is there anyone who’s really experienced In converting models? I would love your help.</p>
<p>link to model - <a href="https://drive.google.com/file/d/1yOkUCmlm8X8VQuh-hl1v4IeJe39PxnuP/view?usp=sharing" rel="nofollow noopener">https://drive.google.com/file/d/1yOkUCmlm8X8VQuh-hl1v4IeJe39PxnuP/view?usp=sharing</a></p>

lawrencekiba: <p>You can try the export module</p>
<p><a href="https://pytorch.org/docs/stable/onnx.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/onnx.html</a></p>
<p>If you want some extra read</p>
<p><a href="https://michhar.github.io/convert-pytorch-onnx/" class="onebox" target="_blank" rel="nofollow noopener">https://michhar.github.io/convert-pytorch-onnx/</a></p>
Implementing loss function for bivariate Gaussian distribution

NerualEngine: <p>Hi,</p>
<p>I’m trying to implement a negative log likelihood loss function for a bivariate Gaussian distribution using torch MultivariateNormal.</p>
<p>my current implementation as follow:</p>
<p>import torch<br>
from torch.distributions.multivariate_normal import MultivariateNormal as MVNormal</p>
<pre><code class="lang-auto">def Gaussian2DLikelihood(outputs, targets):
    #mux is mean of x
    #mux is mean of y
    #sx,sy is std  &gt;0
    #corr is correlation -1&lt;corr&lt;1


    batch_size = targets.shape[0]
    mux, muy, sx, sy, corr = outputs[:,0], outputs[:,1], outputs[:,2], outputs[:,3], outputs[:,4]
    sx = torch.exp(sx) 
    sy = torch.exp(sy)
    corr = torch.tanh(corr)
    mux, muy, sx, sy, corr = mux.unsqueeze(-1), muy.unsqueeze(-1), sx.unsqueeze(-1), sy.unsqueeze(-1), corr.unsqueeze(-1)
    
    mu_xy = torch.cat([mux,muy],1)
    print('mu_xy.shape:',mu_xy.shape)#mu_xy.shape: torch.Size([8, 2])

    cov_xy = torch.zeros(batch_size,2,2) #Covariance matrix
    cov_xy[:,0,0] = (sx*sx).squeeze()
    cov_xy[:,0,1] = (corr*sx*sy).squeeze()
    cov_xy[:,1,0] = (corr*sx*sy).squeeze()
    cov_xy[:,1,1] = (sy*sy).squeeze()
    print('cov_xy.shape:',cov_xy.shape)#cov_xy.shape: torch.Size([8,2, 2])

    
    loglik = -normal.log_prob(targets[0,:])
    for d in range(1,batch_size):
        loglik += -normal.log_prob(targets[d,:])


    return loglik/batch_size
</code></pre>
<p>A test :<br>
<code>Gaussian2DLikelihood(torch.rand(8,5),torch.rand(8,2))</code></p>
<p>I’m not quite sure it’s correct, I’ve seen some examples where people take the log of loglik then they use the log-sum-exp thing.</p>
<p>Any ideas?</p>
Why this code will throw out an inplace error

082a83b8eb9177051864: <pre><code class="lang-auto">    A = torch.nn.Parameter(torch.randn(3, 3))
    B = torch.randn(2, 4, 3)
    B[0, :, :] = B[0, :, :].mm(A)
    loss = B.sum()
    loss.backward()
</code></pre>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4, 3]] is at version 1; expected version 0 instead.<br>
How could I solve this problem? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Nikronic: <aside class="quote no-group" data-username="082a83b8eb9177051864" data-post="1" data-topic="51574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/082a83b8eb9177051864/48/13609_2.png" class="avatar"> 082a83b8eb9177051864:</div>
<blockquote>
<p>B[0, :, :] = B[0, :, :].mm(A)</p>
</blockquote>
</aside>
<p>Hi,</p>
<p>In this line, you are updating <code>B</code> variable but autograd needs all previous values to compute the backpropagation.</p>
<p>You can do something like this to ensure you are not losing the value of <code>B</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

A = torch.nn.Parameter(torch.randn(3, 3))
B = torch.randn(2, 4, 3)
z = B[0, :, :].mm(A)
loss = z.sum()
loss.backward()
</code></pre>
<p>When I assign updated <code>B</code> to <code>z</code>, I am not mutating <code>B</code> so autograd has access to original <code>B</code>.</p>
<p>Best regards<br>
Nik</p>

082a83b8eb9177051864: <p>Thanks for your kind help. Actually, the code is the simplified version of my problem. In the real situation, I cannot assign B[0, :, :].mm(A) to z. However, following your inspiration, I find use .clone() can solve my problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Nikronic: <p>Thank you for sharing your information. Actually, I have not had such a problem before and I just found a solution by rule of thumb after reading <code>inplace</code> definition doc. Your approach is the conventional one.</p>
<p>Good luck</p>
Build Pytorch from source failed

Joel_Wu: <p>I’m trying building pytorch from source following this <a href="https://github.com/pytorch/pytorch#from-source" rel="nofollow noopener">instruction</a>. However, I encounter an error below, what am I missing?</p>
<pre><code class="lang-auto">In file included from ../caffe2/contrib/aten/aten_op.cc:1:0:
./caffe2/contrib/aten/aten_op.h: In lambda function:
./caffe2/contrib/aten/aten_op.h:9551:33: error: ‘pstrf’ is not a member of ‘at’
               auto the_result = at::pstrf(self, upper, tol);
                                 ^
./caffe2/contrib/aten/aten_op.h: In lambda function:
./caffe2/contrib/aten/aten_op.h:9561:33: error: ‘pstrf’ is not a member of ‘at’
               auto the_result = at::pstrf(self, upper);
                                 ^
./caffe2/contrib/aten/aten_op.h: In lambda function:
./caffe2/contrib/aten/aten_op.h:9571:33: error: ‘pstrf’ is not a member of ‘at’
               auto the_result = at::pstrf(self);
                                 ^
</code></pre>

ptrblck: <p>Could you try to <code>sync</code> and <code>update</code> the <code>submodules</code> again, run <code>python setup.py clean</code> and rebuild it again?</p>

Joel_Wu: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> yeah, it works. Thanks!</p>
Trouble with passing manually calculated gradient as argument of backward()

JakeNocentino: <p>Hi all,</p>
<p>I am currently having trouble with passing in a gradient manually as the argument to the <code>backward()</code> function. I am doing this because the loss I compute is done with a graphical model in C++. I then manually calculate the gradient of the C++ loss w.r.t. h.</p>
<p>h is a N x G matrix that is multiplied by a G x 2 (I have 2 classes) matrix to calculate my logits. Anyways, I manually calculate this gradient and then pass it to <code>backward()</code> which I call on the original h in my PyTorch code. However, this results in my performance going down drastically (.5 AUC) and I have noticed that, after calling <code>h.backward(h_grad)</code>, all gradients in my parameters are remarkably larger in magnitude than before (e.g., most of them being in the ±80s, ±90s, ±100s, etc., when originally they were all between -1 and 1).</p>
<p>Am I doing something wrong here? In theory, I believe that if I pass the gradient of the loss w.r.t. h into h, then it should be basically the same thing as calling <code>loss.backward()</code>. If this is wrong, or if I am wrong, someone please let me know!</p>
<p>Thank you!</p>

ptrblck: <p>Are you setting the gradient directly as the attribute of the parameter or are you passing it to the loss function?<br>
Do you have a small code snippet to show your use case?</p>

JakeNocentino: <p>My code looks something like this:</p>
<pre><code class="lang-auto">self.train_one_step(hgrad)
# hgrad is the gradient manually calculated by the graphical model C++ code
</code></pre>
<p>And then, in the <code>train_one_step()</code> method, I call</p>
<pre><code class="lang-auto">h.backward(hgrad)
</code></pre>
<p>I guess my main question is, this is the correct way to handle a scenario like this (where we do not backward() directly from loss), right? If it is, then I guess it narrows my problem down to a data error or a gradient calculation error.</p>
Retrieve Module Constructor Parameters from a saved Pytorch-ONNX model

chrisllc: <p>May I get some help to retrieve the constructor parameters (in_channels &amp; out_channels) in nn.Conv2D from a saved Pytorch-ONNX model.</p>
<p>Background:</p>
<ol>
<li>Load a pre-trained Alex-Net model from troch-vision, and save it to ONNX model.</li>
<li>Load the saved onnx model, and try to recover it back to a Pytorch model, but have difficulties to retrieve required information. For example, in conv operators, it has only 5 attributes (dilations, group, kernel_shape, pads, strides), the required parameters like <strong>in_channels</strong> and <strong>out_channels</strong> are missing.</li>
</ol>
<p>My goal is to recover the model back from Onnx. Is there a way to retrieve enough information to achieve my goal? If no, is there a work-around?</p>
<p>I understand it is extremely similar to the existing issue “Importing Onnx model”, however I still want to raise this up, since from my point of view, the saved onnx model might lack adequate info to re-load by Pytorch itself and other frameworks.</p>
<p>Thanks in advance!</p>
"log_softmax function" in pytorch tutorial example

spnova12: <p>I am studying the link below.<br>
<a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/nn_tutorial.html</a></p>
<p>But I can’t understand “log_softmax” written in this document.</p>
<pre><code class="lang-auto">def log_softmax(x):
    return x - x.exp().sum(-1).log().unsqueeze(-1)
</code></pre>
<p>How this function match to the figure below?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cecef17233977351791a63330c28d2c4634227d7.png" alt="image" data-base62-sha1="tvvJpYwxiFBwghgvMdFFwl8BEkD" width="347" height="76"></p>

KFrank: <p>Hi Dong Wook!</p>
<aside class="quote no-group" data-username="spnova12" data-post="1" data-topic="52041" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spnova12/48/3495_2.png" class="avatar"> spnova12:</div>
<blockquote>
<p>But I can’t understand “log_softmax” written in this document.</p>
<pre><code class="lang-auto">def log_softmax(x):
    return x - x.exp().sum(-1).log().unsqueeze(-1)
</code></pre>
<p>How this function match to the figure below?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/cecef17233977351791a63330c28d2c4634227d7.png" alt="image" data-base62-sha1="tvvJpYwxiFBwghgvMdFFwl8BEkD" width="347" height="76"></p>
</blockquote>
</aside>
<p>My guess is that you’re being thrown off by the “log-sum-exp trick”<br>
that is being used to rewrite the “standard” expression for<br>
<code>log_softmax</code> in a (mathematically-equivalent) form that avoids<br>
floating-point overflow / underflow problems when evaluating<br>
the expression numerically.</p>
<p>See the “log-sum-exp trick for log-domain calculations” section of<br>
the <a href="https://en.wikipedia.org/wiki/LogSumExp" rel="noopener nofollow ugc">LogSumExp</a> Wikipedia article for an explanation.  (This article<br>
is not specifically about the <code>log_softmax</code> function, but instead<br>
about the related LogSumExp function.  <code>log_softmax</code> has the<br>
same potential overflow problems, and they are avoided using<br>
the same log-sum-exp trick.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

spnova12: <p>Thank you very much for your explanation. It would be nice if I could find the same trick elsewhere, but it is not easy to find the same trick.</p>
Should convolution with (+1,-1) valued kernel be faster than with a regular kind?

notacode: <p>I apologize for a long question and if it seems a really strange thing to ask.</p>
<p>If I have an N by N input image and I convolve it with a kernel that consists of (+1,-1) values, should this operation be faster then if kernel had random numbers (say, from a standard normal distribution)? To me, it seems that during the convolution operation the multiplication of the image and kernel values leads to a simple sign change but it still counts as regular multiplication and hence does not really affect the operation count and overall complexity. However, my colleague tells me that I’m wrong and we should see an increase in speed, which makes me sort of doubt that I am implementing everything correctly.</p>
<p>So far I have tried using timing magic in Jupyter notebooks to measure the time per convolution with (+1,-1) kernels and with regular kinds. I saw no difference. My main doubt is: should there be any at all? Is there a way to speed this up for such a specific kernel?</p>
<pre><code class="lang-auto">from torch.nn import functional as f
import torch

kernel = torch.randn((255,3,3,3)).sign_().float()
image = torch.randn((1,3,224,224)).float()
</code></pre>
<hr>
<pre><code class="lang-auto">%%timeit
f.conv2d(image, kernel, stride=1, padding=1,)
</code></pre>
<hr>
<p>This gives me: 10 loops, best of 3: 78.3 ms per loop<br>
while running without <code>sign_()</code> operation yields 78.9 per loop.</p>
<p>Should there <em>be</em> any significant difference?</p>
How to call this network

CF-chen-feng-CF: <p><span class="hashtag">#include</span> “alexnet.h”</p>
<p><span class="hashtag">#include</span> “modelsimpl.h”</p>
<p>namespace vision {<br>
namespace models {<br>
AlexNetImpl::AlexNetImpl(int64_t num_classes) {<br>
features = torch::nn::Sequential(<br>
torch::nn::Conv2d(<br>
torch::nn::Conv2dOptions(3, 64, 11).stride(4).padding(2)),<br>
torch::nn::Functional(modelsimpl::relu_),<br>
torch::nn::Functional(modelsimpl::max_pool2d, 3, 2),<br>
torch::nn::Conv2d(torch::nn::Conv2dOptions(64, 192, 5).padding(2)),<br>
torch::nn::Functional(modelsimpl::relu_),<br>
torch::nn::Functional(modelsimpl::max_pool2d, 3, 2),<br>
torch::nn::Conv2d(torch::nn::Conv2dOptions(192, 384, 3).padding(1)),<br>
torch::nn::Functional(modelsimpl::relu_),<br>
torch::nn::Conv2d(torch::nn::Conv2dOptions(384, 256, 3).padding(1)),<br>
torch::nn::Functional(modelsimpl::relu_),<br>
torch::nn::Conv2d(torch::nn::Conv2dOptions(256, 256, 3).padding(1)),<br>
torch::nn::Functional(modelsimpl::relu_),<br>
torch::nn::Functional(modelsimpl::max_pool2d, 3, 2));</p>
<p>classifier = torch::nn::Sequential(<br>
torch::nn::Dropout(),<br>
torch::nn::Linear(256 * 6 * 6, 4096),<br>
torch::nn::Functional(torch::relu),<br>
torch::nn::Dropout(),<br>
torch::nn::Linear(4096, 4096),<br>
torch::nn::Functional(torch::relu),<br>
torch::nn::Linear(4096, num_classes));</p>
<p>register_module(“features”, features);<br>
register_module(“classifier”, classifier);<br>
}</p>
<p>torch::Tensor AlexNetImpl::forward(torch::Tensor x) {<br>
x = features-&gt;forward(x);<br>
x = torch::adaptive_avg_pool2d(x, {6, 6});<br>
x = x.view({x.size(0), -1});<br>
x = classifier-&gt;forward(x);</p>
<p>return x;<br>
}</p>
<p>} // namespace models<br>
} // namespace vision</p>

yf225: <p>You can use <code>vision::models::AlexNet</code> like any other <code>nn::Module</code>s in C++ frontend. For example: <a href="https://github.com/pytorch/vision/blob/2287c8f2dc9dcad955318cc022cabe4d53051f65/test/test_models.cpp#L9-L19" rel="nofollow noopener">https://github.com/pytorch/vision/blob/2287c8f2dc9dcad955318cc022cabe4d53051f65/test/test_models.cpp#L9-L19</a>.</p>

CF-chen-feng-CF: <p>int main() {<br>
torch::Tensor t = torch::ones({ 2, 3, 224, 224 });<br>
ResNet resnet = resnet50();<br>
t = resnet.forward(t);<br>
std::cout &lt;&lt; t.sizes() &lt;&lt; t &lt;&lt; std::endl;<br>
}<br>
This is my own writing a resnet50, by calling it, entering a tensor, and getting a result, you can print out the result. But for the network inside this link, I will not do the same operation.</p>

CF-chen-feng-CF: <p>int main() {<br>
torch::Tensor t = torch::ones({ 2, 3, 224, 224 });<br>
ResNet resnet = resnet50();<br>
t = resnet.forward(t);<br>
std::cout &lt;&lt; t.sizes() &lt;&lt; t &lt;&lt; std::endl;<br>
}<br>
This is my own writing a resnet50, by calling it, entering a tensor, and getting a result, you can print out the result. But for the network inside this link, I will not do the same operation.</p>
Question regarding differentiation

Haimin_Hunter_Zhang: <p>Hi guys, please see the following image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fd38f3dabc1d78314cab26d2f055a8ad08891af7.png" data-download-href="https://discuss.pytorch.org/uploads/default/fd38f3dabc1d78314cab26d2f055a8ad08891af7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/f/fd38f3dabc1d78314cab26d2f055a8ad08891af7.png" alt="image" data-base62-sha1="A86ORhx25Vtm4p1cFBX5ALGVauj" width="690" height="303" data-dominant-color="080908"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">779×343 7.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I don’t get why the third element of a.grad is -0.333. Could anyone fill me in?</p>

ptrblck: <p>The gradients you are seeing is accumulated from the division and the max operation.<br>
Here is the step by step approach:</p>
<pre><code class="lang-python">a = torch.tensor([1., 2., 3.], requires_grad=True)

# Get div gradients
y = a / a.max().detach()
y.backward(torch.ones_like(y))
print('div grad: ', a.grad)
a.grad.zero_()

# Get max gradients
y = a.detach() / a.max()
y.backward(torch.ones_like(y))
print('max grad: ', a.grad)
a.grad.zero_()

# Get both
y = a / a.max()
y.backward(torch.ones_like(y))
print('both: ', a.grad)

div grad:  tensor([0.3333, 0.3333, 0.3333])
max grad:  tensor([ 0.0000,  0.0000, -0.6667])
both:  tensor([ 0.3333,  0.3333, -0.3333])
</code></pre>

Haimin_Hunter_Zhang: <p>Much appreciated. I get it now.</p>
Aggregating Predictions before Computing Loss

arjung: <p>I am working on binary classification with time-series data. My approach is to, given a 30-second long signal, split it into <em>n</em> (overlapping) 2-second long signals, feed each one of these <em>n</em> 2-second long signals into the model, get <em>n</em> predictions as output in an array, and then compute the prediction of the whole 30-second long signal by aggregating the <em>n</em> predictions, before using this single prediction in the loss.</p>
<p>My model isn’t learning, and I suspect it is because I have not adequately set .requires_grad=true in the right places during this entire computation.</p>
<p>Here’s my code:</p>
<pre><code class="lang-auto">input_size = 30000
window_size = 2000
window_step = 100

for i in range(batch_size):
            
    signal = torch.reshape(batch_x[i], (1, 1, input_size))        
    steps = ((input_size - window_size) / window_step) + 1
            
    agg_pred = []    
    loss = 0
    for j in range(steps):
                
        signal_segment = signal[:, :, j*window_step:j*window_step + window_size]
        y_pred = model(signal_segment)
                               
        agg_pred.append(int(round(y_pred)))
                
    signal_pred = torch.tensor(np.argmax(np.bincount(np.array(agg_pred))), dtype=torch.float, device="cuda", requires_grad=True)
            
    loss += nn.BCELoss(signal_pred, batch_y[i])
    loss_values.append(loss)
            
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

</code></pre>
<p>Any ideas as to what I could try modifying?</p>

ptrblck: <p>You are detaching the computation graph in a few places:</p>
<ul>
<li>
<code>int(round(y_pred))</code> will detach <code>y_pred</code> so that the backward call cannot compute the gradients of all preceding operations</li>
<li>numpy operations are not differentiable in PyTorch, so you would have to use the corresponding PyTorch method or write the backward function manually</li>
<li>re-wrapping the output in a <code>torch.tensor(, requires_grad=True)</code> will also detach the input from all preceding operations</li>
</ul>
Transfer from Tensorflow to Pytorch

111121: <p>a, b = sess.run([c, d], feed_dict={…}), how to substitute this in pytorch?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/53bc702f9e456997f4c4ab4bb6e27fb6e54d3cf1.png" data-download-href="https://discuss.pytorch.org/uploads/default/53bc702f9e456997f4c4ab4bb6e27fb6e54d3cf1" title="14%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/53bc702f9e456997f4c4ab4bb6e27fb6e54d3cf1_2_690x49.png" alt="14%20PM" data-base62-sha1="bWLm8DxyhcNOVTirmF67nn9WZk5" width="690" height="49" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/53bc702f9e456997f4c4ab4bb6e27fb6e54d3cf1_2_690x49.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/53bc702f9e456997f4c4ab4bb6e27fb6e54d3cf1_2_1035x73.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/53bc702f9e456997f4c4ab4bb6e27fb6e54d3cf1_2_1380x98.png 2x" data-dominant-color="3B3A3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">14%20PM</span><span class="informations">1924×138 27.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>In PyTorch you can directly use all operations without compiling a static graph and use sessions (similar to numpy, but with GPUs and Autograd).<br>
Have a look at the <a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html" rel="nofollow noopener">tutorials</a> for a quick start <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to implement 'np.where()' operation with libtorch

Ryan95: <p>Hi, i can get a matrix with numpy like this:</p>
<pre><code class="lang-auto">import numpy as np
label = np.random.randint(1,6,(2,3))
tmp = np.array(np.where(label == 3)).transpose((1, 0))
</code></pre>
<p>if label is <code>array([[1, 2, 3], [3, 3, 4]])</code>.  then tmp will be <code>array([[0, 2], [1, 0], [1, 1]], dtype=int64)</code>.</p>
<p>and i want to get the same result for libtorch. here is my code</p>
<pre><code class="lang-auto">auto ten = torch::randint(1,10,{2,3});
auto y = torch::full({2,3},3)
auto ind = torch::where(torch::operator==(ten,y), ten, y);
</code></pre>
<p>and ind’s data is <code>3 3 3 3 3 3 [Variable[CPUType]{2,3}]</code></p>
<p>where’s my fault?</p>

ptrblck: <p>Note that you are basically using <code>(label==3).nonzero()</code> in your numpy code, since you only provide the condition without <code>x</code> and <code>y</code>. From the <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html">docs</a>:</p>
<blockquote>
<p>Note<br>
When only <em>condition</em> is provided, this function is a shorthand for <code>np.asarray(condition).nonzero()</code> . Using <a href="https://docs.scipy.org/doc/numpy/reference/generated/numpy.nonzero.html#numpy.nonzero"> <code>nonzero</code> </a> directly should be preferred, as it behaves correctly for subclasses. The rest of this documentation covers only the case where all three arguments are provided.</p>
</blockquote>
<p>That being said, let’s check the PyTorch code.<br>
Your C++ code should be the equivalent to this Python code:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.where(
    torch.from_numpy(label) == 3,
    torch.from_numpy(label),
    torch.tensor([3])
)
</code></pre>
<p>If the condition is <code>True</code>, the values from the first tensor will be drawn, else from the second one.<br>
Since you are comparing the tensor to 3, all <code>True</code> conditions will draw exactly the 3 from <code>label</code>. For all <code>False</code> return values in the condition, you will draw the corresponding value from a tensor full of 3s, which is why your output will contain all 3s.</p>
Jit.trace with optimize=True gives wrong output

mpry: <p>When I trace a model with jit.trace and the <code>optimize=True</code> flag, I get a tracer warning saying that the outputs don’t match and the error is also significant. But when I trace with the flag set to False, everything is fine. Any idea why this might be happening?</p>

ptrblck: <p>Could you post a reproducible code example so that we could have a look?</p>
Using Pytorch optim algorithms with gradients computed outside model.backward()

pytorchuser: <p>I’m implementing an algorithm that involves some tricks to allow parallelism in the forward runs and gradient computations during the training process. I’m doing these computations using the multiprocessing library, which seems to mean that I can’t set requires_grad=True for my variables even if I do not use the backward() function and calculate them myself. I wonder whether this also means I’m unable to use the optim optimization routines, such as SGD, or if there is a way to make use of them?</p>

ptrblck: <p>The optimizers should still work, as they are just using the <code>.grad</code> attribute of each passed parameter:</p>
<pre><code class="lang-python">x = torch.zeros(1)
optimizer = torch.optim.SGD([x], lr=1.)

x.grad = torch.tensor([10.])
optimizer.step()

print(x)
&gt; tensor([-10])
</code></pre>