akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error when loading pretrained layers in fasterRCNN torchvision

username: Nithin_Rao
cooked: <p>Hi, I’m trying to use pretrained FasterRCNN network provided in torchvision.<br>
<code>model=torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)</code><br>
doing<br>
<code>model.eval()</code><br>
<code>model([image_tensor])</code><br>
working perfectly fine but when I use sequential to stop at intermediate layers like this<br>
<code>model2=nn.Sequential(*list(model.children())[:-2])</code><br>
to use until FPN network I’m getting error when passing a image as show below<br>
<code>TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not tuple</code></p>

username: Nithin_Rao
cooked: <p>I want this because I want to get Region Proposal Network output from faster rcnn for another model training. Thanks in advance for the help</p>

username: Nithin_Rao
cooked: <p><a class="mention" href="/u/fmassa">@fmassa</a>  can you help me with this?</p>

username: Nithin_Rao
cooked: <p><a class="mention" href="/u/fmassa">@fmassa</a> I have adopted code from <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/faster_rcnn.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/detection/faster_rcnn.py</a> and implemented<br>
<a href="https://github.com/nithinraok/VisualQuestion_VQA/blob/master/common_resources/rpn_test_2.ipynb" rel="nofollow noopener">https://github.com/nithinraok/VisualQuestion_VQA/blob/master/common_resources/rpn_test_2.ipynb</a> to get RPN output. But unfortunately I am unable to get Kx2048 vector output. Instead I am getting 14x4. Can you please help me with this?</p>
Different speed to value the tensor by index on different Pytorch versions (0.2 and 0.4)

username: walk2out
cooked: <p>Hi, all:</p>
<p>these days, I got a problem: there is different speed to value the tensor by index on different pytorch platforms (Pytorch 0.2 and Pytorch 0.4).</p>
<p>This is test code:</p>
<pre><code class="lang-auto">import torch
import time
import numpy as np
def main():
    #tensor_a = (torch.rand(20,100,100)).cuda()
    #tensor_b = torch.rand(20,100,100).cuda()
    #np.save('tensor_a.npy', tensor_a.cpu().numpy())
    #np.save('tensor_b.npy', tensor_b.cpu().numpy())
    tensor_a = torch.from_numpy(np.load('tensor_a.npy', encoding="latin1")).cuda()
    tensor_b = torch.from_numpy(np.load('tensor_b.npy', encoding="latin1")).cuda()
    torch.cuda.synchronize()
    end = time.time()
    for i in range(100):
        tensor_b[tensor_a &lt;= 0.5] = 0
    torch.cuda.synchronize()
    print('run time is:', time.time() - end)
if __name__ == '__main__':
    main()
</code></pre>
<p>This is the speed:<br>
Pytorch 0.2: total time is 0.0015s<br>
Pytorch 0.4: total time is 0.17s</p>
<p>What does cause the speed decline on Pytorch 0.4? And how to solve the speed problem on Pytorch 0.4?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I assume you are using the shapes <code>[20, 100, 100]</code> for your tests?<br>
Could you also time the code using the latest PyTorch version (<code>1.1.0</code>)?</p>
<p>Also, which GPU, CUDA and cuDNN version are you using?</p>

username: walk2out
cooked: <p>the GPU is Geforce GTX 1080 Ti, cuda is cuda9.0, cuDNN is 7.0; I haven’t tried the pytorch1.1, but both pytorch 0.2 and 0.3 is ok.</p>
What are the parameters and use of self.fc_mu() and self.fc_sigma()?

username: Sd_Sad
cooked: <p>I am looking through some example code, and am not new to Machine Learning, but I saw:</p>
<p>self.fc_mu = nn.Linear(hidden_size, hidden_size)<br>
self.fc_sigma = nn.Linear(hidden_size, hidden_size)</p>
<p>I understand that self.fc1(),self.fc2()… are referring to the fully connected layers one and two which are both Linear in my example, but what is self.fc_mu referring to, and what does it return? Also, what are the parameters of self.fc_mu(), and self.fc_sigma() which I saw used many times? Thank you in advance, I am just struggling to figure this out.</p>

username: ptrblck
cooked: <p><code>self.fc_mu</code> and <code>self.fc_sigma</code> are just the attribute names for both linear layers.<br>
Their meaning depends on the context. In this case they might be used to apply the “reparametrization trick”.</p>

username: Sd_Sad
cooked: <p>In the context that I am currently in, this is the code:</p>
<p>class Discriminator(nn.Module):<br>
def <strong>init</strong>(self, input_size=2, output_size=1, hidden_size=128):<br>
super().<strong>init</strong>()<br>
self.fc1 = nn.Linear(input_size, hidden_size)<br>
self.fc2 = nn.Linear(hidden_size, hidden_size)<br>
self.fc_mu = nn.Linear(hidden_size, hidden_size)<br>
self.fc_sigma = nn.Linear(hidden_size, hidden_size)<br>
self.fc3 = nn.Linear(hidden_size, output_size)</p>
<pre><code>    for m in self.modules():
        if isinstance(m, nn.Linear):
            nn.init.kaiming_normal_(m.weight)
            nn.init.constant_(m.bias, 0.0)
            
def forward(self, x):
    x = F.leaky_relu(self.fc1(x), negative_slope=2e-1)
    x = F.leaky_relu(self.fc2(x), negative_slope=2e-1)
    mu = self.fc_mu(x)
    sigma = torch.sigmoid(self.fc_sigma(x))
    x = torch.sigmoid(F.leaky_relu(self.fc3(mu + sigma * torch.randn_like(sigma)), negative_slope=2e-1))
    return x, mu, sigma
</code></pre>
<p>What would this entail? And what is the reparametrization trick entailing?</p>
Cuda runtime error (11)

username: Renat_Hizbullin
cooked: <p>CUDA: 9.0<br>
CUDNN: 7.4<br>
GPU: 2080ti<br>
Python: 3.6<br>
Installed pytorch via “pip3 install torch torchvision”</p>
<p>I have THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument error while run</p>
<blockquote>
<p>import torch<br>
from torchvision.models import vgg16</p>
<p>model =  vgg16().cuda()<br>
x = torch.zeros((32, 3, 227, 227)).cuda()</p>
<p>model(x)</p>
</blockquote>
<p>Crash while run first conv layer.<br>
How to solve this?</p>

username: JuanFMontesinos
cooked: <p>Have u exported the cuda environment paths? I guess you didn’t</p>

username: Renat_Hizbullin
cooked: <p>I’v done changes in .bashrc</p>
<blockquote>
<p>export CUDA_HOME=/usr/local/cuda-9.2<br>
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH<br>
export PATH=${CUDA_HOME}/bin:${PATH}</p>
</blockquote>

username: Renat_Hizbullin
cooked: <p>Solved this by installing cuda via .sh, not via .deb</p>

username: 886e3d995b23a08257c9
cooked: <p>Hello, I have the same question. can I know the detail of how you solve this problem</p>

username: Renat_Hizbullin
cooked: <p>I install CUDA 9.2 using .sh script instead of CUDA 9.0 using .dep package.</p>

username: sunyuanrui
cooked: <p>Hello, I install CUDA 9.0 using .sh script, and have the same question, it must require cuda9.2?</p>

username: Renat_Hizbullin
cooked: <p>It seems like a bug in pytorch. cuda9.2 is solution for me.</p>

username: dawnwch
cooked: <p>I have the same question by using CUDA 9.2, CUDNN 7.4, 2080, and I install CUDA 9.2 with .sh not .deb.<br>
Is the error caused by the graphics card driver version?</p>

username: Renat_Hizbullin
cooked: <p>It seems, last pytorch version(1.0.0) doesn’t work with CUDA 9.2. I intalled CUDA 10 and all works fine.</p>

username: dawnwch
cooked: <p>yes, CUDA 10 and pytorch1.0.0 works fine</p>

username: Yiru_Shen
cooked: <p>I have the same issue and the version for CUDA is 10.0, with RTX 2080. Anyone has the same problem here?</p>

username: Yiru_Shen
cooked: <p>Okay I have solve the problem. You cannot directly install pytorch, instead “pip3 install -U <a href="https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl%E2%80%9D" rel="nofollow noopener">https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl”</a> work for me.</p>

username: jbohnslav
cooked: <p>I had a similar issue with my RTX GPU. I installed PyTorch from source, which fixed my issue.</p>

username: zjsh
cooked: <p>I had a similar issue with RTX 2080 ti, I installed Pytorch via following command, which fixed my issue.<br>
conda install pytorch=0.4.0 torchvision=0.2.1 cudatoolkit=9 -c pytorch</p>

username: ptrblck
cooked: <p>Since you are using a Turing GPU, you should use CUDA10 instead of CUDA9.<br>
Also, I would recommend to install the latest PyTorch version (<code>1.1.0</code>) which ships with some bug fixes and a lot of new features.</p>

username: zjsh
cooked: <p>But, I needed to this version and I reviewed other solutions.<br>
However you’re right, with latest PyTorch version, everything works fine.<br>
Thanks.</p>
How to use the bidirectional_dynamic_rnn with pytorch?

username: Donggeun
cooked: <p>I want to use the initial_state.</p>
<pre><code class="lang-auto">            outputs, states = tf.nn.bidirectional_dynamic_rnn(cell_fw, cell_bw,encoder_conv_output,sequence_length=input_lengths, initial_state_fw=initial_state_fw,initial_state_bw=initial_state_bw,dtype=tf.float32)
</code></pre>
Multi-dataset learning, two output layers

username: overfitted
cooked: <p>Hi everybody,</p>
<p>So Im trying to implement the following framework:</p>
<p>Co-training on 2 different datasets. The two datasets will share a backbone net (say a VGG until the last pooling layer, we throw away fc layers). There will be two fc output layers, one for the predictions of Dataset 1 and the other for the predictions of Dataset 2. Each output layer / dataset will have its own  loss function.</p>
<p>An approach that I tried is compromised of 3 basic steps: 1. give as input argument to model’s forward() the <em>dataset</em> that is forward propagated in the current batch, such that the model will compute and return ONLY the output related to <em>dataset</em>, so in each batch iteration the model will output the values of EITHER of the fc layers and NOT BOTH. 2. I combine the dataloaders of the datasets with itertools.zip_longest(), and 3. In each batch iteration for each dataset separately I do the following: a) compute the output and the loss, b) backpropagate ONLY for the relevant dataset/loss/fc layer. a) and b) are performed independently and sequentially for Dataset 1 and Dataset 2, hence in each iteration 2*batch_size updates are performed.</p>
<p>Currently Im getting very poor results with this approach, and Im not sure if it is because with my implementation something is done wrongly in Autograd, or because the two different datasets are not suitable to be combined together.</p>
<p>Do you see anything wrong in my implementation or do you have any better approach to do something like this? Thank you!</p>
How to change the kernel size in Auxlogits of inception v3 network in pytorch in pretrained model.?

username: SHASHANK_KUMAR_MISHR
cooked: <p>l have runtime error while training inception v3 network in pytorch</p>
<p>RuntimeError: Calculated padded input size per channel: (3 x 3). Kernel size: (5 x 5). Kernel size can’t be greater than actual input size</p>

username: ptrblck
cooked: <p>Answered <a href="https://discuss.pytorch.org/t/error-occurring-in-inception-v3-network/47169/2">here</a>.</p>
Error occurring in inception v3 network

username: SHASHANK_KUMAR_MISHR
cooked: <p>My code is below</p>
<p>loss_epoch_arr = []<br>
max_epochs = 5</p>
<p>min_loss = 1000</p>
<p>n_iters = np.ceil(50000/training_batchsize)</p>
<p>for epoch in range(max_epochs):</p>
<pre><code>for i, data in enumerate(train_loader, 0):

    inputs, labels = data
    inputs, labels = inputs.to(device), labels.to(device)

    opt.zero_grad()

    outputs, aux_outputs = inception(inputs)
    loss = loss_fn(outputs, labels) + 0.3 * loss_fn(aux_outputs, labels)
    loss.backward()
    opt.step()
    
    if min_loss &gt; loss.item():
        min_loss = loss.item()
        best_model = copy.deepcopy(inception.state_dict())
        print('Min loss %0.2f' % min_loss)
    
    if i % 100 == 0:
        print('Iteration: %d/%d, Loss: %0.2f' % (i, n_iters, loss.item()))
        
    del inputs, labels, outputs
    torch.cuda.empty_cache()
    
loss_epoch_arr.append(loss.item())
    
print('Epoch: %d/%d, Test acc: %0.2f, Train acc: %0.2f' % (
    epoch, max_epochs, 
    evaluation_inception(test_loader, inception), 
    evaluation_inception(train_loader, inception)))
</code></pre>
<p>plt.plot(loss_epoch_arr)<br>
plt.show()</p>
<p>Error occurred:-</p>
<p>RuntimeError:-Calculated padded input size per channel: (3 x 3). Kernel size: (5 x 5). Kernel size can’t be greater than actual input size</p>
<p>How to resolve it?</p>

username: ptrblck
cooked: <p>The Inception model expects inputs with a spatial size of <code>299x299</code>. If your input size is small than this, some activations might be too small regarding their spatial size, such that the kernel size is actually bigger, which yields this error.<br>
In that case, you could use <code>torchvision.transforms.Resize((299, 299)</code> to resize your input to the desired size.</p>

username: SHASHANK_KUMAR_MISHR
cooked: <p>yeah it worked like charm Thanks @ ptrbick</p>
Is F.conv_transpose2d transpose of F.conv2d?

username: ybj14
cooked: <p>For an input image x and a convolution layer w, output y = w(x)<br>
Let’s say we expand x as a giant column vector X and w as a giant matrix W so that Y=WX.<br>
Now we want to compute W^T(WX). Should we use F.conv_transpose2d or F.conv2d with tranposed weight matrix? These two operations give different values since the second assertion fails.</p>
<pre><code class="lang-auto">import torch
from torch import nn
import torch.nn.functional as F

i = torch.randn(16, 3, 32, 32)
w = torch.randn(3, 32, 5, 5)
o1 = F.conv2d(i, w.transpose(1, 0), padding=2)
o2 = F.conv_transpose2d(i, w, padding=2)

assert o1.shape == o2.shape, "shape doesn't match!"
assert torch.all((o1 - o2) &lt; 1e-3), "value doesn't equal!"
</code></pre>
<p>I found when the conv is 1*1 and padding=0 the second assertion passes. But using larger kernel and setting padding=(kernel_size-1)/2 to keep shape will cause failure. How to solve this problem?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The transpose convolution is the common name for the inverse of the convolution. So if you use the correct parameters for stride and padding, you should have <code>W^T(WX) = X</code>.</p>
Why use Parameter if tensor by default has grad on?

username: whoab
cooked: <p>Is it just to add it to the modules list of Params? So that is it backpropagated when you add module.params to optimizer?</p>

username: albanD
cooked: <p>Hi,</p>
<p>By default the grad_mode is on, but the Tensors you create do not require gradients.<br>
Also it is indeed used to automatically add to the list of parameters of the module.</p>
How to convert below tensor into numpy array in pytorch?

username: SHASHANK_KUMAR_MISHR
cooked: <p>How to convert below tensor  to numpy array in pytorch?<br>
[tensor([[ 0.1191, -0.0661],         [-0.2500,  0.0451],         [-0.0905,  0.1674],         [-0.0326,  0.3001],         [ 0.0547,  0.1601]], grad_fn=), tensor([[ 0.1737,  0.1025],         [-0.1487,  0.1500],         [ 0.1235,  0.1284],         [ 0.1825,  0.0397],         [-0.0554,  0.2457]], grad_fn=), tensor([[-0.1453, -0.1910],</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can use <code>your_tensor.numpy()</code> to convert it to numpy.<br>
Note that since numpy does not support GPU, you will need to send the tensor back to the cpu before converting it with <code>.cpu()</code>.<br>
Note as well that since numpy does not support autograd, if you tensor requires gradient, you will need to get a version that do not with <code>.detach()</code>.</p>
How to backward the derivative？

username: 11184
cooked: <p>how to backward if my loss is something about the derivative of the net output?<br>
for example:<br>
my input is ‘u’, then my net output can be seemed as f(u).<br>
and my target is target = a*( f ’ (u)|u ) + b,        " f ’ (u)|u" means the derivative of ’ f ’ w.r.t ‘u’<br>
so my loss may will be ‘| ground truth - target |’.<br>
Is this kind of loss can be backwarded in pytorch?<br>
I tried to backward something about the derivative, but it can’t work.<br>
here is my code :</p>
<p>import torch<br>
from torch.autograd import Variable<br>
x = Variable(torch.ones(2, 2), requires_grad=True)<br>
y = x + 2<br>
out = y<br>
out.backward(torch.ones(2, 2))<br>
k = 2*x.grad<br>
k.backward()</p>
<p>the error is :<br>
k.backward()<br>
RuntimeError: element 0 of variables tuple is volatile</p>
<p>it seems like pytorch will not save the parameters when it calculates the derivative.<br>
Has there any method to solve this kind of questions?<br>
Thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>For this to work you need to pass the <code>create_graph=True</code> option to the first <code>.backward()</code> to let it know that you need to be able to call <code>.backward()</code> on the grad itself.<br>
Also if you need the gradient wrt a specific variable, you can use the <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.grad"><code>autograd.grad(outputs, inputs)</code></a> function to get the derivative of the output(s) wrt the input(s). For example <code>out.backward()</code> is equivalent to <code>autograd.grad(out, model.parameters())</code>.</p>

username: 11184
cooked: <p>It works !!!<br>
Sorry for I haven’t read the document completely.<br>
Thank you !</p>

username: saurabheights
cooked: <p>Thank you, I have been looking for an example on how to use autograd api. IMO, a small example of using autograd.grad api should be added to pytorch doc. Furthermore, examples of higher-order derivatives would help as well. Say, for example Double Backpropagation, especially since <code>loss.backward(create_graph=True)</code> no longer works, causing memory leak due to some changes in C++ backend(weak to strong pointers), <a href="https://github.com/pytorch/pytorch/issues/4661" rel="nofollow noopener">issue page</a>.</p>
Implementation of convolutional auto encoder makes no progress on MNIST

username: rpathak1
cooked: <p>I tried to implement a convolutional auto encoder in pytorch and applied it on MNIST. Unfortunately the train loss makes almost no progress (it does not decrease) during training. Can someone please help? I have posted my code (&lt; 95 lines) at the following Github Gist: <a href="https://gist.github.com/reesepathak/8f7f708d18c7d5d5248d6736a2976ecf" rel="nofollow noopener">code is here</a>.</p>

username: ptrblck
cooked: <p>Hi <a class="mention" href="/u/rpathak1">@rpathak1</a>,</p>
<p>could you try to remove the normalization from the transformations in your <code>Dataset</code>?<br>
While this is generally a good idea for classification use cases, since it will standardize your data to have a mean of zero and a stddev of 1, this might cause some troubles for your autoencoder, since you are using a sigmoid at the output layer, which limits the output values to the range <code>[0, 1]</code>.<br>
Alternatively, you could try to remove the sigmoid and let your model learn the standardized inputs.</p>
<p>Let me know, if that helps.</p>
Semantics of structure of PyTorch model

username: BramVanroy
cooked: <p>I have a model that looks like this:</p>
<pre><code class="lang-auto">RegressionRNN(
  (ms_group): MorphosynGroup(
    (rlayer): GRU(147, 256, batch_first=True, bidirectional=True)
  )
  (ft_group): FastTextGroup(
    (rlayer): GRU(147, 256, batch_first=True, bidirectional=True)
  )
  (concat): Concatenator()
  (linear_layer): Linear(in_features=512, out_features=1, bias=True)
)
</code></pre>
<p>This is a model with two different kind of inputs. One goes into ms_group, the other goes into ft_group. The results of these groups are then concatenated, and on the concatenation the linear layer is then executed. I feel, though, that the model as represented above is not semantically clear about what is being concatenated. The concatenator class looks like this</p>
<pre><code class="lang-python">class Concatenator(nn.Module):
    def __init__(self):
        super(Concatenator, self).__init__()

    def forward(self, inputs):
        if len(inputs) &gt; 1:
            catted = torch.cat(inputs, dim=1)
        else:
            catted = inputs[0]

        return catted 
</code></pre>
<p>It is simple enough, but I wanted to put this in its separate group so that you can see from the model representation that somewhere a concatenation is taking place. I feel, though, that improvements are possible but I’m not sure how. Should I place the ms_group, ft_group, and concat in their own group (e.g. ‘encoder’), and the linear layer as the decoder layer?</p>
<p>The code is written in a highly modular way so that you can choose to only use ms_group, or only ft_group, and based on the used number of groups the results are concatenated or not, and so on. So I wish to make it more visibly clear what is going on but I’m not sure what the conventions in PyTorch are to do this.</p>
Caffe2 vs Torch performance

username: Benjamin_Navarro
cooked: <p>Hi, I recently thrown myself in the real of machine learning because I need some kind of non-linear function estimation. I ultimately need to run everything in C++ because it will be used for robot control at a frequency up to 1kHz.</p>
<p>I started  by building, training and testing a network with Python using Caffe2, then exported it so I could load it and run in C++. The C++ part was a bit painful because of the clear lack of documentation but I finally managed to get it working and was very pleased with the performance.</p>
<p>While struggling with Caffe2 C++ API, I found out that the PyTorch C++ one has some decent documentation, examples and seemed easier to use. So I rebuilt, trained and ran the same network architecture (everything directly in C++ this time) and I observed that it’s around 6x slower than the Caffe2 implementation.</p>
<p>To give some numbers, a prediction (on CPU) takes around 8us with Caffe2 and 46us with Torch. It still pretty fast but as I intend to expand my network architecture to add more variables and take and more parameters into account I fear that the difference in performance might become a problem.</p>
<p>Is this difference in performance between the two libraries is a known thing or should they both perform the same and the issue is probably with my code?</p>
<p>For the info, my network currently has 2 inputs, 1 output and 3 hidden fully connected layers with 50, 30 and 10 neurons and a tanh function after each layer.</p>
LSTM returns tuple

username: Avocet
cooked: <p>I am building a network and the LSTM layer is returning a tuple. I understand that I will need to split this to enable the next layer to work, however can’t figure out how to do this. My model looks as follows:</p>
<p>m = nn.Sequential(<br>
nn.Embedding(MAX_WORDS, 128),<br>
nn.LSTM(input_size=128, hidden_size=32, num_layers=1,bidirectional = True),<br>
nn.Linear(32, 128),<br>
nn.ReLU(),<br>
nn.Linear(128, 64),<br>
nn.ReLU(),<br>
nn.Linear(64, 3),<br>
nn.Sigmoid())</p>
<p>I need to split it after the LSTM - how do I do this? Appreciate the help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thank you!</p>

username: gaikin
cooked: <p>m1 = nn.Sequential(<br>
nn.Embedding(MAX_WORDS, 128),<br>
nn.LSTM(input_size=128, hidden_size=32, num_layers=1,bidirectional = True))</p>
<p>m2 = nn.Sequential(<br>
nn.Linear(<strong>64</strong>, 128),<br>
nn.ReLU(),<br>
nn.Linear(128, 64),<br>
nn.ReLU(),<br>
nn.Linear(64, 3),<br>
nn.Sigmoid())</p>
<p>output, (h,c) = m1(…)<br>
m2(output)</p>
<p><strong>Note</strong>: lstm will return double hidden_size dim if bidirectional is true, or you have to process outputs.</p>

username: Richard_Kahn
cooked: <p>What is supposed to be in the …?</p>
Windows build (out-of-source)

username: traits
cooked: <p>Disclaimer: I am new to source builds of pytorch.</p>
<p>I was able to build and install pytorch successfully, following <a href="https://pytorch.org/get-started/locally/#windows-from-source" rel="nofollow noopener">https://pytorch.org/get-started/locally/#windows-from-source</a> against a miniconda environment. Two questions remain for me:</p>
<ol>
<li>
<p>torchvision: Is this part of the package? I couldn’t found any egg/whl. Any knobs, I am not aware yet?</p>
</li>
<li>
<p>Out-of-Source builds gives me the following error:</p>
</li>
</ol>
<pre><code class="lang-auto">Building wheel torch-1.1.0a0
-- Building version 1.1.0a0
[0/1] cmd.exe /C "cd /D V:\src\git\pytorch\build\build &amp;&amp; C:\Users\micha\Miniconda3\Library\bin\cmake.exe -P cmake_install.cmake"
-- Install configuration: "Release"
Traceback (most recent call last):
  File "..\src\pytorch\setup.py", line 734, in &lt;module&gt;
    build_deps()
  File "..\src\pytorch\setup.py", line 299, in build_deps
    build_dir='build')
  File "v:\src\git\pytorch\src\pytorch\tools\build_pytorch_libs.py", line 325, in build_caffe2
    shutil.copyfile(proto_file, "caffe2/proto/" + os.path.basename(proto_file))
  File "C:\Users\micha\Miniconda3\lib\shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'caffe2/proto/caffe2_legacy_pb2.py'
</code></pre>
<p>I am not very well-versed in python’s setup.py framework, so it is easily possible that I make very simple errors here. My call to setup.py comes from my intended build directory for cmake and has the form:</p>
<pre><code class="lang-auto">python ..\src\pytorch\setup.py install
</code></pre>
<p>Are there any options, providing setup.py with the correct build directory in order to transfer this information to cmake? (The mentioned caffe2/proto/caffe2_legacy_pb2.py was copied to this build directory - but the leading path seems to be missing).</p>
<p>Unrelated: Would a ‘build’ category possible for this forum (similar to ‘deployment’)?</p>
<p>Michael</p>

username: peterjc123
cooked: <ol>
<li>No, you will need to install it seperately.</li>
<li>Out-of-source builds are not supported. Please do this as an alternative.</li>
</ol>
<pre><code class="lang-auto">cd ..\src\pytorch\
python setup.py install
</code></pre>
Inserting pre-trained network to Siamese Network

username: pnambiar
cooked: <p>I am new to deep learning as well as pytorch. I have got the following code (siamese network) working with pytorch:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/KrisG04/Siamese-using-PyTorch/blob/master/Siamese-networks-medium.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/KrisG04/Siamese-using-PyTorch/blob/master/Siamese-networks-medium.ipynb" target="_blank" rel="nofollow noopener">KrisG04/Siamese-using-PyTorch/blob/master/Siamese-networks-medium.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One Shot Learning with Siamese Networks\n",
    "\n",
    "This is the jupyter notebook that accompanies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "All the imports are defined here"
   ]
  },
  {
</code></pre>

  This file has been truncated. <a href="https://github.com/KrisG04/Siamese-using-PyTorch/blob/master/Siamese-networks-medium.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Now I would like to replace the CNN with a pretrained network (trained on places dataset):<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/CSAILVision/places365" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/1420020?s=400&amp;v=4" class="thumbnail onebox-avatar" width="240" height="240">

<h3><a href="https://github.com/CSAILVision/places365" target="_blank" rel="nofollow noopener">CSAILVision/places365</a></h3>

<p>places365 - The Places365-CNNs for Scene Classification</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>How do I do this using pytorch</p>

username: n0obcoder
cooked: <p>did u train your siamese net from scratch? without using any pretrained weights?</p>
How many image pairs do i need to train a Siamese network?

username: n0obcoder
cooked: <p>I am trying to train a Siamese network. I am using <strong>att_faces dataset</strong> ,  which has 40 face IDs with 10 face images each for each face ID. I am using <strong>Triplet Loss</strong>  And i am using resnet18 pretrained weights. But the model doesnt seem to learn much on the training set. Any suggestions?</p>
<p>I have 2 major doubts.</p>
<ol>
<li>
<p>Is att_faces dataset (40x10  = 400 images) sufficient to train a Siamese network (from scratch or even by using pretrained resnet18) ?</p>
</li>
<li>
<p>What margin should i use for the Triplet Loss?</p>
</li>
</ol>
<p>You can check my code <a href="https://www.kaggle.com/n0obcoder/siamese-net/edit/run/15169681" rel="nofollow noopener">here</a></p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Data must be a sequence (got numpy.int64)?

username: deeplearner93
cooked: <p>state = Tensor(state).to(device)</p>
<p>It shows that particular error whenever it tries to execute the above line. Can anyone please explain what is going on here and how to rectify the problem.</p>
<p>Thanks a lot!</p>

username: ptrblck
cooked: <p>Could you try to run</p>
<pre><code class="lang-python">state = torch.from_numpy(state)
</code></pre>

username: deeplearner93
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="20697">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>state = torch.from_numpy(state)</p>
</blockquote>
</aside>
<p>expected np.ndarray (got numpy.int64)<br>
^ I get the above error after running the line you suggested</p>

username: ptrblck
cooked: <p>Yeah, you are right.<br>
You have to pass a valid <code>np.array</code> to the function, so</p>
<pre><code class="lang-python">state = torch.from_numpy(np.array(state))
</code></pre>
<p>should work.</p>

username: huangbiubiu
cooked: <p>Try <code>torch.as_tensor()</code>:</p>
<pre><code class="lang-auto">print(torch.as_tensor(1.))
</code></pre>
Calculating AUROC and Accuracy

username: ousan
cooked: <p>Hello dear all,</p>
<p>I have two different classes(binary classification) and i am trying to calculate AUROC, Accuracy and plot ROC. However, I could not understand clearly how ROC graphs are plotted. I am sharing the code and network with you.</p>
<p>NETWORK:(class Net(nn.Module))</p>
<pre><code>def __init__(self):
    super(Net, self).__init__()
    self.pool = nn.MaxPool2d(2, 2)
    self.conv1 = nn.Conv2d(3, 10, 5)
    self.conv2 = nn.Conv2d(10, 20, 5)
    self.fc1 = nn.Linear(20 * 5 * 5, 256)
    self.fc2 = nn.Linear(256, 2)

def forward(self, x):
    # print x.size()
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 20*5*5) # flatten the sensor
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x
</code></pre>
<p>TRAIN:(def train())</p>
<pre><code>running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    inputs, labels = data
    inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())
    optimizer.zero_grad()

    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
</code></pre>
<p>TEST:(def test())</p>
<pre><code>for data in testloader:
    images, labels = data
    outputs = net(Variable(images.cuda()))
    _, predicted = torch.max(outputs.data, 1)
    c = (predicted == labels.cuda()).squeeze().cpu().numpy()
    y_score.append(outputs.cpu())
    y.append(labels.cpu().numpy())
</code></pre>
<p>Above, I am trying to add y and y_scores to array. Then I am calculating roc and accuracy like below. In the above code “y_score.append(outputs.cpu())” this line give an error. But my main problem is not actually this. As I said before, I could not be sure whether this method is true or not when determining auroc.</p>
<blockquote>
<p>fpr, tpr, _ = roc_curve(y, y_score)<br>
roc_auc = auc(fpr, tpr)<br>
print "average_precision_score : " + str(average_precision_score(y, y_score))<br>
print "roc_auc_score : " + str(roc_auc_score(y, y_score))<br>
print "accuracy : " + str(accuracy_score(y, y_score))</p>
</blockquote>
<p>Thank you all</p>

username: SHASHANK_KUMAR_MISHR
cooked: <p>I think u have not declared  y_score=[],y=[]</p>
How to modify the below function to get Accuracy and AUC?

username: SHASHANK_KUMAR_MISHR
cooked: <p>def evaluation(dataloader, model):<br>
total, correct = 0, 0<br>
for data in dataloader:<br>
inputs, labels = data<br>
inputs, labels = inputs.to(device), labels.to(device)<br>
outputs = model(inputs)<br>
_, pred = torch.max(outputs.data, 1)<br>
total += labels.size(0)<br>
y_score.append(outputs.cpu())<br>
y.append(labels.cpu().numpy())<br>
correct += (pred == labels).sum().item()<br>
return 100 * correct / total</p>
Tensorboard Projector: Invalid checkpoint state loaded

username: tarrencev
cooked: <p>I’m attempting to generate an embedding visualization using PyTorch 1.1.0. I’m following the example in the <a href="https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_embedding" rel="nofollow noopener">documentation</a> but running into an issue. When I try loaded the “Projector” component in Tensorboard, the backend returns a 500 and logs the following error:</p>
<pre><code class="lang-auto">I0605 22:51:01.434675 140375100487424 _internal.py:97] ::ffff:172.17.13.114 - - [05/Jun/2019 22:51:01] "GET /data/plugin/projector/runs HTTP/1.1" 500 -
E0605 22:51:01.438719 140375100487424 _internal.py:97] Error on request:
Traceback (most recent call last):
  File "/home/tvanas/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 302, in run_wsgi
    execute(self.server.app)
  File "/home/tvanas/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 290, in execute
    application_iter = app(environ, start_response)
  File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/backend/application.py", line 309, in __call__
    return self.data_applications[clean_path](environ, start_response)
  File "/home/tvanas/.local/lib/python3.6/site-packages/werkzeug/wrappers/base_request.py", line 235, in application
    resp = f(*args[:-2] + (request,))
  File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/plugins/projector/projector_plugin.py", line 471, in _serve_runs
    return Respond(request, list(self.configs.keys()), 'application/json')
  File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/plugins/projector/projector_plugin.py", line 320, in configs
    _latest_checkpoints_changed(self._configs, run_path_pairs)):
  File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/plugins/projector/projector_plugin.py", line 181, in _latest_checkpoints_changed
    ckpt_path = _find_latest_checkpoint(logdir)
  File "/home/tvanas/.local/lib/python3.6/site-packages/tensorboard/plugins/projector/projector_plugin.py", line 663, in _find_latest_checkpoint
    ckpt_path = tf.train.latest_checkpoint(os.path.join(dir_path, os.pardir))
  File "/home/tvanas/.local/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_management.py", line 336, in latest_checkpoint
    ckpt = get_checkpoint_state(checkpoint_dir, latest_filename)
  File "/home/tvanas/.local/lib/python3.6/site-packages/tensorflow/python/training/checkpoint_management.py", line 278, in get_checkpoint_state
    + checkpoint_dir)
ValueError: Invalid checkpoint state loaded from /home/.../results/scannet_poster/logs/..
</code></pre>
<p>Anyone got any pointers on how to resolve this? I’m not sure why Tensorboard is looking for a checkpoint.</p>
<p>Output of <code>pip list|grep tensor</code> is:</p>
<pre><code class="lang-auto">tensorboard                      1.13.1
tensorboardX                     1.6
tensorflow                       2.0.0a0
tensorflow-estimator-2.0-preview 1.14.0.dev2019060500
</code></pre>
Trouble flexibly implementing a tensor manipulation

username: Brent_Werness
cooked: <p>Hello,</p>
<p>I’m trying to figure out how to do the following operation in general (and if it has a name).  I am given a list of tensors <code>[X_1,X_2,...,X_k]</code> all of which are of shape n by m.  I want to do the following operation:</p>
<pre><code class="lang-auto">T_{i_1,...,i_k} = \sum_m [X_1]_{m,i_1}*[X_2]_{m,i_2} *...* [X_k]_{m,i_k}.
</code></pre>
<p>I can do this easily for a fixed number of tensors by doing:</p>
<pre><code class="lang-python">torch.einsum('ni,nj,nk,nl -&gt; ijkl', [X,Y,Z,W])
</code></pre>
<p>However, this is not satisfactory for unknown length lists.  Any thoughts on how to implement this well?</p>

username: LeviViana
cooked: <p>Since the output is a tensor having <code>m ** k</code> elements, you actually won’t be able to make it work for any <code>m</code> and <code>k</code> values, you’ll allocate a lot of memory even for small values of <code>m</code> and <code>k</code> (for instance, 37 Gb for <code>m = k = 10</code> if your tensor uses 32 bit floating points).</p>
<p>This being said, you can analyze your use case and see how big can <code>k</code> be. Lets say if <code>k &lt; 26</code>, then you can create a function that generates the arguments for your <code>torch.einsum</code> call, using all the lower case letters (which happen to be the only ones supported by PyTorch for the moment).</p>
<p>A side note about <code>torch.einsum</code> compared to <code>numpy.einsum</code> is that numpy supports lists of indices too, making it possible to handle Einstein Summation for any number of tensors easily. Here is an illustration:</p>
<pre><code class="lang-python">import numpy as np

x = np.random.rand(6).reshape(2, 3)
y = np.random.rand(6).reshape(2, 3)
z = np.random.rand(6).reshape(2, 3)
w = np.random.rand(6).reshape(2, 3)
r1 = np.einsum(x, [0, 1], y, [0, 2], z, [0, 3], w, [0, 4], [1, 2, 3, 4]) # Not currently supported by PyTorch
r2 = np.einsum('ni,nj,nk,nl -&gt; ijkl', x, y, z, w) # Supported by PyTorch
print((r1 == r2).all())  # Prints True
</code></pre>

username: Brent_Werness
cooked: <p>Thanks!  A common m will be 2, so going up to 20 or so is not implausible.  I am also aware of the fact I could do this easily in numpy, but then I’ll be without all the niceties (automatic differentiation, pre-made optimizers, etc) that make torch friendly.</p>
<p>I was indeed leaning this way (at the end of the day it is still only 26 lines of code), but I felt that there needed to be a better way to do this!</p>

username: LeviViana
cooked: <p>I actually opened an <a href="https://github.com/pytorch/pytorch/issues/21412" rel="nofollow noopener">issue</a> in PyTorch’s GitHub about that subject. It looks like a PR implementing the support of sublist argument (i.e. like numpy) is welcome.</p>

username: Brent_Werness
cooked: <p>Thanks! I posted there with some additional context!</p>
Get derivatives from your network

username: smu226
cooked: <p>Hello! I have this (simplified version) code:</p>
<pre><code class="lang-auto">grads = {}
def save_grad(name):
    def hook(grad):
        grads[name] = grad
    return hook

class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(2, 1,  bias=False)
        self.linear2 = nn.Linear(1, 2,  bias=False)
            
    def forward(self, x):
        z = self.linear1(x)
        y_pred = self.linear2(z)

        return y_pred, z

for epoch in range(1000):
    model.train()
    for i, dt in enumerate(data.trn_dl):
        optimizer.zero_grad()
        output = model(dt[0])
        
        output[1].register_hook(save_grad('z_x_hat'))
        output[0][0].backward(retain_graph=True)
        z_x_hat = grads['z_x_hat']
        
        output[1].register_hook(save_grad('z_y_hat'))
        output[0][1].backward(retain_graph=True)
        z_y_hat = grads['z_y_hat']
        
        z_x_hat.requires_grad = True
        z_y_hat.requires_grad = True
                                     
        loss = abs(torch.sqrt(z_x_hat**2+z_y_hat**2)-1)
        print(z_x_hat,z_y_hat,loss)
        loss.backward()
        
        optimizer.step()
</code></pre>
<p>So my network has 2 outputs (x_hat, y_hat) and I want the loss function to be given by the derivative of the outputs with respect to the value of z (I need this as part of something more complex for the actual project, but this is the part where I am stuck). In this simple case, the derivatives are just the value of the weights from z to x_hat and z to y_hat (in the real case is the product of all the partial derivatives from x_hat to z). Using the code above:</p>
<pre><code class="lang-auto">output[1].register_hook(save_grad('z_x_hat'))
output[0][0].backward(retain_graph=True)
z_x_hat = grads['z_x_hat']
</code></pre>
<p>I am able to get the value of this derivative (and hence the weight), but the problem is that I need to use <code>retain_graph=True</code> in order to do that, and that screws up my whole backprop, as I want to optimize the network using backprop only on the loss at the end, not on these intermediate calls. Can someone help me with this? Thank you!</p>
Modify ResNet50 to give Multiple Outputs

username: AmardeepGanguly
cooked: <p>I am dealing with a multi-label classification problem ,the image belongs to  one of the<br>
10 classes from two distinct labels i.e desired output is [batch_size,2,10],how can i modify ResNet50  to Get Multiple outputs</p>

username: ptrblck
cooked: <p>If I understand your use case correctly you are dealing with two “labels”, each consisting of 10 classes.<br>
Each sample belongs to one particular class of each label.</p>
<p>I think in that particular use case you could use two linear layers, one for each label, and return these two outputs:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, num_classes1, num_classes2):
        super(MyModel, self).__init__()
        self.model_resnet = models.resnet18(pretrained=True)
        num_ftrs = self.model_resnet.fc.in_features
        self.model_resnet.fc = nn.Identity()
        self.fc1 = nn.Linear(num_ftrs, num_classes1)
        self.fc2 = nn.Linear(num_ftrs, num_classes2)

    def forward(self, x):
        x = self.model_resnet(x)
        out1 = self.fc1(x)
        out2 = self.fc2(x)
        return out1, out2
</code></pre>

username: AmardeepGanguly
cooked: <p>Thanks so much ,really helped a lot</p>

username: AmardeepGanguly
cooked: <p>I am getting the following error<br>
AttributeError: module ‘torch.nn’ has no attribute ‘Identity’</p>

username: ptrblck
cooked: <p>Probably you are using an older PyTorch version, as the <code>nn.Identity</code> module was introduced in <code>1.1.0</code>.<br>
Have a look <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">here</a> for install instructions.</p>

username: AmardeepGanguly
cooked: <p>I used nn.Sequential() it worked fine ,actually i am implementing all this in Kaggle ,the pytorch version must be an older one.Thanks Peter for all the help I was stuck with this for a long time:yum:</p>

username: sai_tharun
cooked: <p>Hey can you tell me which loss function and optimizer did you use and did you freeze the other layers??</p>

username: AmardeepGanguly
cooked: <p>Sure , I am using SGD with learning_rate of 0.20 .It is working fine ,try this .Hope it helps you .<img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
How to implement last layer of SkipGram model?

username: SamRoss
cooked: <p>Hello, I am trying to modify the code from <a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html" rel="nofollow noopener">here</a> to implement skipgram. I’m having trouble implementing the last layer, where we should have as many softmax’s as we have in our context size. My code works for a context size of 1, since this is just a usual softmax layer, but how do we extend this to a context window larger than 1? Here is my modified code:</p>
<pre><code class="lang-auto">
CONTEXT_SIZE = 2
EMBEDDING_DIM = 10
# We will use Shakespeare Sonnet 2
test_sentence = """When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
Thy youth's proud livery so gazed on now,
Will be a totter'd weed of small worth held:
Then being asked, where all thy beauty lies,
Where all the treasure of thy lusty days;
To say, within thine own deep sunken eyes,
Were an all-eating shame, and thriftless praise.
How much more praise deserv'd thy beauty's use,
If thou couldst answer 'This fair child of mine
Shall sum my count, and make my old excuse,'
Proving his beauty by succession thine!
This were to be new made when thou art old,
And see thy blood warm when thou feel'st it cold.""".split()
# we should tokenize the input, but we will ignore that for now
# build a list of tuples.  Each tuple is ([ word_i-2, word_i-1 ], target word)
trigrams = [([test_sentence[i], test_sentence[i+1]], test_sentence[i + 2])
            for i in range(len(test_sentence) - 2)]
# print the first 3, just so you can see what they look like
#print(trigrams[:3])

vocab = set(test_sentence)
word_to_ix = {word: i for i, word in enumerate(vocab)}


class MySkipGram(nn.Module):

    def __init__(self, vocab_size, embedding_dim, context_size):
        super(MySkipGram, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size*context_size)

    def forward(self, inputs):
        embeds = self.embeddings(inputs).view((1, -1))
        #out = F.relu(self.linear1(embeds))
        out = self.linear1(embeds)
        out = self.linear2(out)
        log_probs = F.log_softmax(out, dim=1)
        return log_probs


losses = []
loss_function = nn.NLLLoss()
model = MySkipGram(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)
optimizer = optim.SGD(model.parameters(), lr=0.001)

for epoch in range(100):
    total_loss = 0
    for context, target in trigrams:

        context_idxs = torch.tensor([word_to_ix[w] for w in context], dtype=torch.long)
        target_idx = torch.tensor([word_to_ix[target]], dtype=torch.long)
        model.zero_grad()

        log_probs = model(target_idx)

        loss = loss_function(log_probs, context_idxs)

        loss.backward()
        optimizer.step()

        total_loss += loss.item()
    losses.append(total_loss)
print(losses)  # The loss decreased every iteration over the training data!
</code></pre>
<p>How can I have several softmax layers at the end that work “in parallel”?</p>
RuntimeError: leaf variable has been moved into the graph interior

username: Malvina_Nikandrou
cooked: <p>Hi all,</p>
<p>I have a piece of code that performs a transformation on the input tokens of a network that looks like this:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    def __init__(self, val, max=15000, step=50, device='cpu'):
        super(MyModule, self).__init__()
        self.device = device
        self.tt = torch.arange(0, max, step=step, device=device, requires_grad=False).unsqueeze(0).long()
        self.val = val

    def forward(self, x, st):
        st = st.unsqueeze(-1).repeat(1, 1, self.tt.size(1)).long()
        mask = (st &lt;= self.tt).long()
        out = x.clone().detach()  # this doesn't work
        out = out.unsqueeze(-1).repeat(1, 1, self.tt.size(1))
        out = (out * mask).sum(1)
        out[out == 0] = self.val
        return out, mask.long()

</code></pre>
<p>The problem is that when backward is called I get the following error:</p>
<pre><code class="lang-auto">    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/lib/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: leaf variable has been moved into the graph interior

</code></pre>
<p>From what I have read it is related to in place modifications of variables, but I can’t seem to find a solution.</p>
<p>Any suggestions?</p>
<p>Thanks</p>