akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Accessing last layer in torchvision model in a general way

Tom_F: <p>Is there an easy way to access the last layer in a torchvision model, without coding a special version for each type of model as described in this tutorial: <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Finetuning Torchvision Models — PyTorch Tutorials 1.2.0 documentation</a></p>
<p>Ie. the problem is that the last layer has different names for different pretrained models, like “fc” for ResNet, “classifier” for DenseNet etc. I tried to access the layer like accessing an OrderedDict (d.items()…) but didn’t succeed.</p>

ptrblck: <p>I don’t believe there is a safe way to get the last layer, since even indexing the last module from <code>model.children()</code> would return the module initialized last and not necessarily the layer used last in the <code>forward</code>. Since your model could use different execution path in the <code>forward</code> (e.g. Inception models return either the final logits or additionally the aux. logits as well), the cleanest way would be to check the <code>forward</code> of each model and make sure you are indeed replacing the “last” layer.</p>

Tom_F: <p>Thanks for the quick reply! I hadn’t thought about the extra complexity of different forward methods.</p>
How to change the size of tensor

StephDoc: <p>Dear all,</p>
<p>I have the following tensor:<br>
y = (torch.rand(size=(1,81)) &lt; 0.25).long()</p>
<p>how can I change the size to # Out[288]: torch.Size([81])<br>
?</p>

ptrblck: <p><code>y</code> will have the shape <code>torch.Size([1, 81])</code>. To remove <code>dim0</code> you could use:</p>
<pre><code class="lang-python">y = (torch.rand(size=(1,81)) &lt; 0.25).long().squeeze(0)
</code></pre>

StephDoc: <p>Great! Thanks a lot!</p>
Relative (percentage) Loss without target tensor

electric93: <p>Hello!<br>
I was wondering how someone would implement a loss function that uses a percentage value of “how far is the predicted tensor from the target tensor”, without knowing the actual target. I tried implementing fake target tensors but I guess the math behind that is not correct, since what I tried was subtracting the percentage from my predicted tensor. Meaning the target tensor would constantly change in proportion to the predicted tensor, which seems wrong to me. But how would I go differently about that? I also tried replacing the value of the predicted tensor with the percentage, but since the percentage calculation is not integrated into the loss calculation, the loss is constantly rising to infinity (at least this is what I believe). All help is appreciated, since I’m very new to pytorch!</p>
<p>Have a lovely day, everyone!</p>
Why set cuda(non_blocking=False) for target variables?

laoreja: <p>In this example: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a> and some other places, they set the non_blocking in cuda(device=None, non_blocking=False) to be True for the target variable only.</p>
<pre><code>    target = target.cuda(True)
</code></pre>
<p>Why do they do this? And why this is only set to be True for the target Variable, but not the input Variable, in the above example, both the train and test data loaders’ pin_memory is set to be True. (I cannot understand the pin_memory setting in torch.utils.data.DataLoader as well)</p>
<pre><code>train_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
    num_workers=args.workers, pin_memory=True, sampler=train_sampler)

val_loader = torch.utils.data.DataLoader(
    datasets.ImageFolder(valdir, transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        normalize,
    ])),
    batch_size=args.batch_size, shuffle=False,
    num_workers=args.workers, pin_memory=True)</code></pre>

samster25: <p>Non-Blocking allows you to overlap compute and memory transfer to the GPU. The reason you can set the target as non-blocking is so you can overlap the compute of the model and the transfer of the ground-truth. If you set the input to also non-blocking, it would yield no benefit due to the fact that the model has a dependency on the input data.</p>
<p>Pinned Memory allows the non-blocking calls to actually be non-blocking.</p>

Omar_AlSuwaidi: <p>Hey thanks for the thorough explanation.</p>
<p>Could you please clarify what do you mean by “overlap <strong>compute</strong> and memory transfer to the GPU.”? I understand what’s referred to here by “memory transfer” (asynchronously) , but what does the “compute” part refer to here?</p>
<p>Does that mean that the model (kernel) will start its execution process on the input data while the target is being transferred to GPU memory?</p>
Recommendations for Multi-GPU training options

JamesDickens: <p>I have completed many projects with PyTorch using my own desktop computer, with a good GPU/CPU setup. I’m interested in taking things to the next level, but I have no experience in this matter. If anyone has thoughts/recommendations for resources to explore:</p>
<ol>
<li>
<p>Training with multiple GPUs on some service like AWS with the use of MY OWN pre-defined directory structure with dataset files  (images, annotations, etc.)</p>
</li>
<li>
<p>Inference on a server after training</p>
</li>
<li>
<p>Thoughts about which versions of PyTorch and other Python packages like Numpy, PIL, etc. I would be able to access with such a proposed setup</p>
</li>
</ol>
<p>Any thoughts/insights/resources much appreciated.</p>
Spatial transformation network not outputting correct tensor

stuffedfrog27: <p>Hello,</p>
<p>I am working on implementing a spatial transformer network (somewhat following <a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Spatial Transformer Networks Tutorial — PyTorch Tutorials 1.9.0+cu102 documentation</a>). The problem is that my network outputs the same tensor with every input (a tensor of all 0s). When I use <code>affine_grid</code> and <code>grid_sample</code> as a standalone and not within the network, I get a correct output. As such, it seems that the problem arises within the network itself. The localization network which regresses the affine matrix parameters works fine. My code for the network is below.</p>
<pre><code class="lang-auto">class stl_net(nn.Module):
    def __init__(self):
        super(stl_net, self).__init__()
        
        # localization network 
        # encoder
        self.enc1 = nn.Linear(in_features=784*2, out_features=256)
        self.enc2 = nn.Linear(in_features=256, out_features=128)
        self.enc3 = nn.Linear(in_features=128, out_features=64)
        self.enc4 = nn.Linear(in_features=64, out_features=32)
        self.enc5 = nn.Linear(in_features=32, out_features=16)
        # decoder 
        self.dec1 = nn.Linear(in_features=16, out_features=32)
        self.dec2 = nn.Linear(in_features=32, out_features=64)
        self.dec3 = nn.Linear(in_features=64, out_features=128)
        self.dec4 = nn.Linear(in_features=128, out_features=256)
        self.dec5 = nn.Linear(in_features=256, out_features=6) # 2x3 affine matrix 

    def forward(self, fixed_img, moving_img):
        # localization network 
        # encoder
        concat_np = np.concatenate((fixed_img, moving_img), axis=None)
        x = torch.tensor(concat_np, dtype=torch.float32)
        x = F.relu(self.enc1(x))
        x = F.relu(self.enc2(x))
        x = F.relu(self.enc3(x))
        x = F.relu(self.enc4(x))
        x = F.relu(self.enc5(x))
        # decoder
        x = F.relu(self.dec1(x))
        x = F.relu(self.dec2(x))
        x = F.relu(self.dec3(x))
        x = F.relu(self.dec4(x))
        
        # affine matrix 
        theta = F.relu(self.dec5(x)) 
        theta = theta.view(-1, 2, 3) 
        # displacement field 
        grid_shape = ((fixed_img.unsqueeze(0)).unsqueeze(0)).size()
        displacement_grid = F.affine_grid(theta, grid_shape)
        # warp 
        moving_img = torch.tensor(moving_img, dtype=torch.float32)
        moving_img = moving_img.unsqueeze(0)
        moving_img = moving_img.unsqueeze(0)
        deformed_image = F.grid_sample(moving_img, displacement_grid)
        deformed_image_reshaped = torch.squeeze(deformed_image)
        
        return deformed_image_reshaped
</code></pre>
<p>Also, the inputs are MNIST images.</p>
<p>Thanks!</p>
Dropout in RNN for each layer

akurniawan: <p>I’m currently reading c++ implementation of RNN in ATen (<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/RNN.cpp" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/RNN.cpp</a>). Based on the following implementation, is my understanding correct that in every layer there will be dropout applied with the same mask for every steps?</p>
<pre><code class="lang-auto">template&lt;typename io_type, typename hidden_type, typename weight_type&gt;
LayerOutput&lt;io_type, std::vector&lt;hidden_type&gt;&gt;
apply_layer_stack(const Layer&lt;io_type, hidden_type, weight_type&gt;&amp; layer, const io_type&amp; input,
                  const std::vector&lt;hidden_type&gt;&amp; hiddens, const std::vector&lt;weight_type&gt;&amp; weights,
                  int64_t num_layers, double dropout_p, bool train) {
  AT_CHECK(num_layers == hiddens.size(), "Expected more hidden states in stacked_rnn");
  AT_CHECK(num_layers == weights.size(), "Expected more weights in stacked_rnn");

  auto layer_input = input;
  auto hidden_it = hiddens.begin();
  auto weight_it = weights.begin();
  std::vector&lt;hidden_type&gt; final_hiddens;
  for (int64_t l = 0; l &lt; num_layers; ++l) {
    auto layer_output = layer(layer_input, *(hidden_it++), *(weight_it++));
    final_hiddens.push_back(layer_output.final_hidden);
    layer_input = layer_output.outputs;

    if (dropout_p != 0 &amp;&amp; train &amp;&amp; l &lt; num_layers - 1) {
      layer_input = dropout(layer_input, dropout_p);
    }
  }

  return {layer_input, final_hiddens};
}
</code></pre>

cwings316: <p>I had the same question as you, until I saw this post：<a href="https://discuss.pytorch.org/t/dropout-in-nn-rnn/2789" class="inline-onebox">Dropout in nn.RNN</a></p>
RuntimeError: The expanded size of the tensor must match the existing size

Mendel123: <p>I have a tensor with shape [2,3] and I want to expand it to [4,3].<br>
If I use <code>torch.expand</code> directly, a error occurs.</p>
<pre><code class="lang-python">&gt;&gt;&gt;x
tensor([[4., 5., 6.],
        [1., 2., 3.]])
&gt;&gt;&gt;x.size()
torch.Size([2, 3])
&gt;&gt;&gt;x.expand(4,3)
RuntimeError: The expanded size of the tensor (4) must match the existing size (2) at non-singleton dimension 0.  Target sizes: [4, 3].  Tensor sizes: [2, 3]
</code></pre>
<p>However, if I add a new axis before expanding it, it works.</p>
<pre><code class="lang-python">&gt;&gt;&gt;x[None,...].expand(2,-1,-1).reshape(4,3)
tensor([[4., 5., 6.],
        [1., 2., 3.],
        [4., 5., 6.],
        [1., 2., 3.]])
</code></pre>
<p>Does it mean we can only expand the axis with size 1?</p>

ptrblck: <aside class="quote no-group" data-username="Mendel123" data-post="1" data-topic="105523">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mendel123/48/11639_2.png" class="avatar"> Mendel123:</div>
<blockquote>
<p>Does it mean we can only expand the axis with size 1?</p>
</blockquote>
</aside>
<p>Yes, from the <a href="https://pytorch.org/docs/stable/tensors.html?highlight=expand#torch.Tensor.expand">docs</a>:</p>
<blockquote>
<p>Returns a new view of the <code>self</code> tensor with singleton dimensions expanded to a larger size.<br>
[…]<br>
Any dimension of size 1 can be expanded to an arbitrary value without allocating new memory.</p>
</blockquote>
<p>You could use <code>x.repeat(2, 1)</code> for your use case. Note that this operation would allocate new memory for the tensor.</p>

Syzygianinfern0: <blockquote>
<p>Note that this operation would allocate new memory for the tensor</p>
</blockquote>
<p>Any way to work around it?</p>

ptrblck: <p>No, you cannot <code>expand</code> a dimension with a different size than 1 and would need to create a new tensor.</p>
Loss backward is slower than expected when there are multiple models in the gpu

aminshabani: <p>Hi,</p>
<p>I have several small models all in the same GPUs (used by nn.DataParallel) and I am training them in two ways:</p>
<ul>
<li>Forward one model and backward the loss:</li>
</ul>
<pre><code class="lang-auto">output = models[0](x)
loss = mse_loss(output, target)
optimizer[0].zero_grad()
tic = time.time()
loss.backward()
print(time.time()-tic)
optimizer[0].step()
</code></pre>
<ul>
<li>Forward one model, forward other models with torch.no_grad(), and backward the loss:</li>
</ul>
<pre><code class="lang-auto">output = models[0](x)
with torch.no_grad():
    tmp_output = [models[i](x) for i in range(100)] 
loss = mse_loss(output, target)
tic = time.time()
loss.backward()
print(time.time()-tic)
optimizer[0].step()
</code></pre>
<p>Considering that the loss function is independent of the tmp_output, I believe that the execution time of the “loss.backward()” should be the same for both, however, the second one is significantly slower. I was wondering if I am missing something in here? I have also checked the generated graphs using torchviz and the graphs of the loss functions are the same for both.</p>

arman-yekkehkhani: <p>Hi, how did you measure the execution time of <code>loss.backward()</code>? You have done a considerable amount of computation in <code>[models[i](x) for i in range(100)]</code>, which I guess is the source of longer execution time.</p>

aminshabani: <p>Thank you for your reply, Yes this part needs to be clear, I only consider the time for the line of loss.backward(). I will edit the post.</p>

arman-yekkehkhani: <p>I tried a code on both CPU and a single GPU, yet the results were as expected. Both cases take the same amount of time. However, I should point that the difference you see might be due to the first-time allocation of tensors. Besides, your method for measuring execution time is not good. Try something like <code>timeit</code>. If the issue persists, please leave a detailed demo of your code(you can replace the model with a simple network and use arbitrary input and target), plus the output.</p>
Logit clipping in pointer network

israRe: <p>Hi,<br>
I’ve implemented a pointer network of <a href="https://arxiv.org/abs/1611.09940" rel="noopener nofollow ugc">NEURAL COMBINATORIAL OPTIMIZATION WITH REINFORCEMENT LEARNING</a>, it’s almost all okay, but logit clipping section if I implement that feature the output gets saturated and trimmed by the hyperbolic function.  So all the outputs gets the same value. Any suggest?.<br>
If I deactivate this function all ok, but the output is extremely overconfident.</p>
<p>An example would be a tensor with the values:<br>
[15,13,12.1,12,40]<br>
after apply logit clipping it’s saturated:<br>
[10,10,10,10,10]</p>
<pre><code class="lang-auto"># logits clipping
# self.C is a constant equal to 10 in the  paper
vector_pointer = self.C*torch.tanh(vector_pointer)

</code></pre>

arman-yekkehkhani: <p>I think it is the expected behavior. The purpose of introducing  <code>const * tanh(u)</code> is to increase the exploration, as mentioned in the paper. By applying a softmax on these numbers, you get the same <code>A</code> for all of them. Which can motivate the algorithm to explore more.</p>

israRe: <p>Oh, I think you’re right, it’s only used in training and deactivate in test mode. But it was something weird  when I saw it.</p>
Supporting Pytorch for Custom Compiler Backend

Darshan: <p>Hi Pytorch Team,</p>
<p>We have Custom Deep Learning hardware and a Compiler for the Hardware. For now, we have only Caffe framework support.</p>
<p>Having said that, we want to work on adding support for the Pytorch framework.</p>
<p>Is there any reference for the same?</p>
<p>Thanks,<br>
Darshan C G</p>

garymm: <p>You might look into:</p>
<ol>
<li>
<p>Convert from PyTorch to ONNX using <a href="https://pytorch.org/docs/master/onnx.html" rel="noopener nofollow ugc">torch.onnx.export</a>. Compile from ONNX format to your machine code. This is the approach taken by Intel OpenVINO and a few other hardware vendors, and it will let you easily add support for TensorFlow as well, since there is also a TensorFlow-&gt;ONNX converter.</p>
</li>
<li>
<p>I think the <a href="https://ai.facebook.com/tools/glow/" rel="noopener nofollow ugc">Glow compiler</a> can load TorchScript, so you could look into that.</p>
</li>
</ol>
<p>What’s the name of your project / company?</p>
API to access contents of derivatives.yaml

garymm: <p>Hi,<br>
I want to go from a <code>torch::jit::Operator</code> or a <code>torch::jit::Node</code> for an op it in a particular graph to the <code>torch::jit::Graph</code> that represents its derivative.<br>
I think this information is encoded in some form in <a href="https://github.com/pytorch/pytorch/blob/30e48bbeae545c3292c2ab3fed0cb2dba4a92fed/tools/autograd/derivatives.yaml#L187" rel="noopener nofollow ugc">derivatives.yaml</a>, but I’m not sure how to get it from the APIs available.</p>
<p>Either Python Node’s / Graphs or C++ would work for my purposes.</p>
<p>Thanks for any tips!</p>

soulitzer: <p>There is currently no API for this, though I’m curious what is your use case for this?</p>

garymm: <p>Thanks for taking the time to reply <a class="mention" href="/u/soulitzer">@soulitzer</a>.</p>
<p>So there’s no way to go from a <code>ScriptModule</code> or <code>jit::Graph</code> to the equivalent <code>ScriptModule</code> or <code>jit::Graph</code> for the gradient / backwards?</p>
<p>Could we somehow leverage <code>torch.jit.trace()</code> to produce a graph from a backwards pass?</p>
<p>Summary of the use case: Currently <code>torch.onnx.export()</code> has an ATEN_FALLBACK mode, which means if some ATen op doesn’t have an ONNX equivalent, the ATen op is exported as-is. Some consumers of the exported ONNX model need to be able to compute the backwards / gradient graph. They currently can do this for a pure ONNX graph, but I’d like to extend this to ATen ops as well. Hence the request.</p>
3D input in LSTMs for multiple variables

Papaa_Emeritus: <p>Hello everyone, I am trying to implement Pytorch LSTM to stock market closing prices using historical values.</p>
<p>My input features are Open, High, Low, Close and Volume values for 12 days and my target variable is the closing price for the 13th day. So from my understanding, my input size is 5, output size is 1 and number of time steps is 12.</p>
<p>So, I can construct my LSTM like;</p>
<p>lstm = nn.LSTM(input_size=5, hidden_size=20, batch_first=True)</p>
<p>linear = nn.Linear(in_features=20, out_features=1)</p>
<p>I think my output shape should be (20, 1), one output for each of the sequence in the batch.  To achieve that, how do I create my input to the LSTM?</p>
<p>Thanks in advance.</p>

ptrblck: <p>Based on your description of the use case I would assume your output would have the shape <code>[batch_size, 1]</code> and you are working on a regression task.<br>
To do so you could pass the output activation from the last time step of the lstm to the linear layer, which should have the expected shape afterwards.</p>

Papaa_Emeritus: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>Thanks for the reply. Sorry, I forgot to mention my batch size which is 20. Yeah, I am expecting an final output of [batch_size, 1], one output for each sequence.</p>
<p>However, I seem to be getting an output of shape, <code>[batch_size, seq_len, 1]</code> ~ (20, 5, 1).</p>
<p>This is what I have been doing.</p>
<pre><code class="lang-auto">out, _ = lstm(input)
out = linear(out)
</code></pre>
<p>Is there something I am doing wrong? Or is <code>[batch_size, seq_len, 1]</code> is the expected output size here?</p>

Papaa_Emeritus: <p>Update:</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Made it work. Seems like I didn’t clearly read the second half of your answer. Passed the output from the last time step to the linear layer and got the required shape.</p>
<p>Thanks a lot.</p>
How to get the original class name of a RecursiveScriptModule?

AlphaBetaGamma96: <p>Hi All,</p>
<p>I just have a quick question regarding <code>RecursiveScriptModule</code>, I’ve built a custom optimizer and within it I cycle through all layers via <code>for module in net.modules()</code> and call the name of the <code>module</code> via <code>module.__class__.__name___</code>. I need this as my optimizer handles different layers differently, and I determine what type of layer it is via <code>module.__class__.__name___</code>.</p>
<p>However, if I jit my network this fails because all the name of all the modules become <code>RecursiveScriptModule</code>, I did notice that when printing out <code>module</code> it has <code>RecursiveScriptModule(original_name=Linear)</code>. So, I was wondering how can I get the <code>orginal_name</code> variable here?</p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p><code>module.original_name</code> would return a string containing the original module name, e.g.:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1)
s = torch.jit.script(lin)
print(s.original_name)
&gt; 'Linear'
</code></pre>
<p>so you could try to use this attribute instead.</p>

AlphaBetaGamma96: <p>Thank you! Just what I wanted! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Grouped Conv for Repeated Convolution Backprop

Syzygianinfern0: <p>I have a 3D tensor with each channel to be convolved with the one single kernel. From a quick search, the <strong>fastest</strong> way to do this was to use grouped convolution with number of groups to be the number of channels.</p>
<p>Here is a small reproducible example:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
torch.manual_seed(0)


x = torch.rand(1, 3, 3, 3)
first  = x[:, 0:1, ...]
second = x[:, 1:2, ...]
third  = x[:, 2:3, ...]

kernel = nn.Conv2d(1, 1, 3)
conv = nn.Conv2d(3, 3, 3, groups=3)
conv.weight.data = kernel.weight.data.repeat(3, 1, 1, 1)
conv.bias.data = kernel.bias.data.repeat(3)

&gt;&gt;&gt; conv(x)
tensor([[[[-1.0085]],

         [[-1.0068]],

         [[-1.0451]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)

&gt;&gt;&gt; kernel(first), kernel(second), kernel(third)
(tensor([[[[-1.0085]]]], grad_fn=&lt;ThnnConv2DBackward&gt;),
 tensor([[[[-1.0068]]]], grad_fn=&lt;ThnnConv2DBackward&gt;),
 tensor([[[[-1.0451]]]], grad_fn=&lt;ThnnConv2DBackward&gt;))
</code></pre>
<p>Which you can see perfectly works.</p>
<p>Now coming to my question. I need to do backprop on this (<code>kernel</code> object). While doing this, each weight of the <code>conv</code> gets its own update. But actually, <code>conv</code> is made up of <code>kernel</code> repeated 3 times. At the end I require only an updated <code>kernel</code>. How do I do this?</p>
<p>PS: I need to optimize for speed</p>

Syzygianinfern0: <p>One possible answer is to take a mean after the gradient updates like so</p>
<pre><code class="lang-python">kernel.weight.data = conv.weight.data.mean(0).unsqueeze(0)
</code></pre>
<p>Is this the best way to do it. Or is this even right in the first place?</p>

Syzygianinfern0: <p>For anyone stumbling across the same doubt, find the answer <a href="https://stackoverflow.com/a/68855066/8878627" rel="noopener nofollow ugc">here</a></p>
How to copy model parameters between multiprocessing

skerlet_flandorle: <p>How do I copy model parameters between multiprocessing? Please tell me who understands</p>
<p>It is a model of cuda</p>
<p>Normal way I know</p>
<blockquote>
<p>self.agentQN.load_state_dict(mainQN.state_dict())</p>
</blockquote>
<p>But this method doesn’t seem to work because of a different process. For cuda .share_memory () doesn’t seem to work either</p>
Graph in Docs for LeakyReLU wrong

JamesDickens: <p>The graph used in the docs available here<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1a408353c42a175fa22181523d714b2e2c0f5cb.png" alt="image" data-base62-sha1="ytEvsDL54QJVYrKS1hhjGTzfuen" width="640" height="480"></p>
<p>is the graph of the regular ReLU function. The slope of the graph for x&lt;0 should be non-zero. Is there somewhere to report this so it can be fixed?</p>

ptrblck: <p>I don’t think the plot shows the plain <code>ReLU</code>, but a <code>LeakyReLU</code> using the default <code>negative_slope</code> value of <code>1e-2</code>.<br>
You can zoom in a bit and would see the tiny slope:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/9302b0b47c4136be8f526812db3e9afec8d6f02d.png" data-download-href="https://discuss.pytorch.org/uploads/default/9302b0b47c4136be8f526812db3e9afec8d6f02d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/9302b0b47c4136be8f526812db3e9afec8d6f02d_2_690x157.png" alt="image" data-base62-sha1="kYvX1tV9z5wZcuTjTGclb7hwTZr" width="690" height="157" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/3/9302b0b47c4136be8f526812db3e9afec8d6f02d_2_690x157.png, https://discuss.pytorch.org/uploads/default/original/3X/9/3/9302b0b47c4136be8f526812db3e9afec8d6f02d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/3/9302b0b47c4136be8f526812db3e9afec8d6f02d.png 2x" data-dominant-color="F5F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">895×204 19.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To reproduce the figure, run:</p>
<pre data-code-wrap="python"><code class="lang-python">act = torch.nn.LeakyReLU()
x = torch.arange(-7, 7, 0.01)
out = act(x)

f = plt.figure()
plt.xlim([-7, 7])
plt.ylim([-7, 7])
plt.grid()
plt.plot(x, out)
</code></pre>
<p>which will create:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e68c067d0c6149391c25c0a8319c276cad0a811.png" alt="image" data-base62-sha1="bbDFETtdWfITIkFO9UgSGEqYc6J" width="370" height="248"></p>
<p>To better see the slope, change the default value:</p>
<pre data-code-wrap="python"><code class="lang-python">act = torch.nn.LeakyReLU(negative_slope=0.5)
x = torch.arange(-7, 7, 0.01)
out = act(x)

f = plt.figure()
plt.xlim([-7, 7])
plt.ylim([-7, 7])
plt.grid()
plt.plot(x, out)
</code></pre>

JamesDickens: <p>Wow ok my mistake! I didn’t look carefully enough, given the small slope.<br>
Thanks a lot Peter.</p>
Memory leak - Transformer model

Mary: <p>Hi,</p>
<p>I am loading one of the transformer models using (from_pretrained).<br>
I am only using this model for inference. After getting results from this model I delete the model by: del model</p>
<p>I only use CPU for this process. I noticed that when the process is done and the model is deleted the memory usage decreases but still 1GB of memory is being used and is not freed.<br>
Is there any step that I am missing after using this model during test?<br>
I appreciate your help regarding this.</p>
<p>Thank you very much</p>

Mary: <p>Hi I am wondering if anyone could help me with this issue.</p>
<p>Thank you very much in advance!</p>

ptrblck: <p>I don’t know what the expected memory usage in PyTorch on the host would be at the moment, but did you measure it after just importing PyTorch? I would expect the import to preload some data as well as 3rd party libs (e.g. MKL), which would take some space (but it might be way lower than 1GB).<br>
In case you are sure the memory is used by your model, you could check if any outputs, optimizers etc. are still stored, which could reference the parameters and/or activations.</p>
Calculating derivative of loss w.r.t. single parameter

NondairyCreamer: <p>The following code correctly calculates the derivative of the full_model w.r.t each parameter in x</p>
<pre><code class="lang-auto">import torch

def loss(x): return torch.sum(torch.pow(x, 3))
x = torch.tensor([1, 3, 5], dtype=torch.float64, requires_grad=True)
loss_out = loss(x)
first_derivative = torch.autograd.grad(loss_out, x, create_graph=True)[0]
</code></pre>
<p>first derivative = [3, 27, 75] = 3*x^2</p>
<p>However, if I want to only calculate the derivative with respect to a single variable, running with a modified final line throws an error<br>
<code>first_derivative = torch.autograd.grad(loss_out, x[0], create_graph=True)[0]</code><br>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
<p>I must be missing something quite basic about pytorch.<br>
Any indexing I perform of x (such as x[0:3]) returns this error, claiming that the variable is not used in the graph.</p>
<p>Can anyone help me see what I’ve done wrong?</p>
<p>Thanks</p>

soulitzer: <p>Any indexing you do on <code>x</code> creates a new view tensor of the base tensor <code>x</code>. In terms of gradients, the flow is one directional, i.e., if the view is used in the loss, the gradients will flow back to the base, but if you modify the base only (which isn’t actually aware of any of the views), the gradient wrt to the view is not updated. What you could do instead is index after computing the gradient (<code>x.grad[0]</code>), or compute the loss using the view instead of the base.</p>
How to destroy computational graph after setting create_graph to True?

SakuraTea: <p>Hi,</p>
<p>Can I ask how to destroy the computational graph after setting create_graph to True?</p>
<p>Say that I have:</p>
<pre><code class="lang-auto">x = torch.randn(10,5)
y = torch.randn(10,5)

def f(x,y):
    return torch.sum(x*y*y)

def jacobian_matrix_with_graph(x, y):
    jac = torch.autograd.functional.jacobian(f,(x,y),create_graph=True)
    return jac

def partial_hes(y): # only compute hessian matrix 1. f /partial_x /partial_y 2. f /partial_x /partial_y
    hes = jacobian_matrix_with_graph(x, y) # x being global variable
    return hes
    
hessian_matrix_pxpy = torch.autograd.functional.jacobian(partial_hes,(y))[1]
print(hessian_matrix_pxpy)
</code></pre>
<p>After I have used this computation graph, how do I destroy it manually?</p>
<p>Thanks in advance.</p>

soulitzer: <p>There is no way to manually “destroy” the graph. It will be freed automatically once all references to it are gone. If you wish to obtain a tensor with the value of the output, but no longer attached to this graph, you can use <code>detach</code></p>
Jacobian error?

A_Rzhechitskii: <pre><code class="lang-auto">import torch
x = torch.tensor([1., 2., 3., 4.], requires_grad=True)

def y_v(x):
    return x

ja = torch.autograd.functional.jacobian(y_v, x, create_graph=True)
s = ja.sum()
s.backward()
x.grad

# Traceback (most recent call last):
# in &lt;module&gt;
#    s.backward()
# File anaconda3/envs/pytorch/lib/python3.9/site-packages/torch/autograd/__init__.py", line 147, in backward
#    Variable._execution_engine.run_backward(
# RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>The error only occurs for a linear function.<br>
For example, the function succeeds</p>
<pre><code class="lang-auto">def def y_v(x):
    return x.exp()
</code></pre>

soulitzer: <p>The Jacobian of the function <code>f(x) = x</code> is always the identity (constant wrt x), so it is expected that the gradient does not flow back to x.</p>
Unsure how to write Gradient for this Method

soulslicer: <p>I am trying to implement the backward / gradient pass for the following kernel.</p>
<p>It’s basically an image tensor of [3, H, W] which is split up as <code>org_cloud_x_ptr, org_cloud_y_ptr, org_cloud_z_ptr</code>. And I am iterating through it, checking its neighbourhood, and computing 6 additional tensors  of which combined is [6, H, W] which is split up as <code>dxdz_tensor..</code>, which is basically the output of the method. The method is something like below:</p>
<pre><code class="lang-auto">  // Iterate Image
  for (auto yy = 0; yy &lt; height; yy++) {
    for (auto xx = 0; xx &lt; width; xx++) {

      const float bad_point = std::numeric_limits&lt;float&gt;::quiet_NaN();
      const int index = yy * width + xx;
      const float px = org_cloud_x_ptr[index];
      const float py = org_cloud_y_ptr[index];
      const float pz = org_cloud_z_ptr[index];

      // Kernel Local Neighbourhood search
      for (auto v = yy - search_size; v &lt;= yy + search_size; v += step_size) {
        for (auto u = xx - search_size; u &lt;= xx + search_size; u += step_size) {

          if (u &lt; 0 || u &gt;= width || v &lt; 0 || v &gt;= height)
            continue;

          const int index2 = v * width + u;

          const float qx = org_cloud_x_ptr[index2];
          const float qy = org_cloud_y_ptr[index2];
          const float qz = org_cloud_z_ptr[index2];

          if (std::isnan(qx) || qz == 0)
            continue;

          const float dz = qz - pz;
          const float dx = qx - px;
          const float dy = qy - py;


          dxdx_sum += f * dx * dx;
          dxdy_sum += f * dx * dy;
          dydy_sum += f * dy * dy;
          dxdz_sum += f * dx * dz;
          dydz_sum += f * dy * dz;
        }
      }

      // Set
      dxdz_tensor[index] =  dxdx_sum
      ...

    }
  }
</code></pre>
<p>How would I even start to think about how to write a backward pass / gradient for this?</p>
Autograd expecting wrong variable version

pozimek: <p>Hey, I don’t understand why doesn’t the following bit of code work:</p>
<pre><code class="lang-auto">d1, d2 = 12, 24
timesteps = 3

FC1 = nn.Linear(d1,d2)
FC2 = nn.Linear(d2,d1)
L1 = nn.L1Loss()

a = torch.randn(d1)
b_storage = torch.zeros(timesteps, d2)

for t in range(timesteps):
    b_storage[t] = FC1(a)
    a = FC2(b_storage[t])
    print(b_storage._version)  #&lt;- 1,2,3
    
target = torch.randn_like(a)
loss = L1(a, target)
loss.backward()  #&lt;- expects version 2?!

[out]: **RuntimeError:** one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 24]], which is output 0 of UnsqueezeBackward0, is at version 3; expected version 2 instead.
</code></pre>
<p>Whatever I set the number of timesteps to be, autograd expects the final version to be at timesteps-1. If I don’t use the b_storage tensor and instead feed FC2 with the output of FC1(a) directly then the issue doesn’t come up. However the problem doesn’t make sense: looking at the code makes it obvious that autograd should be expecting b_storage to be at version=timesteps, not timesteps-1. Is there a bug with UnsqueezeBackward0?</p>

ptrblck: <p>I think you are running into a similar issue as described <a href="https://discuss.pytorch.org/t/encounter-in-place-operation-in-delayed-online-learning/119415/6">here</a>.<br>
In particular:</p>
<blockquote>
<p>One source of error is that the model needs the unmodified data_in but the versioning does not capture regions of <code>data_in</code>.</p>
</blockquote>
<p>and you could fix it by using <code>a = FC2(b_storage[t].clone())</code>.</p>
Data division in incremental fashion

pramesh: <p>I want to divide the data set in following fashion<br>
30000 input size<br>
1st iteration : 15000<br>
2nd iteration:30000<br>
in incremental fashion for MNIST dataset.</p>
<pre><code class="lang-auto">num_clients = 2

traindata = datasets.MNIST('./data', train=True, download=True,
                       transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.1307,), (0.3081,))])
                       )
traindata_split = torch.utils.data.random_split(traindata, [int(traindata.data.shape[0] / num_clients) for _ in range(num_clients)])
</code></pre>
<p>i wish to divide the traindata_split  into two or more chunk in incremental fashion eg,1st chunk 15000,second 30000</p>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 2: invalid start byte

Mona_Jalal: <p>Do you know how I can fix this problem in PyTorch 1.9?</p>
<pre><code class="lang-auto">File "main.py", line 138, in main
    checkpoint = torch.load(args.resume)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/serialization.py", line 608, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/serialization.py", line 787, in _legacy_load
    result = unpickler.load()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 2: invalid start byte
</code></pre>
<p>I have:</p>
<pre><code class="lang-auto">$ pip freeze
h5py==3.3.0
joblib==1.0.1
numpy==1.21.2
Pillow==8.3.1
scikit-learn==0.24.2
scipy==1.7.1
sklearn==0.0
threadpoolctl==2.2.0
torch==1.9.0
torchaudio==0.9.0
torchvision==0.10.0
typing-extensions==3.10.0.0
</code></pre>

Mona_Jalal: <p><code>checkpoint = torch.load(args.resume, encoding='latin1')</code></p>
<p>added <code>encoding='latin1'</code> and it’s working now.</p>
Load a saved model on Pytorch AttributeError: 'dict' object has no attribute 'seek'

CasellaJr: <p>I trained a model, and I saved the best one, with:</p>
<pre><code class="lang-auto"># Store params at the best validation accuracy
        if save_param and accuracy_star &gt; best_accuracy:
          torch.save({'vgg_a':nets[0].state_dict(),'classifier':nets[3].state_dict()}, f"{model_name}_best_test.pth")
</code></pre>
<p>SO, at the end of the training I downloaded this “federated_mnist_best_test.pth” on my local computer.<br>
Now, i created a new notebook on google colab, and I tried din this way to upload the model:</p>
<pre><code class="lang-auto">from google.colab import files
uploaded= files.upload()
</code></pre>
<p>Inside uploaded I put my .pth file, and then:</p>
<pre><code class="lang-auto">state_dict = torch.load(uploaded)
</code></pre>
<p>gives me this error:</p>
<pre><code class="lang-auto">AttributeError: 'dict' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.
</code></pre>

ptrblck: <p>This error is raised, if you are trying to <code>torch.load</code> a <code>dict</code> as seen here:</p>
<pre><code class="lang-python">torch.load({})
&gt; AttributeError: 'dict' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.
</code></pre>
<p>so I assume that <code>uploaded</code> is not a valid path but a <code>dict</code> object instead.</p>

CasellaJr: <p>Yes.<br>
Solution was this: <code>state_dict = torch.load('federated_mnist_best_test.pth')</code></p>
Take a few layers from a model?

Z_Huang: <p>This question may sound silly. Is it possible to take a few layers from a built-in model for example resnet50?</p>
<p>For example, I only want to pull out the first 2 layers from resnet50. Is it a possible way to make that happen?</p>

ptrblck: <p>Yes, that’s possible and you could e.g. directly reuse them as seen here:</p>
<pre><code class="lang-python">model = models.resnet50()
conv1 = model.conv1
bn1 = model.bn1

x = torch.randn(1, 3, 224, 224)
out = conv1(x)
out = bn1(out)
print(out.shape)
&gt; torch.Size([1, 64, 112, 112])
</code></pre>
<p>Depending on your use case you might want to assign these layers as internal modules to a custom <code>nn.Module</code>.</p>

Z_Huang: <p>Thank you. What if I reference a few layers from resnet50 and make these withdrew layers as an independent neural network?</p>

ptrblck: <p>Could you explain what “make these withdrew layers” means in this context?</p>

Z_Huang: <p>If I withdraw a few layers from resnet50. Then, I wish I can use these withdrew layers and combine them into a new neural network.</p>
<p>Well, I wish I can make the following layers as an independent neural network. Like the following</p>
<p>class new_net(nn.Module):<br>
model.conv1<br>
model.bn1</p>
<p>When the new neural network is created. Then, I can use the new neural network like: model = new_net().</p>

ptrblck: <p>Yes, this will also work:</p>
<pre><code class="lang-python">class NewNet(nn.Module):
    def __init__(self):
        super().__init__()
        resnet = models.resnet50()
        self.conv1 = resnet.conv1
        self.bn1 = resnet.bn1
        
    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        return x

model = NewNet()
x = torch.randn(1, 3, 224, 224)
out = model(x)
</code></pre>
Applying differentiable affine transformation

Carlos_Martin: <p>Hello,</p>
<p>I have 4 anatomical views, one is 3D (sa) and the other ones are 2d(_la). I want to perform the following operation(multiview_affine) in a differentiable way. The problems that I am facing are mainly that I have to deal with a non integer operations and that a lot of points between 3D and 2D views lay outside. So I solve that iterating over all the 3d view points, computing the position of that points on each one of the other views and retriving its value, verifying that the float positions are converted to integers. I am not sure if this is the optimal way to do that, an dof course I don’t know how to write that on a pytorch differentiable neural network.</p>
<pre><code class="lang-auto">from nibabel.affines import apply_affine
import nibabel as nib
import numpy as np

def apply_affine_sa_la(_sa,_la,idx):

    sa2la_affine = np.linalg.inv(_la.affine).dot(_sa.affine)

    sape = _sa.get_fdata().shape

    data_la = _la .get_fdata()[...,idx]

    ph = np.zeros_like( _sa.get_fdata()[...,idx])

    for x in range(sape[0]):

        for y in range(sape[1]):

            for z in range(sape[2]):

                la_pos = apply_affine(sa2la_affine,np.array([x,y,z]))

                #la_pos = np.floor(la_pos)

                if np.abs(la_pos[2]) &lt; 0.5:

                    try:

                        ph[x,y,z] = data_la[int(la_pos[0]),int(la_pos[1]),0]

                    except:

                        continue

                else:

                    continue

    return ph

def multiview_affine(_sa,_4c,_3c,_2c,idx):

    ph_0 = apply_affine_sa_la(_sa,_4c,idx)

    ph_1 = apply_affine_sa_la(_sa,_3c,idx)

    ph_2 = apply_affine_sa_la(_sa,_2c,idx)

    d = np.stack([ph_0,ph_1,ph_2],axis = 3)

    ph = d.max(axis=3)

    return ph
</code></pre>
How to get gradient wrt a parameter defining the mask size?

Haozhi: <p>I have a tensor <code>R</code> containing distances to the center position and a parameter <code>r_max</code> defining the maximum distance. I want to generate a mask like <code>R.lt(r_max)</code> and get the gradient wrt <code>r_max</code>. How should I generate the mask tensor to make this possible? I tried <code>lt()</code> and <code>torch.where()</code>and both failed.</p>

soulitzer: <p>If you want a boolean mask (e.g., as returned by <code>lt</code>), that is not floating type so you won’t be able to compute gradient of it wrt to anything. If you do something like <code>clamp_max</code>, you could get gradients wrt <code>R</code> and<code> r_max</code>, but not sure if that’s what you need.</p>

Haozhi: <p>I will use the mask to do calculation so I think <code>clamp</code> can solve my problem. Thanks a lot.</p>

Haozhi: <aside class="quote no-group" data-username="soulitzer" data-post="2" data-topic="124441">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/48.png" class="avatar"> soulitzer:</div>
<blockquote>
<p>’</p>
</blockquote>
</aside>
<p>I’m sorry. It seems that <code>clamp</code> only accepts <code>Number</code> as <code>min/max</code> which can not be differentiable.</p>

Haozhi: <p>I found a way to make the ‘r_max’ differentiable.</p>
<pre><code class="lang-auto">import torch

x = torch.linspace(-20, 19, 40)
Y, X = torch.meshgrid(x, x)
R = (X**2 + Y**2)**0.5

r_max = torch.tensor(15.0, requires_grad=True)

R_norm = R / r_max
mask = R_norm.where(R_norm&gt;1, torch.zeros_like(r))
mask = mask.where(R_norm&lt;1, torch.ones_like(r))
</code></pre>

yanielc: <p>thanks for sharing! i had the same question</p>

Haozhi: <p>Sorry for ignoring the difference between <code>tensor.where(condition, y)</code> and <code>np.where(condition, y)</code>. In order to get results like <code>R.lt(r_max)</code>, the code should be:</p>
<pre><code class="lang-auto">mask = R_norm.where(R_norm&gt;1, torch.ones_like(r))
mask = mask.where(R_norm&lt;1, torch.zeros_like(r))
</code></pre>
Seeing memory used by each variable in Python / PyTorch?

arman.avesta: <p>Hi Gang!</p>
<p>I’m working with large 3D MR images and I’m making 3D tensors out of them. These tensors take a lot of memory, and I want to optimize my algorithm to minimize memory usage.</p>
<p>Is there any way to see how much memory each tensor is occupying? I tried sys.getsizeof(tensor), but it doesn’t show the memory usage by the tensor.</p>
<p>I would appreciate any help!</p>

ptrblck: <p>Assuming you would like to check the device memory used to store the tensor data, you could multiply the <code>tensor.nelement()</code> with the number of bytes per element (4 for the default <code>float32</code> dtype) and would get the actual size.</p>

arman.avesta: <p>Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Mask RCNN for Production

AI-P-K: <p>I have trained Mask RCNN from pre-trained resnet50_fpn. I want to use the saved weights in a production environment where i store the savedmodel.pt on a server and i make send data to it for inference. All goes well until 3 or more requests are made to the model. If more than 3 requests are made at a same time the inference time goes from 2s/image to 15 seconds. Is this normal? Is there any way to improve this?<br>
PS The behaviour is the same on CPU or GPU</p>
How to use multi-gpus in Libtorch?

PistonY: <p>Does anyone has example?</p>

glaringlee: <p><a class="mention" href="/u/pistony">@PistonY</a><br>
Yes, you can. You can create a TensorOptions obj by passing both the device type and its device index, the default is -1 which means pytorch will always use the same single device. You can explicitly specify this (0,1,etc)<br>
Here is an example:<br>
/<a href="https://github.com/pytorch/pytorch/blob/5fd037ce4450d2a7bb477fa0f58677d7b256fdfe/test/cpp/api/tensor_cuda.cpp#L16" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/5fd037ce4450d2a7bb477fa0f58677d7b256fdfe/test/cpp/api/tensor_cuda.cpp#L16</a>’</p>

PistonY: <p>Thanks for reply.That’s mean if I want to deploy using libtorch I need to write all the code by myself like what <code>nn.DataParallel</code> do?</p>

glaringlee: <p><a class="mention" href="/u/pistony">@PistonY</a><br>
I don’t think so. Since what you need to do is to mark your tensor (input and output) which gpu you want it to be allocated/processed on. It is not related to the model.</p>

PistonY: <p>The model need not to put on other devices?</p>

Taewoo_Lee: <p>Hi. Is there any update regarding this question?<br>
<a class="mention" href="/u/glaringlee">@glaringlee</a></p>