akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why to expect gradient of non learnable parameter

username: matthias.l
cooked: <p>When writing a custom function with custom forward and backward:</p>
<p>Assume the module</p>
<pre data-code-wrap="python"><code class="lang-python">class IMyFunction(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, a, b):
        # a is a learnable parameter. b is a constant tensor
        # b.requires_grad is False
        ctx.save_for_backward(x, a, b)
        # Compute y ...
        return y

    @staticmethod
    def backward(ctx, grad_y):
        x, a, b = ctx.saved_tensors
        # Compute gradient of x and learnable parameter a
        return d_x, d_a
</code></pre>
<p>where <code>a</code> is a trainable parameter and <code>b</code> a constant tensor.</p>
<p>Using this module results in the error message:</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “./test.py”, line 217, in test_both<br>
y.backward(y_grad)<br>
File “/home/matthias/.local/lib/python3.7/site-packages/torch/tensor.py”, line 166, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/matthias/.local/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 103, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: function MyFunctionBackward returned an incorrect number of gradients (expected 3, got 2)</p>
</blockquote>
<p>My question: Why are backward functions required to return a gradient for non learnable parameters?<br>
How to mark a inputs as constant? (No gradient is computed)</p>
<p>Or Is this there a bug in<br>
<a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/custom_function.h" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/custom_function.h</a> ?</p>

username: albanD
cooked: <p>No this is expected. You should return as many elements as there were inputs to the forward function. If something is non-differentiable, you should return <code>None</code>.</p>
How can I use 2 models in a training loop?

username: shamoons
cooked: <p>I have 2 models: <code>model</code> and <code>baseline_m</code>. I froze all of the weights of <code>model</code>, so I can use those gradients to update the parameters of <code>baseline_m</code>.</p>
<pre><code class="lang-auto">            baseline_output = baseline_m(baseline_input)

            out, output_sizes = model(baseline_output, input_sizes)
            decoded_output, decoded_offsets = decoder.decode(out, output_sizes)
</code></pre>
<p>But then I get:</p>
<pre><code class="lang-auto">RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>Which makes sense, since the gradients are frozen for <code>model</code>. But still, shouldn’t <code>baseline_m</code> update?</p>

username: ptrblck
cooked: <p>Yes, the workflow should work as shown in this dummy example:</p>
<pre><code class="lang-python">baseline = nn.Linear(1, 1)
model = nn.Linear(1, 1)

for param in model.parameters():
    param.requires_grad_(False)

x = torch.randn(1, 1)
out = baseline(x)
print(out.grad_fn) # prints valid grad_fn
out = model(out)
out.mean().backward()
print(baseline.weight.grad)
</code></pre>
<p>Try to debug <code>baseline_m</code> and make sure your output has a valid <code>.grad_fn</code>.<br>
You might accidentally detach the activation by e.g. wrapping it into another tensor, using numpy etc.</p>

username: shamoons
cooked: <p>Perfect - thank you.</p>

username: shamoons
cooked: <p>When I do <code>print(out.grad_fn)</code> after <code>model(out)</code>, I get <code>&lt;TransposeBackward0 object at 0x16b03b898&gt;</code>. Is that normal? Shouldn’t I have no grad fn for model?</p>

username: shamoons
cooked: <p>And when I do <code>print(baseline_m.weight.grad)</code>, I get <code>AttributeError: 'BaselineModel' object has no attribute 'weight'</code></p>

username: ptrblck
cooked: <p>My <code>baseline_m</code> is just a single linear layer, so you would have to access a valid module and its parameter.</p>
<p>If <code>baseline_m</code> contains available parameters, the output of <code>model</code> should also have a <code>grad_fn</code>, although you should check that the gradients are not set in <code>model</code>.</p>
LSTM/GRU gate weights

username: aron-bordin
cooked: <p>Hi <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I would like to have a custom weight initialization to each gate of my rnn (GRU and LSTM). How can I get the weights of a specific gate in the GRU/LSTM implementation ?</p>

username: ruotianluo
cooked: <pre><code>#LSTM
net = nn.LSTM(100, 100) #Assume only one layer
w_ii, w_if, w_ic, w_io = net.weight_ih_l0.chunk(4, 0)
w_hi, w_hf, w_hc, w_ho = net.weight_hh_l0.chunk(4, 0)

#GRU
net = nn.GRU(100,100)
w_ir, w_ii, w_in = net.weight_ih_l0.chunk(3, 0)
w_hr, w_hi, w_hn = net.weight_ih_l0.chunk(3, 0)
</code></pre>
<p>reference:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/nn/modules/rnn.py</a></h4>
<pre><code class="lang-py">import math
import torch
import warnings

from .module import Module
from ..parameter import Parameter
from ..utils.rnn import PackedSequence


class RNNBase(Module):

    def __init__(self, mode, input_size, hidden_size,
                 num_layers=1, bias=True, batch_first=False,
                 dropout=0, bidirectional=False):
        super(RNNBase, self).__init__()
        self.mode = mode
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bias = bias
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Dheeraj_M_Pai
cooked: <p>How do I get the biases?</p>

username: purvanshi_mehta
cooked: <p>You can get weights of all layers by</p>
<pre><code class="lang-auto">rnn = nn.LSTM(100, 100,20)
for name in rnn.named_parameters():
    if 'weight' in name[0]:
        weight_list.append(name[1])

</code></pre>

username: Kris
cooked: <p>Great answer, thanks!<br>
But I think that for the GRU, the second <code>weight_ih_l0</code> should be changed to <code>weight_hh_l0</code></p>

username: miranda
cooked: <p>@ <a href="https://discuss.pytorch.org/u/Dheeraj_M_Pai">Dheeraj_M_Pai</a>: You can get biases by: <code>net.bias_hh_l0</code>, and <code>net.bias_ih_l0</code>. 0 in <code>l0</code> stands for layer number. If you have more than one lstm layer then change 0 to 1, 2, or the number of whichever layer you want to manipulate.</p>
Two Loss Functions with weight factor

username: jonas15
cooked: <p>Good evening,</p>
<p>I’m using at the moment for my approach two loss functions and weight them with a factor:</p>
<pre><code class="lang-auto">loss_1= loss_1_function(output, args)
loss_2= loss_2_function(output, args)



overall_loss_tensorboard = args.alpha_loss*loss_1.item() + (1-args.alpha_loss)*loss_2.item()
(args.alpha_loss*loss_1+ (1-args.alpha_loss)*loss_2).backward()

</code></pre>
<p>Is this the right way, how one would weight two losses and combine them?</p>
<p>Thanks and best regards<br>
Jonas</p>

username: Stefano_Savian
cooked: <p>Hi Jonas, I am also in the process of tuning multiple loss functions. Your approach seems somehow manual tuning.  I suggest you to find a coarse optimization for alpha. After, I suggest you to start from this article <a href="https://towardsdatascience.com/self-paced-multitask-learning-76c26e9532d0" rel="nofollow noopener">https://towardsdatascience.com/self-paced-multitask-learning-76c26e9532d0</a> and try some automatic tuning method.</p>
<p>Best,<br>
Stefano</p>

username: jonas15
cooked: <p>Hi Stefano,</p>
<p>Yep that’s right and the worst thing is that this alpha factor is just one of 5 parameters… So thanks a lot for the link! I also wanted to look into the field of Bayesian Hyperparameter Optimization.</p>
<p>However just to be sure, the approach how I factor the two losses is correct or?</p>
<p>Thanks and best regrads<br>
Jonas</p>

username: ayalaa2
cooked: <p>The way you are weighing the two losses is correct.</p>
Move input tensor to cuda will cause 'Segmentation fault'

username: savior
cooked: <p>If I use opencv to read an image (1248x384 ), convert it to tensor and move it onto cuda in python , it won’t have problems.However , when I do the same thing in c++ libtorch , it will report Segmentation fault.Furthermore , I find it’s the moving onto cuda process that causes the Segmentation fault.Is there anyone know why or how to solve the problem?</p>

username: tom
cooked: <p>It’s hard to tell with the information you give and no code.</p>
<p>One common pitfall: if you use <code>from_blob</code> to get the tensor from OpenCV, you do not own the memory, i.e. you need to call the copying(! mind you) to cuda before the OpenCV memory goes out of scope / keep the memory in scope until you have made your first copy.</p>
<p>Best regards</p>
<p>Thomas</p>

username: savior
cooked: <p>Sorry , I forgot to post the code. Do you mean that I should use gpuMat to load my image onto gpu firstly and then calculate it ?</p>
<p>/<em>pytorch 1.2</em>/</p>
<p><span class="hashtag">#include</span> “torch/script.h”<br>
<span class="hashtag">#include</span> “torch/torch.h”<br>
<span class="hashtag">#include</span> &lt;opencv2/opencv.hpp&gt;<br>
<span class="hashtag">#include</span> </p>
<p><span class="hashtag">#include</span> <br>
<span class="hashtag">#include</span> </p>
<p>using namespace std;<br>
using namespace cv;</p>
<p>int main()<br>
{<br>
int height = 1248;<br>
int width = 384;</p>
<pre><code>Mat img = imread("/media/usr515/ed65d8df-d015-4269-90bc-ac3cccb109a8/dsq/zz/stereo/train/2011_09_26_drive_0002_sync/image_02/data/0000000007.png");   
Mat temImg;
resize(img,temImg,Size(height,width),0,0,INTER_LINEAR);
cvtColor(temImg, temImg, CV_RGB2BGR);    
temImg.convertTo(temImg,CV_32FC3,1.0);
torch::Tensor tensor_image = torch::from_blob(temImg.data, {temImg.rows, temImg.cols , 3}, torch::kFloat);        
tensor_image = tensor_image.permute({2,0,1});
tensor_image = tensor_image.unsqueeze(0);
tensor_image = tensor_image.cuda();


Mat img_1 = imread("/media/usr515/ed65d8df-d015-4269-90bc-ac3cccb109a8/dsq/zz/stereo/train/2011_09_26_drive_0002_sync/image_03/data/0000000007.png");   
Mat temImg_1;
resize(img_1,temImg_1,Size(width,height),0,0,INTER_LINEAR);    
cvtColor(temImg_1, temImg_1, CV_RGB2BGR);
temImg_1.convertTo(temImg_1,CV_32FC3,1.0);
torch::Tensor tensor_image_1 = torch::from_blob(temImg_1.data, {temImg_1.rows, temImg_1.cols , 3}, torch::kFloat);        
tensor_image_1 = tensor_image_1.permute({2,0,1});
tensor_image_1 = tensor_image_1.unsqueeze(0);
tensor_image_1 = tensor_image_1.cuda();

torch::jit::script::Module module = torch::jit::load("/media/usr515/ed65d8df-d015-4269-90bc-ac3cccb109a8/dsq/zz/stereo/psmnet_256_1.pt" , torch::kCUDA);


torch::Tensor result = module.forward({tensor_image , tensor_image_1}).toTensor();

cout &lt;&lt; result &lt;&lt; endl;

return 1;
</code></pre>
<p>}</p>

username: savior
cooked: <p>I know what you mean.It works! Thank you for solving my problem.</p>

username: solarflarefx
cooked: <p><a class="mention" href="/u/savior">@savior</a> What was the ultimate solution you came to?  Did you use the from_blob method with a GpuMat?</p>
CUDAExtension for multiple GPU Architectures

username: Michael_Baumgartner
cooked: <p>Ahoi <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I’m working on a project which needs some additional CUDA Code and I want to compile it during the installation of my python package for multiple GPU architecture.</p>
<p>The whole setup works fine for my local GPU (RTX 2080Ti, CUDA 10.1) but when my job is running on a <strong>different</strong> GPU model (e.g. on our cluster) it crashes with the following message:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: no kernel image is available for execution on the device
</code></pre>
<p>I tried <code>export TORCH_CUDA_ARCH_LIST="3.5;3.7;5.0;5.2;6.0+PTX;6.1+PTX;7.0+PTX;7.5+PTX"</code>  but it still does not work (the <code>setup.py</code> is very similar to the detection2 one <a href="https://github.com/facebookresearch/detectron2/blob/master/setup.py" rel="nofollow noopener">https://github.com/facebookresearch/detectron2/blob/master/setup.py</a>). Any idea what the problem might be? Are there any additional steps to ensure that it really compiles for multiple architectures (I’m running a trial and error setup right now)?</p>
<p>Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Michael_Baumgartner
cooked: <p>Apparently it was some kind of problem with an old cached version … works now …</p>
Post training static quantization calibration inference

username: Chame_call
cooked: <p>I’m trying to quantize a pretrained model which I haven’t its dataset for.<br>
In <a href="https://pytorch.org/docs/stable/quantization.html#quantization-workflows" rel="noopener nofollow ugc">the Quantization Workflow</a> I see the following 5 step:</p>
<blockquote>
<p>Calibrate the model by running inference against a calibration dataset</p>
</blockquote>
<p>And again in <a href="https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html#post-training-static-quantization" rel="noopener nofollow ugc">the static quantization tutorial</a> is used <code>Dataloader</code> with real data.</p>
<p><strong>So the question is :</strong><br>
can I quantize a pretrained model without its dataset? (with just a random data of needed shape like in the <code>torch.jit.trace()</code>)</p>
Test user written GRU with GRUCell in Pytorch

username: mikeaalv
cooked: <p>I’m trying to write my own GRU code (in Python). The code will implement a modified GRU cell with additional part that is controlled by a parameter (N). If N=0, in structure/math, it should be the same as GRUCell in Pytorch. I’m trying to test whether this is true by setting the same random seed</p>
<pre><code class="lang-auto">random.seed(args.seed)
torch.manual_seed(args.seed)
</code></pre>
<p>As is GRUCell is itself implemented in C++ but my new code is implemented in Python. Should I have the expectation that under same random seed and N=0, the whole training result will be the same. Sorry I cannot share the implementation code for the current.</p>
<p>Thank you.</p>
<p>Best.<br>
Yue</p>

username: albanD
cooked: <p>Hi,</p>
<p>No you cannot expect the training to give the exact same result. This is because floating point operations are not associative. So different implementations can lead to very small difference. If you then do gradient descent, it has the tendency to increase these differences very quickly leading to different results.</p>
<p>One way to test this is to set the same weights for both modules and make sure that the difference for a single forward/backward is &lt;1e-5.</p>

username: mikeaalv
cooked: <p>Thank you very much. Will test on that.</p>
I want to find a function

username: Christopher_Miao
cooked: <p>Excuse me my friends<br>
In the past I used Keras to finish my NLP tasks.<br>
recently I turned to Pytorch<br>
The thing I want to ask is<br>
In keras I use<br>
tokenizer.fit_on_texts(texts + pos_en1 + pos_en2)<br>
to build a vocab of str and index .and then I could use<br>
texts_to_sequence to change my texts and position into index<br>
Is there any function in pytorch I could do the same things?<br>
Thank you for answering</p>
Segmented operators (segment_sum)

username: jaromiru
cooked: <p>I’m looking for an equivalent operator as tf.segment_sum, I couldn’t find any in the docs. If it’s not implemented, then this is a feature request.</p>

username: yibo
cooked: <p>Similarly I need the equivalent of <a href="https://www.tensorflow.org/api_docs/python/tf/unsorted_segment_sum" rel="nofollow noopener">tf.unsorted_segment_sum</a>. Such segment reduction ops were implemented in <a href="https://caffe2.ai/docs/operators-catalogue.html" rel="nofollow noopener">Caffe2</a>, which seem to be getting merged into PyTorch: <a href="https://github.com/pytorch/pytorch/blob/master/caffe2/operators/segment_reduction_op.cc" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/caffe2/operators/segment_reduction_op.cc</a></p>
<p>Hopefully they’ll be available in PyTorch 1.0…</p>

username: bbrighttaer
cooked: <p>Well, I just tried my hands on the segment_sum. You can check if out here: <a href="https://gist.github.com/bbrighttaer/207dc03b178bbd0fef8d1c0c1390d4be" rel="nofollow noopener">https://gist.github.com/bbrighttaer/207dc03b178bbd0fef8d1c0c1390d4be</a><br>
You can let me know about its correctness after your verification.<br>
Thanks.</p>

username: jaromiru
cooked: <p>I’d like to point to a pytorch_scatter library, that implemements this functionality: <a href="https://github.com/rusty1s/pytorch_scatter" rel="nofollow noopener">https://github.com/rusty1s/pytorch_scatter</a></p>
Creating custom bipartite graph in pytorch geomerty

username: AsmaAdil
cooked: <p>I have used the code as it is in the tutorial provided in official site. Its giving me error for the line batch = next(iter(dataloader))<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/286d8333aaf42c4e80ca19a514460201a3a614f4.png" data-download-href="https://discuss.pytorch.org/uploads/default/286d8333aaf42c4e80ca19a514460201a3a614f4" title="errorbipartite graph"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/286d8333aaf42c4e80ca19a514460201a3a614f4.png" alt="errorbipartite graph" data-base62-sha1="5LDJ9w5wGYYBkdXVaiI66ubvfhy" width="690" height="338" data-dominant-color="EDEAF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">errorbipartite graph</span><span class="informations">1158×568 37.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
#####<span class="hashtag-raw">#Bipartite</span> Graph#######<br>
class BipartiteData(Data):<br>
def <strong>init</strong>(self, edge_index, x_s, x_t):<br>
super(BipartiteData, self).<strong>init</strong>()<br>
self.edge_index = edge_index<br>
self.x_s = x_s<br>
self.x_t = x_t</p>
<pre><code>def __inc__(self, key, value):
    if key == 'edge_index':
        return torch.tensor([self.x_s.size(0), self.x_t.size(0)])
    else:
        return super(BipartiteData, self).__inc__(key, value)
</code></pre>
<p>edge_index = torch.tensor([<br>
[0, 0, 1, 1],<br>
[0, 1, 1, 2],<br>
])<br>
x_s = torch.randn(2, 16)  # 2 nodes.<br>
x_t = torch.randn(3, 16)  # 3 nodes.</p>
<p>data = BipartiteData(edge_index, x_s, x_t)<br>
print (data)<br>
data_list = [data,data]<br>
dataloader = DataLoader(data_list, batch_size=2)<br>
batch = next(iter(dataloader))<br>
<span class="hashtag-raw">#print</span>(batch.edge_index)</p>

username: AsmaAdil
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64c2e40416c00e6192266f4ef51a3b95c878bc6b.png" data-download-href="https://discuss.pytorch.org/uploads/default/64c2e40416c00e6192266f4ef51a3b95c878bc6b" title="code b"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64c2e40416c00e6192266f4ef51a3b95c878bc6b.png" alt="code b" data-base62-sha1="enniv76B6vr5YSIaoO1DsVLDy7p" width="514" height="500" data-dominant-color="F7F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">code b</span><span class="informations">767×746 27.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Better code dispaly</p>
Batch gradient descent (Vanilla)

username: John88
cooked: <p>Hi,</p>
<p>I would like to use the batch gradient version (BGD) and I am not sure to understand how to use it in pyTorch (yes, I already search on this forum but I still not understand).<br>
The SGD implementation is a single step implementation but the user has to select randomly the data point. So is it true to say that the BGD is the SGD minibatch with batch_size equals to the number of data points ?</p>
<p>If it is true, I have the following implementation in pyTorch</p>
<pre><code class="lang-auto">trainloader = torch.utils.data.DataLoader(src_data, batch_size=len(src_data))
optimizer   = SGD(model.parameters(), lr=0.1)
criterion   = MSELoss(reduction='sum')
for epoch in range(epoch):
   for inputs, labels in trainloader:
      optimizer.zero_grad()
      outputs  = model(inputs)
      loss     = criterion(outputs, labels)
      loss.backward()
      optimizer.step()
</code></pre>
<p>but results are quite bad (especially with the same learning used for SGD).<br>
If my implementation is correct, I guess that BGD misses the/a local optimum and go further.</p>

username: quanguet
cooked: <p>Even the definition of SGD is applicable to one data point but in Pytorch, as you said, SGD with <code>batch_size &gt; 1</code> will be similar to BGD.</p>
<p>Your code is, I think, okay. However, could you check your code part of <code>optimizer</code>. It seems <code>model.parameters()</code> is missing in <code>SGD</code> initialization. Please try to reduce the <code>lr</code> to see the result as well. <code>lr=1e-3</code> would be a good init try.</p>

username: John88
cooked: <p>Thank you!<br>
I forgot in this snippet code (I just edited it) but I did not in my code.</p>
<p>I will try lower lr <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to solve the normalization of four-dimension array

username: JasonChenhx
cooked: <p>I have already tried the sklearn.preprocessing.scale(), but it doesn’t work. The error information is ValueError: Found array with dim 4. the scale function expected &lt;= 2.</p>

username: ptrblck
cooked: <p>You could maybe use <code>torchvision.transforms.Normalize</code> or write the normalization manually.<br>
E.g. if you want to standardize the input you could subtract the mean and divide by the stddev.</p>
<p>Feel free to add more information about your use case, if you get stuck.</p>

username: JasonChenhx
cooked: <p>Thanks！ I will try it by your method</p>
Error making data set

username: kungkookie
cooked: <p>When I used labelme to make my own data set, the following error occurred when parsing the json file: would you please help to see why<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c9a65f01fa6f90e16e8f1b61e08b36e6243a034.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c9a65f01fa6f90e16e8f1b61e08b36e6243a034" title="无标题"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c9a65f01fa6f90e16e8f1b61e08b36e6243a034_2_690x338.png" alt="无标题" data-base62-sha1="mlnhtZXuBeOZyITrWxliZnohBWY" width="690" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c9a65f01fa6f90e16e8f1b61e08b36e6243a034_2_690x338.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c9a65f01fa6f90e16e8f1b61e08b36e6243a034_2_1035x507.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c9a65f01fa6f90e16e8f1b61e08b36e6243a034.png 2x" data-dominant-color="9EA0A2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">无标题</span><span class="informations">1327×651 80.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>This seems to be a <code>lableme</code> issue and you might get better support in their GitHub.<br>
Anyway, the error points to a missing <code>Qt</code> lib, so you might need to reinstall it.</p>

username: kungkookie
cooked: <p>I have reinstalled labelme and all the required libraries, but the following questions still appear. The imgviz module is available, thank you for your answer<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29252a3f95573be4ed0c8bbfabf56b4e3c4ee555.png" data-download-href="https://discuss.pytorch.org/uploads/default/29252a3f95573be4ed0c8bbfabf56b4e3c4ee555" title="无标题"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29252a3f95573be4ed0c8bbfabf56b4e3c4ee555_2_690x356.png" alt="无标题" data-base62-sha1="5RZcwv9jpk3DsW0hQRiIdQB1sR7" width="690" height="356" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29252a3f95573be4ed0c8bbfabf56b4e3c4ee555_2_690x356.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29252a3f95573be4ed0c8bbfabf56b4e3c4ee555_2_1035x534.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29252a3f95573be4ed0c8bbfabf56b4e3c4ee555.png 2x" data-dominant-color="959799"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">无标题</span><span class="informations">1182×610 51 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to update a non-differentiable parameter in a custom backward pass?

username: AK47
cooked: <p>I’d like to update a non-differentiable parameter, which is given as a function of variables that appear in backward(), in a custom backward pass. What can I do?</p>
<p>This may be a silly question, but if I multiply a parameter X with an intermediate variable in forward() that requires a custom backward() (since other ops in forward() require custom backward() and cannot be separated), do I still have to manually specify grad_X in backward(), though such operation as multiplication usually doesn’t require its gradient to be specified in backward()?</p>
How is dataloader implemented

username: championballer
cooked: <p>I am looking for details and files which handle the implementation of the dataloader. Primarily to understand how it handles the heterogeneity in forms and extensions of different data sources. Any help or info will be appreciated.</p>

username: quanguet
cooked: <p>It seems that you are looking for the implementation of <code>collate_fn</code> which handles the heterogeneity of the data sources from your own Dataset. You should pass your own <code>collate_fn</code> as the input of <code>Dataloader</code>. See more at <a href="https://pytorch.org/docs/stable/data.html" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html</a></p>

username: championballer
cooked: <p>Yeah, this is it. I don’t know how I missed it. Thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pretrained Model after fine-tuning taking longer inference time

username: bluesky314
cooked: <p>I am training a segmentation model from <a href="https://github.com/qubvel/segmentation_models.pytorch" rel="nofollow noopener">https://github.com/qubvel/segmentation_models.pytorch</a> and I first load the model with pretrained imagenet weights and the inference time on cpu is around 1 second. After I load my fine-tuned weights, the inference time jumps to about 10 seconds.</p>
<p>I checked the dtype of both, and both are float32. So what is going on? Why are fine-tuned weights taking much longer? Are the image-net weights pre-quantised or something?</p>

username: ptrblck
cooked: <p>Could you use <code>torch.set_flush_denormal(True)</code> to flush the denormals and see, if your performance increases again?</p>

username: bluesky314
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="73371">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.set_flush_denormal(True)</code></p>
</blockquote>
</aside>
<p>It did become a tiny bit faster but there is still a lag big between the two models. I manually checked the weighhts and it seems that the weights of both are only till 4 decimal places… Any other suggestions?</p>

username: bluesky314
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> please reply we are trying to deploy something  <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

username: ptrblck
cooked: <p>Unfortunately, I don’t know what else could slow down the CPU. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
I assume that you are using exactly the same setup and just load the weights?</p>

username: bluesky314
cooked: <p>Yes, its happening on colab and my coworkers pc as well</p>
How to do the same random crop on 2 images?

username: 07hyx06
cooked: <p>Hi! I want to do some augmentation on pix2pix model. I need to do the same random crop on 2 images. Ho to use <code>transforms.RandomCrop</code> to do that?</p>
<p>I added <code>torch.manual_seed(1)</code> before  <code>transforms.RandomCrop</code> , but it fails, output 2 different crop image <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: ptrblck
cooked: <p>I would recommend to use the functional API as shown <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">here</a>.</p>
C++ Autograd error

username: yessyw
cooked: <p>I can run mnist.cpp example.<br>
but, there is error in this simple code</p>
<p>torch::Tensor a = torch::ones({ 2, 2 }, torch::requires_grad());<br>
torch::Tensor b = torch::randn({ 2, 2 });<br>
auto c = a + b;<br>
c.backward();    &lt;— error</p>
<p>Also.<br>
I checked all of tensor’s options() using this code<br>
std::cout &lt;&lt; a.options() &lt;&lt; std::endl;<br>
std::cout &lt;&lt; b.options() &lt;&lt; std::endl;<br>
std::cout &lt;&lt; c.options() &lt;&lt; std::endl;</p>
<p>Result<br>
TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false)<br>
TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false)<br>
TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false)</p>
<p>I don’t know why there is no “requires_grad=true”</p>
<p>I need help</p>
How to match the output of one convolutional layer with the input of the next one?

username: Jessica_Castillo
cooked: <p>Let’s say I have this small architecture and I want to remove conv3. However, since I have pre-trained weights, I can’t (or don’t know how) directly change the output of conv2 to match conv4. Is there a way to resize the input of the fourth sequential without doing a convolution?</p>
<pre><code class="lang-auto">        # conv2
        self.conv2 = nn.Sequential(
            nn.MaxPool2d(2, stride=2, ceil_mode=True),  # 1/2
            nn.Conv2d(64, 128, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(128, 128, 3, padding=1),
            nn.ReLU(inplace=True),
        )

        # conv3
        """
        self.conv3 = nn.Sequential(
            nn.MaxPool2d(2, stride=2, ceil_mode=True),  # 1/4
            nn.Conv2d(128, 256, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, 3, padding=1),
            nn.ReLU(inplace=True),
        )"""

        # conv4
        self.conv4 = nn.Sequential(
            nn.MaxPool2d(2, stride=2, ceil_mode=True),  # 1/8
            #Something here that matches 128 with 256 without performing another convolution?
            nn.Conv2d(256, 512, 3, padding=1), 
            nn.ReLU(inplace=True),
            nn.Conv2d(512, 512, 3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(512, 512, 3, padding=1),
            nn.ReLU(inplace=True),
        )
</code></pre>

username: ptrblck
cooked: <p>You could use <code>nn.Upsample</code> to manipulate the spatial size of your activation.<br>
However, since your model is pretrained, you might need to finetune it again, as the input to <code>self.conv4</code> contains now new activations.</p>
White background fill with affine_grid and grid_sample

username: saraansh1999
cooked: <p>I am trying to use torch.nn.function.affine_grid and grid sample functionalities for a Spatial Transformer.  After the affine transformation the the image has black regions near the edges due to the shifting and rotation of the original image. Is there some way to define the fill value instead of having a default black fill in the empty regions. (like in the attatched image)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7ab466855c866c0ebcbf23358dff6e6b26d7ab07.png" alt="Screenshot from 2020-03-17 20-09-00" data-base62-sha1="hvuKYj75mxOrLT9RBgeWbmNNLw3" width="307" height="322"></p>

username: ptrblck
cooked: <p>You could set <code>padding_mode="border"</code> in <code>F.grid_sample</code>, where “border” defines the value you would like to use for out-of-grid locations.</p>
RuntimeError: cuda runtime error (46) : all CUDA-capable devices are busy or unavailable

username: agl29
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/fe25359d152c52afb53437035dc931c8da9a3f15.png" data-download-href="https://discuss.pytorch.org/uploads/default/fe25359d152c52afb53437035dc931c8da9a3f15" title="error(46)"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fe25359d152c52afb53437035dc931c8da9a3f15_2_690x213.png" alt="error(46)" data-base62-sha1="AggZTz592RZG57FxJupkNglC7aJ" width="690" height="213" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/fe25359d152c52afb53437035dc931c8da9a3f15_2_690x213.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/fe25359d152c52afb53437035dc931c8da9a3f15_2_1035x319.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/fe25359d152c52afb53437035dc931c8da9a3f15.png 2x" data-dominant-color="E6E2E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error(46)</span><span class="informations">1359×421 62.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>cuda version 8.0<br>
pytorch 0.4<br>
Gpu Memory available 8GB</p>
<p><strong>How can i solve this</strong></p>

username: agl29
cooked: <p>After resetting my GPU this error didn’t appear again.<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:"></p>
<p><strong>Edit</strong>: In my case although GPU memory was available but GPU cores were busy with some other process. Hence i got the above error.</p>

username: Yvonne_Yang
cooked: <p>How do I reset the gpu</p>
Increasing scores instead of decreasing loss in training

username: hjcho83
cooked: <p>I’m trying to solve a binary classification problem, and wondering whether it’s possible to improve/train a model by increasing scores like F1 instead of decreasing BCEWithLogits loss. I know decreasing loss naturally leads to the scores get increased, but I want to try to train model by explicitly increasing F1 scores.</p>

username: albanD
cooked: <p>Hi,</p>
<p>We usually do minimization so you can try to minimize <code>-F1</code> to get what you want.</p>

username: hjcho83
cooked: <p>I, myself, found a good explanation and implementation of using F1 as training loss as below, it didn’t work for my dataset though.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://miro.medium.com/fit/c/256/256/1*ChFMdf--f5jbm-AYv6VdYA@2x.png" class="site-icon" width="16" height="16">
      <a href="https://towardsdatascience.com/the-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d" target="_blank" title="06:34PM - 11 March 2020" rel="nofollow noopener">Medium – 11 Mar 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:99/36;"><img src="https://miro.medium.com/max/1200/0*hYwgNA39daffpTCb" class="thumbnail" width="99" height="36"></div>

<h3><a href="https://towardsdatascience.com/the-unknown-benefits-of-using-a-soft-f1-loss-in-classification-systems-753902c0105d" target="_blank" rel="nofollow noopener">The Unknown Benefits of using a Soft-F1 Loss in Classification Systems</a></h3>

<p>How to not maintain a decision threshold and still optimize directly for what you care about!</p>

  <p><span class="label1">Reading time: 14 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Pytorch Model Memory Leaks Issue

username: Vamshi_Krishna
cooked: <p>Hello folks,</p>
<p>Actually I have a requirement that convert keras model to pytorch, and I did it. But when am training keras model it is consuming 100MB of GPU memory and with the same data set when am training in pytorch it almost consuming 1GB of GPU memory, even i didn’t change the data set.</p>
<p>When i set pytorch model to(device), occupying 800MB of GPU memory but this is not happening with keras</p>
<p>appreciate for the quick response.</p>
<p>Thanks,<br>
Vamshi</p>

username: ptrblck
cooked: <p>The first CUDA operation will create the CUDA context, which will use ~700-800MB of device memory.<br>
Are you seeing a difference in the memory usage while running the training or only during the model loading?</p>

username: Vamshi_Krishna
cooked: <p>Yes, i could see difference while training model in keras and pytorch. Actually I haven’t specified specific gpu device in keras but it’s utilizing little bit GPU memory. Can I know the reason please?</p>

username: ptrblck
cooked: <p>What CUDA and cudnn versions are used in Keras/TF and how large is the allocated memory during the run in comparison to PyTorch?</p>
Learning rate setup when model is not being trained end-to-end

username: lazy_learner
cooked: <p>I am currently working on a model that consists of 2 parts a frontend and a backend.<br>
Now I am supposed to train this model in stages ie first frontend, then backend, and finally the entire model on an end-to-end basis.<br>
My question is that in the first and second stage which of the following 2 options should I choose:</p>
<ol>
<li>Set grad enabled true for the entire model but take 2 learning rates. The part to be trained(say frontend) will have a non-zero learning rate and the other(say backend) a learning rate of 0.</li>
<li>Set a single non-zero learning rate but make grad_enabled true for only the part that is to be trained(say frontend) and make sure that backend has lr = 0.</li>
</ol>

username: ptrblck
cooked: <p>I would suggest to use the second approach and not even creating an optimizer for the backend.<br>
Once you have finished training the frontend, you could create the optimizer for the backend and set <code>requires_grad=False</code> for the frontend.</p>
1D Kernel vs 1D Conv speed

username: Ilia_Karmanov
cooked: <p>I was curious why it’s a  bit slower to run 1D kernel using conv3D(k,1,1) versus Conv1D(k) and reshaping the input.</p>
<p>For example, if I define conv1,2,3 as just kernel:(k) the below is around 5x faster:</p>
<pre><code class="lang-auto">    def forward(self, x):
        # Tensor is NxCxTxHxW
        b,c,t,h,w = x.size()
        # Temporal axis
        conv0 = self.conv1(
            x.view(b,c,t*h*w)).view(b,c,t,h,w)
        # Height axis
        conv1 = self.conv2(
            x.permute(0,1,3,4,2).reshape(b,c,h*w*t)).view(b,c,h,w,t).permute(0,1,4,2,3)
        # Width axis
        conv2 = self.conv3(
            x.permute(0,1,4,3,2).reshape(b,c,w*h*t)).view(b,c,w,h,t).permute(0,1,4,3,2)
        # Out
        out = self.relu(conv0) + self.relu(conv1) + self.relu(conv2)
        return out
</code></pre>
<p>Than using:</p>
<pre><code class="lang-auto">        # Temporal
        self.conv0 = nn.Conv3d(
            in_channels,
            out_channels,
            kernel_size=(kernel_size[0], 1, 1),
            stride=(1, 1, 1),
            padding=(padding[0], 0, 0),
            groups=groups,
            bias=bias)

        # Height
        self.conv1 = nn.Conv3d(
            in_channels,
            out_channels,
            kernel_size=(1, kernel_size[1], 1),
            stride=(1, 1, 1),
            padding=(0, padding[1], 0),
            groups=groups,
            bias=bias)

        # Width
        self.conv2 = nn.Conv3d(
            in_channels,
            out_channels,
            kernel_size=(1, 1, kernel_size[2]),
            stride=(1, 1, 1),
            padding=(0, 0, padding[2]),
            groups=groups,
            bias=bias)

    def forward(self, x):
        t0, t1, t2 = self.conv0(x), self.conv1(x), self.conv2(x)
        out = self.relu(t0) + self.relu(t1) + self.relu(t2)
        return out

</code></pre>
<p>And as far as I’m aware the resulting operation should be the same?</p>

username: ptrblck
cooked: <p>Did you synchronize the code before starting and stopping the timer?<br>
If so, could you disable cudnn via <code>torch.backends.cudnn.enabled = False</code> and profile the code again, please?</p>
Is there any different between torch.sigmoid and torch.nn.functional.sigmoid?

username: Teaonly
cooked: <p>I am confused with torch.xxxx which support Tensor and Variable, it should be a function accepting Variable or just a normal function which handling input tensors.  But what different between torch.xxxx and <a href="http://torch.nn.functional.xxx" rel="nofollow noopener">torch.nn.functional.xxx</a>  when input is a Variable ?</p>

username: apaszke
cooked: <p><code>torch.*</code> functions work for both Tensors and Variables. There’s no difference between <code>torch.sigmoid</code> and <code>torch.nn.functional.sigmoid</code>.</p>

username: fmassa
cooked: <p>Actually, there is a difference.<br>
The implementations in <code>torch.xxx</code> have the <a href="https://github.com/pytorch/pytorch/blob/master/torch/autograd/_functions/pointwise.py#L70">backward implemented using <code>python</code> calls</a>, while the functional counterparts have their backward implemented entirely in <a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/THNN/generic/Sigmoid.c#L28">C</a>/<a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/THCUNN/Sigmoid.cu#L20">Cuda</a>, so the <code>functional</code> backward code is more efficient.</p>

username: Teaonly
cooked: <p>Thanks, i see.</p>
<p>More different is help message, there are help text for python object, no text for registered functions in C.</p>

username: cakeeatingpolarbear
cooked: <p>Is this still the case? If so should be writing all my code using <code>functional</code> api?</p>

username: Haydnspass
cooked: <p>I guess this is deprecated (at least as of Pytorch 1.0.0). But since one can find it so easily via google, I wrote this reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
  warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
</code></pre>

username: drevicko
cooked: <p>Hi <a class="mention" href="/u/fmassa">@fmassa</a>: is this still the case?</p>
Multiprocessing to speed up data pre-processing during training?

username: T.Liu
cooked: <p>Hi everyone,</p>
<p>We have some data pre-processing using PyTorch &amp; Numpy before feeding into training of the network. And these are now done serially, which slows down our training process. Is there any approach in PyTorch to speed it up such as using multiprocessing? Any kinds of suggestsions are also welcome. Thanks.</p>

username: ayalaa2
cooked: <p><a href="https://pytorch.org/docs/stable/data.html#single-and-multi-process-data-loading" rel="nofollow noopener">PyTorch supports multi-process data loading.</a></p>
<p>The general idea is to create a dataset class and implement the pre-processing within the get functionality. If you create a data loader instance with that dataset and set <code>num_workers</code>, worker processes will handle fetching and preprocessing data.</p>
Importing TensorFlow model for Inference in Torchscript

username: solarflarefx
cooked: <p>I would like to use a trained model I have in TensorFlow for inference in TorchScript.  I would like to compare the performance between the two.</p>
<p>What is the best method to do this?</p>
<p>I believe it would be the following:<br>
Convert TensorFlow model to frozen graph (.pb) --&gt; Convert to ONNX (.onnx) --&gt; Convert to PyTorch model (.pt)</p>
<p>Is this correct?  Has anyone tried this approach before?  Is there another approach that is recommended?</p>

username: solarflarefx
cooked: <p>Or is there a way to create a TorchScript from an onnx model?</p>
NotImplementError while inspecting PyTorch nn model

username: paul_shuvo
cooked: <p>I’m trying to inspect the model(PyTorch) but it shows the following error!</p>
<pre><code class="lang-auto">Unable to display children:Error resolving variables Traceback (most recent call last):
  File "D:\Program Files\JetBrains\PyCharm 2019.2.3\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 1208, in do_it
    xml.write(pydevd_xml.var_to_xml(val, k, evaluate_full_value=evaluate_full_value))
  File "D:\Program Files\JetBrains\PyCharm 2019.2.3\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_xml.py", line 369, in var_to_xml
    elif hasattr(v, "__len__") and not is_string(v):
  File "D:\Anaconda3\envs\pt37\lib\site-packages\torch\nn\backends\backend.py", line 10, in __getattr__
    raise NotImplementedError
NotImplementedError
</code></pre>
<p>How should I implement the  <code>__getattr__</code> method here?</p>

username: ptrblck
cooked: <p>It seems <code>var_to_xml</code> expects a <code>__len__</code> method of the inspected object, which is not implemented for classes of <code>nn.Module</code>.</p>

username: PersonalFinance004
cooked: <p>I also have that problem when trying to inspect an instance of torch.nn.Module in PyCharm’s debugger. How to resolve?</p>

username: PersonalFinance004
cooked: <p>Please can someone offer a solution. This is driving me nuts.</p>

username: Hongming_Xu
cooked: <p>I am also facing the same issue. I want check Pytorch model and model layers’ info. But we cannot see it using Pycharm debugger. Any one has good solutions on it, please?</p>